Polars Data Pro

Polars Data Pro turns Claude into a Polars 1.x specialist for production data engineering and analysis in Python. It writes idiomatic expression-based code, reasons about lazy query plans, and translates legacy Pandas pipelines into Polars that runs faster and uses far less memory. Built for data engineers, analysts, and backend developers who process tables that are slow or impossible to handle in Pandas.

When to use it

• Your Pandas job is slow or runs out of RAM on a multi-GB CSV/Parquet file.
• You're porting a Pandas script and keep hitting SettingWithCopyWarning, .apply() bottlenecks, or index headaches.
• You need correct groupby aggregations, asof/semi/anti joins, or window functions and want them expressed cleanly.
• You want a lazy pipeline that streams data larger than memory instead of materializing it.

How it works

The skill works in the expression API and knows the LazyFrame vs DataFrame split cold. It builds pipelines with pl.scan_parquet/scan_csv, chains .filter(), .with_columns(), .group_by().agg(), and .join(), then triggers execution with .collect(). It uses pl.col(), pl.when().then().otherwise(), and window expressions via .over(). For aggregations it reaches for pl.sum, pl.mean, pl.len, pl.first, and conditional aggregates. It applies engine="streaming" in .collect() for out-of-memory workloads and uses .explain() to inspect the optimized query plan (predicate/projection pushdown). Joins cover how="inner"/"left"/"semi"/"anti"/"cross" plus join_asof. It prefers native expressions over .map_elements() and knows when .lazy() unlocks optimization.

Example

curl -fsSL --create-dirs -o ~/.claude/skills/polars-data-pro/SKILL.md https://claudeskil.com/skills/polars-data-pro/SKILL.md

import polars as pl

out = (
    pl.scan_parquet("events/*.parquet")          # lazy, no I/O yet
    .filter(pl.col("status") == "paid")
    .with_columns(
        rolling_avg=pl.col("amount").mean().over("user_id")
    )
    .group_by("country")
    .agg(
        revenue=pl.col("amount").sum(),
        users=pl.col("user_id").n_unique(),
    )
    .sort("revenue", descending=True)
    .collect(engine="streaming")                  # executes optimized plan
)

Tips & gotchas

• Stay lazy until the end. Calling .collect() mid-pipeline defeats predicate and projection pushdown — chain everything, collect once.
• .over() does NOT collapse rows like a groupby; it broadcasts the aggregate back across the partition. Use group_by().agg() when you want one row per key.
• Avoid .map_elements() — it drops to slow Python. Express logic with native expressions or .list/.str namespaces instead.
• Streaming supports most but not all operations; if a node falls back, .explain(streaming=True) shows which part won't stream.

FAQ

Is Polars a drop-in replacement for Pandas? No. There's no row index, assignment is via .with_columns() not df["x"] = ..., and everything is expression-driven. The skill handles the translation patterns for you.

How do I move data to/from Pandas? Use df.to_pandas() and pl.from_pandas(df). Zero-copy works for Arrow-backed columns; the skill flags when a copy is forced.

When is streaming actually worth it? When the dataset exceeds RAM or you're scanning many Parquet files. For data that fits in memory, the default in-memory engine is usually faster.