---
name: polars-data-pro
description: "Polars DataFrames — lazy evaluation, expressions, joins, window functions, the fast Pandas alternative. Polars 1.x for production data work: LazyFrame vs DataFrame, expressions API, joins, groupby aggregations, window functions, streaming for out-of-memory data, and the Pandas migration patterns that get you 10x speedup."
license: MIT
---

# Polars Data Pro

Polars Data Pro turns Claude into a Polars 1.x specialist for production data engineering and analysis in Python. It writes idiomatic expression-based code, reasons about lazy query plans, and translates legacy Pandas pipelines into Polars that runs faster and uses far less memory. Built for data engineers, analysts, and backend developers who process tables that are slow or impossible to handle in Pandas.

## When to use it

- Your Pandas job is slow or runs out of RAM on a multi-GB CSV/Parquet file.
- You're porting a Pandas script and keep hitting `SettingWithCopyWarning`, `.apply()` bottlenecks, or index headaches.
- You need correct groupby aggregations, asof/semi/anti joins, or window functions and want them expressed cleanly.
- You want a lazy pipeline that streams data larger than memory instead of materializing it.

## How it works

The skill works in the expression API and knows the LazyFrame vs DataFrame split cold. It builds pipelines with `pl.scan_parquet`/`scan_csv`, chains `.filter()`, `.with_columns()`, `.group_by().agg()`, and `.join()`, then triggers execution with `.collect()`. It uses `pl.col()`, `pl.when().then().otherwise()`, and window expressions via `.over()`. For aggregations it reaches for `pl.sum`, `pl.mean`, `pl.len`, `pl.first`, and conditional aggregates. It applies `engine="streaming"` in `.collect()` for out-of-memory workloads and uses `.explain()` to inspect the optimized query plan (predicate/projection pushdown). Joins cover `how="inner"/"left"/"semi"/"anti"/"cross"` plus `join_asof`. It prefers native expressions over `.map_elements()` and knows when `.lazy()` unlocks optimization.

## Example

```python
import polars as pl

out = (
    pl.scan_parquet("events/*.parquet")          # lazy, no I/O yet
    .filter(pl.col("status") == "paid")
    .with_columns(
        rolling_avg=pl.col("amount").mean().over("user_id")
    )
    .group_by("country")
    .agg(
        revenue=pl.col("amount").sum(),
        users=pl.col("user_id").n_unique(),
    )
    .sort("revenue", descending=True)
    .collect(engine="streaming")                  # executes optimized plan
)
```

## Tips & gotchas

- Stay lazy until the end. Calling `.collect()` mid-pipeline defeats predicate and projection pushdown — chain everything, collect once.
- `.over()` does NOT collapse rows like a groupby; it broadcasts the aggregate back across the partition. Use `group_by().agg()` when you want one row per key.
- Avoid `.map_elements()` — it drops to slow Python. Express logic with native expressions or `.list`/`.str` namespaces instead.
- Streaming supports most but not all operations; if a node falls back, `.explain(streaming=True)` shows which part won't stream.

## FAQ

**Is Polars a drop-in replacement for Pandas?**
No. There's no row index, assignment is via `.with_columns()` not `df["x"] = ...`, and everything is expression-driven. The skill handles the translation patterns for you.

**How do I move data to/from Pandas?**
Use `df.to_pandas()` and `pl.from_pandas(df)`. Zero-copy works for Arrow-backed columns; the skill flags when a copy is forced.

**When is streaming actually worth it?**
When the dataset exceeds RAM or you're scanning many Parquet files. For data that fits in memory, the default in-memory engine is usually faster.