Polars: High-Performance DataFrame Operations in Python

Overview

Polars is a DataFrame library written in Rust with Python bindings. It stores data as Apache Arrow columns, parallelizes across all CPU cores, and in lazy mode rewrites the whole query plan before reading a single row. The difference shows up in benchmarks, not just marketing: in Polars’ own PDS-H run (May 2025, scale factor 10), the streaming engine finished the suite in 3.89s where pandas took 365.71s.

I reach for Polars when a pandas job starts crawling on multi-GB files but spinning up Spark feels like overkill. The expression API takes about an hour to click — you write pl.col() expressions instead of indexing columns — and after that the migration is mostly mechanical.

When to Use

• Datasets from 1GB to ~100GB that make pandas swap or crawl. Typical case: CSV/Parquet ETL on one machine.
• Group-by and join-heavy pipelines where pandas is the bottleneck. The benchmark above is exactly this workload.
• Replacing pandas inside an existing ETL pipeline without moving to Spark.
• Pipelines where the optimizer matters, because lazy evaluation skips columns and rows you never use.
• Reading and writing Parquet, CSV, or IPC (Arrow) files at scale.

When NOT to Use

• Small datasets (<100MB): pandas has better ecosystem compatibility and you won’t notice the speedup.
• When you depend on pandas-native libraries (geopandas, statsmodels, most of scikit-learn’s preprocessing).
• Heavy interactive notebook exploration: pandas’ eager evaluation is more forgiving when you’re poking at data.
• A team already deep into pandas where speed isn’t the problem.

Solution

Basic DataFrame operations

import polars as pl

# Read CSV
df = pl.read_csv("data/orders.csv")

# Read Parquet
df = pl.read_parquet("data/orders.parquet")

# Create from dict
df = pl.DataFrame({
    "order_id": [1, 2, 3, 4, 5],
    "customer": ["Alice", "Bob", "Alice", "Charlie", "Bob"],
    "amount": [100.0, 250.0, 75.5, 300.0, 150.0],
    "order_date": ["2025-01-01", "2025-01-02", "2025-01-03", "2025-01-04", "2025-01-05"],
})

# Select columns
df.select(["order_id", "amount"])

# Filter rows
df.filter(pl.col("amount") > 100)

# Sort
df.sort("amount", descending=True)

# Add derived columns
df.with_columns([
    (pl.col("amount") * 1.1).alias("amount_with_tax"),
    pl.col("customer").str.to_uppercase().alias("customer_upper"),
])

# Group by and aggregate
df.group_by("customer").agg([
    pl.col("amount").sum().alias("total_spent"),
    pl.col("order_id").count().alias("order_count"),
    pl.col("amount").mean().alias("avg_order"),
])

Lazy evaluation

# Use scan_* for lazy evaluation (doesn't load data yet)
lf = pl.scan_parquet("data/orders.parquet")

# Build the query plan; nothing executes yet
result = (
    lf
    .filter(pl.col("amount") > 50)
    .with_columns([
        pl.col("order_date").str.strptime(pl.Date, "%Y-%m-%d").alias("date"),
        (pl.col("amount") * pl.col("quantity")).alias("revenue"),
    ])
    .group_by(["customer", pl.col("date").dt.year().alias("year")])
    .agg([
        pl.col("revenue").sum().alias("total_revenue"),
        pl.col("order_id").n_unique().alias("orders"),
    ])
    .sort("total_revenue", descending=True)
)

# Execute: Polars rewrites the plan and reads only the needed columns
df = result.collect()

Because the plan is lazy, Polars can:

• Push the amount > 50 filter into the scan, so non-matching rows never get decoded.
• Read only the columns used downstream: order_date, amount, quantity, customer, order_id.
• Reorder joins and merge repeated subexpressions.

Reading and writing

# Read CSV with explicit schema
df = pl.read_csv("data/orders.csv", schema_overrides={
    "order_id": pl.Int64,
    "amount": pl.Float64,
    "order_date": pl.Date,
}, try_parse_dates=True)

# Write to Parquet
df.write_parquet("data/output.parquet", compression="zstd")

# Write to CSV
df.write_csv("data/output.csv")

# Write to IPC (Arrow format, fastest round-trip for Polars)
df.write_ipc("data/output.arrow")

# Read many files at once
df = pl.scan_csv("data/part-*.csv").collect()

Joins

orders = pl.DataFrame({
    "order_id": [1, 2, 3],
    "customer_id": [101, 102, 101],
    "amount": [100, 200, 150],
})

customers = pl.DataFrame({
    "customer_id": [101, 102, 103],
    "name": ["Alice", "Bob", "Charlie"],
    "city": ["NYC", "LA", "Chicago"],
})

# Inner join
joined = orders.join(customers, on="customer_id", how="inner")

# Left join
joined = orders.join(customers, on="customer_id", how="left")

# Join with different column names
orders = orders.rename({"customer_id": "cust_id"})
joined = orders.join(customers, left_on="cust_id", right_on="customer_id", how="left")

# Join on multiple columns
joined = orders.join(customers, on=["customer_id", "region"], how="inner")

Conditional expressions

df = df.with_columns([
    pl.when(pl.col("amount") > 200)
    .then(pl.lit("high"))
    .when(pl.col("amount") > 100)
    .then(pl.lit("medium"))
    .otherwise(pl.lit("low"))
    .alias("tier"),
])

# Map values: replace_strict is the fast path for value mapping
df = df.with_columns([
    pl.col("status").replace_strict({
        "P": "pending",
        "C": "completed",
        "X": "cancelled",
    }, default=pl.col("status")).alias("status_label"),
])

replace_strict raises if a value has no mapping, so pass default= to keep unmapped rows. For complex per-element logic, map_elements(callable) still works, but it runs a Python function per value and is the slowest path.

Window functions

df = df.with_columns([
    # Running total per customer
    pl.col("amount").cum_sum().over("customer").alias("running_total"),

    # Row number per customer ordered by date
    pl.col("order_id").rank().over("customer").alias("order_seq"),

    # Lag: previous amount per customer
    pl.col("amount").shift(1).over("customer").alias("prev_amount"),

    # Moving average
    pl.col("amount").rolling_mean(window_size=3).over("customer").alias("ma_3"),
])

Concatenation

# Vertical: stack rows
df_all = pl.concat([df_jan, df_feb, df_mar])

# Horizontal: side by side
df_wide = pl.concat([df_left, df_right], how="horizontal")

# Diagonal: fill missing columns with null
df_combined = pl.concat([df_a, df_b], how="diagonal")

Streaming for large datasets

# Stream processing: batches through the pipeline with bounded memory
lf = pl.scan_csv("data/huge_file.csv")

result = (
    lf
    .filter(pl.col("amount") > 0)
    .group_by("customer")
    .agg(pl.col("amount").sum())
    .sort("amount", descending=True)
)

# engine="streaming" runs the query in bounded memory
df = result.collect(engine="streaming")

To make streaming the default for every lazy query in a project, set pl.Config.set_engine_affinity(engine="streaming") once at startup.

Interop with pandas

import pandas as pd
import polars as pl

# pandas to Polars
pdf = pd.read_csv("data.csv")
plf = pl.from_pandas(pdf)

# Polars to pandas
plf = pl.read_csv("data.csv")
pdf = plf.to_pandas()

# Use Polars for heavy computation, convert back to pandas for plotting
result = (
    pl.from_pandas(pdf)
    .lazy()
    .filter(pl.col("amount") > 100)
    .group_by("customer")
    .agg(pl.col("amount").sum())
    .collect()
    .to_pandas()
)

result.plot(kind="bar", x="customer", y="amount")

SQL interface

df = pl.read_parquet("data/orders.parquet")

result = pl.sql("""
    SELECT customer, SUM(amount) as total, COUNT(*) as orders
    FROM df
    WHERE amount > 50
    GROUP BY customer
    ORDER BY total DESC
""").collect()

Explanation

How the lazy optimizer works

pl.scan_* returns a LazyFrame: a description of the work, not the data. Nothing is read until you call .collect(). Between the two, Polars builds a logical plan and rewrites it. The optimizations that matter most in practice:

• Predicate pushdown: filters move into the scan. On Parquet, non-matching row groups are skipped before decoding.
• Projection pushdown: only columns referenced downstream get read. On a wide file where you use 5 of 200 columns, most of the IO disappears.
• Common subplan elimination: if two branches of the query compute the same thing, Polars computes it once.

You can inspect the plan before running it, which is the fastest way to confirm a filter actually made it into the scan:

lf = pl.scan_parquet("data/orders.parquet").filter(pl.col("amount") > 50)
print(lf.explain())
# Shows PARQUET SCAN with the pushed-down predicate and projected columns

Polars vs pandas vs Dask vs Spark

	pandas	Polars	Dask	PySpark
Memory format	NumPy	Apache Arrow	pandas partitions	JVM / Arrow
Execution	Eager, single thread	Eager or lazy, all cores	Lazy, multi-machine	Lazy, cluster
Query optimizer	No	Yes	Partial	Catalyst
Sweet spot	<1GB, ecosystem glue	1GB–100GB, one node	>100GB, a few nodes	Existing cluster
Out-of-memory data	No	engine="streaming"	Disk spilling	Built in

My rule of thumb: pandas for exploration and ecosystem compatibility, Polars when the same machine should go faster, Dask or Spark only when the data genuinely doesn’t fit one node. For the alternatives, see Dask parallel DataFrames and Spark group-by aggregations.

Memory and Arrow

Polars columns are Arrow arrays: contiguous, typed, and shared-memory friendly. Two practical consequences:

• No index. Row selection goes through filter() and expressions, not .loc / .iloc. This is the thing that trips pandas users on day one.
• Zero-copy interop with Arrow-aware tools (DuckDB, PyArrow). to_pandas() is also cheap when you pass use_pyarrow_extension_array=True, because pandas 2.x can hold Arrow-backed columns without copying.

Because columns are contiguous and aligned, Polars vectorizes heavily. Much of the speed comes from that memory layout alone, before the optimizer even runs.

Schema drift and versions

Two habits prevent most production surprises:

• Pass schema_overrides (or raise infer_schema_length) when reading CSVs. The default inference samples a limited number of rows, and a file where row 50,000 suddenly contains a decimal in an integer column will fail the load.
• Pin a major version. The examples here target polars>=1.0; several APIs changed at the 1.0 boundary — pivot’s columns became on, and collect(streaming=True) became engine="streaming".

For validation beyond column types — value ranges, nullability, custom checks — pair Polars with Pandera, which ships a Polars backend.

Variants

Using with PyArrow

import pyarrow as pa
import polars as pl

# PyArrow Table to Polars
table = pa.Table.from_pandas(pdf)
plf = pl.from_arrow(table)

# Polars to PyArrow Table
table = plf.to_arrow()

Custom aggregation functions

# Custom aggregation with map_elements (a callable, not a dict)
df.group_by("customer").agg([
    pl.col("amount").map_elements(lambda x: x.quantile(0.95)).alias("p95_amount"),
    pl.col("amount").std().alias("std_amount"),
    pl.col("amount").median().alias("median_amount"),
])

# Group multiple columns into a struct
df.group_by("customer").agg([
    pl.struct(["amount", "order_id"]).alias("order_details"),
])

Prefer pl.col("amount").quantile(0.95) when a native expression exists; map_elements is the fallback for logic Polars doesn’t cover.

Pivot tables

# Pivot: rows=customer, columns=month, values=amount
pivoted = (
    df
    .with_columns(
        pl.col("order_date").str.strptime(pl.Date, "%Y-%m-%d").dt.month().alias("month")
    )
    .pivot(on="month", index="customer", values="amount", aggregate_function="sum")
)

Best Practices

• For a deeper guide on the distributed alternative, see Parallel DataFrame Operations with Dask.
• Use scan_* (lazy) instead of read_* (eager) for files so the optimizer sees the whole pipeline.
• Call .collect() once at the end; every extra collect materializes an intermediate result.
• Filter early in lazy pipelines: a pushed-down predicate never touches the row.
• Write expressions with pl.col() rather than raw column names so calls chain cleanly.
• Write Parquet with compression="zstd": the best size/speed tradeoff in practice.
• Use engine="streaming" for datasets larger than RAM; the streaming engine processes batches in bounded memory.
• Reach for replace_strict, when/then/otherwise, and built-in expressions before map_elements; the Python callback path is far slower.
• Prefer over() window functions to sorting plus manual loops.
• Validate inputs early with schema_overrides; a mistyped column poisons every downstream expression.

Common Mistakes

• Using eager mode for large files: pl.read_csv loads everything at once. pl.scan_csv(...).collect(engine="streaming") keeps memory bounded.
• Converting to pandas too early: to_pandas() copies data and drops the Arrow benefits. Stay in Polars until you actually need a pandas-only library.
• Passing a dict to map_elements: it expects a callable, so map_elements({"a": 1}) raises TypeError. Value mapping is replace_strict or replace.
• Copying pre-1.0 snippets: pivot(columns=...) and collect(streaming=True) were renamed in Polars 1.0. Check signatures against your installed version.
• Ignoring schema_overrides: Polars infers CSV types from a row sample, and on large files the sample misses edge cases. Declare types explicitly.
• Keeping map_elements in hot loops: a Python callback per element is the slow path; built-in expressions vectorize.

Summary

• Polars is a single-node DataFrame engine: Arrow memory, all cores, optional lazy optimization.
• Start lazy with scan_*, build one plan, call collect() once. Switch to engine="streaming" when data exceeds RAM.
• The API differences that bite pandas users: pl.col() expressions instead of column indexing, no index, and replace_strict instead of .map(dict).
• Examples target polars>=1.0; pre-1.0 snippets you’ll find online use different parameter names.

See Also

• Polars user guide: official documentation for the lazy API, streaming engine, and expression reference.
• Polars Python API reference: current signatures for pivot, replace_strict, collect(engine=...).
• Updated PDS-H benchmark results: the official numbers behind the pandas comparison in the Overview.
• Apache Arrow columnar format: the memory model Polars builds on.