Skip to content
Tutorials Center
My learning

Topics

Lesson 25 of 37 ยท IO and Formats

Parquet, JSON and Arrow IPC

Pick the right file format for Polars: Parquet for storage, Arrow IPC for speed, JSON and NDJSON for exchange. Learn the read/write calls and trade-offs.

CSV is easy to open in a text editor but poor at everything else: it has no types, no compression, and reading it means parsing text character by character. Once your data leaves the "send someone a spreadsheet" stage, binary formats are better. Polars speaks several fluently:

Format Best for Polars calls
Parquet Long-term storage, analytics, sharing read_parquet, scan_parquet, write_parquet, sink_parquet
Arrow IPC (Feather v2) Fast hand-off between tools and processes read_ipc, scan_ipc, write_ipc
JSON / NDJSON APIs, logs, nested records read_json, read_ndjson, write_json, write_ndjson
CSV Human-readable exchange read_csv, write_csv
Excel, databases Business and application data read_excel, read_database (need extra packages)

What runs here. The browser build of Polars on this site includes CSV and Arrow IPC, but it was compiled without its own Parquet and JSON readers and writers. The pl.read_parquet(...), scan_parquet(...) and read_ndjson(...) snippets below are therefore read-only and marked as such. Where it helps, the runnable examples use the pyarrow and json packages (available in the browser) as clearly labelled stand-ins: they produce the same file contents and then hand the data to Polars. On a normal installation, the Polars calls themselves work as shown.

Parquet: the default for analytics

Parquet stores data by column, compressed, with the schema and per-column statistics inside the file. A million-row table that is 60 MB as CSV is often 5 to 10 MB as Parquet, and it reads many times faster because Polars only touches the columns you ask for and can skip whole blocks of rows using the statistics.

Typical use (read-only, run on your machine):

Python
import polars as pl

df = pl.read_parquet("sales.parquet")                       # whole file
df = pl.read_parquet("sales.parquet", columns=["region", "revenue"])

lf = pl.scan_parquet("sales/*.parquet")                      # lazy, many files
result = lf.filter(pl.col("year") == 2025).group_by("region").agg(pl.sum("revenue")).collect()

df.write_parquet("out.parquet", compression="zstd", compression_level=6)

Compression choices: "zstd" (best general choice), "snappy" (very fast, larger), "lz4", "gzip", or "uncompressed". You can also write a table partitioned into folders (year=2025/...) with pl.PartitionByKey on newer versions, which lets later queries skip entire folders.

A runnable stand-in with pyarrow

This example writes a Parquet file into memory with pyarrow, shows how small it is, reads back only one column, and finally converts the table to Polars:

Python Parquet round trip (pyarrow stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The repeating values compress extremely well. Notice that the schema comes back exactly as it went in: no try_parse_dates, no type guessing.

Arrow IPC: zero-friction speed

Arrow IPC (the file format also called Feather v2) is Polars' own in-memory layout written straight to disk. It compresses less than Parquet but reads almost instantly, because there is nothing to decode. It is excellent for scratch files and for passing data between processes. This one runs for real in the browser:

Python Arrow IPC round trip

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Compare what survived: unlike CSV, nulls and the Categorical dtype round-trip exactly. A real path works the same way, and pl.scan_ipc("file.arrow") gives you a lazy version.

A good division of labour: Parquet for data you keep, IPC for data you pass along in the next five minutes, CSV only when a human or another system demands it.

JSON and NDJSON

JSON can represent nesting, which tables cannot do directly. Polars maps JSON objects to structs and arrays to lists. There are two layouts:

  • JSON array: [{"a": 1}, {"a": 2}], one document; read with pl.read_json.
  • NDJSON (newline-delimited, also called JSON Lines): one object per line; read with pl.read_ndjson or lazily with pl.scan_ndjson. This layout is the right choice for logs and large exports because it can be read in pieces.

Read-only snippets for a normal installation:

Python
df = pl.read_ndjson("events.ndjson")
df.write_ndjson("copy.ndjson")
df.write_json()                       # returns a JSON string

Runnable stand-in: Python's json module

Python's json module turns JSON text into dicts, and pl.from_dicts builds a DataFrame from them. Missing keys become null, and arrays become list columns:

Python NDJSON text to a DataFrame and back

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

JSON inside a column

APIs often deliver a column of JSON strings. On a normal installation you decode them with str.json_decode(dtype) and then unnest the struct into ordinary columns (read-only here: the browser build lacks the JSON feature):

Python
dtype = pl.Struct({"city": pl.String, "temp": pl.Float64})
flat = df.with_columns(pl.col("payload").str.json_decode(dtype)).unnest("payload")

The runnable stand-in below decodes with Python's json module and then uses unnest, which does work in the browser, to show the same end result:

Python Decode a JSON string column (stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

You can also work on nested data with the struct namespace (pl.col("payload").struct.field("city")).

Choosing a format

flowchart TD
    Q{"What is the data for?"}
    Q -->|"keep and analyze"| P["Parquet"]
    Q -->|"hand to another process now"| I["Arrow IPC"]
    Q -->|"API payloads, nested logs"| J["NDJSON"]
    Q -->|"a person will open it"| C["CSV / Excel"]

Match the format to the job, not to habit.

A few more rules of thumb: Parquet and IPC preserve types, so you never lose Int32 vs Int64, dates or categories; CSV and JSON lose or guess types. Parquet's column statistics make filtered reads fast. NDJSON can be appended to line by line, which Parquet cannot.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is a Parquet file usually much smaller and faster to query than the same data as CSV?

    Columnar layout plus compression and per-block statistics shrink the file and let Polars read only the needed columns and row groups.
  2. 2. Which format is best for quickly handing a DataFrame to another process, with no decoding cost?

    IPC is Polars' in-memory Arrow layout written to disk; it reads almost instantly and keeps all types.
  3. 3. What is NDJSON?

    Newline-delimited JSON puts a complete JSON document on each line, which suits logs and streaming.
  4. 4. After str.json_decode(dtype) on a string column, how do you turn the struct fields into normal columns?

    unnest expands the fields of a struct column into separate columns of the DataFrame.

Summary

  • Parquet is the default for stored analytics data: compact, typed, fast, with column pruning and row-group skipping.
  • Arrow IPC is the fastest to read and keeps all types; ideal for temporary hand-offs.
  • JSON arrays and NDJSON carry nested data; objects become structs, arrays become lists.
  • str.json_decode(dtype) plus unnest converts JSON strings in a column into columns.
  • In the browser runner, use the pyarrow and json stand-ins shown above; on your own machine the native read_parquet, scan_parquet and read_ndjson calls work directly.

Next: Interoperability: pandas, NumPy, Arrow and SQL