Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 36 ยท Storage Formats and Lakehouses

Parquet in Practice

Open a Parquet file and see inside: row groups, column chunks, min/max statistics, dictionary encoding, compression codecs, filters and schema evolution.

Parquet is the default file format of the modern data stack. Spark, DuckDB, Trino, BigQuery, Snowflake external tables, pandas and Polars all read it. To use it well, you should know what is inside a file, because the layout controls how much data a query must read.

In this lesson you build Parquet files with pyarrow, inspect their metadata, and watch filters skip data. You also learn how compression and encoding choices change size, and how to evolve a schema without rewriting history.

Anatomy of a Parquet file

A Parquet file is split into row groups (typically 128 MB to 1 GB uncompressed in big-data tools, a few thousand to a million rows in small files). Inside a row group, each column has a column chunk, which is divided into pages. The file ends with a footer holding the schema and statistics for every chunk.

flowchart TB
  F["Parquet file"] --> RG1["Row group 1"]
  F --> RG2["Row group 2"]
  F --> FT["Footer: schema + statistics"]
  RG1 --> C1["Column chunk: id"]
  RG1 --> C2["Column chunk: price"]
  C2 --> P["Pages (compressed)"]

A reader starts at the end: it reads the footer, decides which row groups and columns it needs, and fetches only those byte ranges. This is why Parquet works well on object storage, where you can request a range of a file.

Inspect the metadata

Python Row groups and statistics

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Because id increases through the file, each row group has a distinct id range. A filter such as id >= 7500 can skip groups 0 to 2 completely.

Filters that skip data

Python Predicate pushdown in pyarrow

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Engines apply the same logic automatically; the lesson of the example is that sorting or clustering data by the filter column makes the statistics sharp. If ids were random, every group would span the full range and nothing could be skipped.

Encoding and compression

Parquet applies two layers to each column chunk:

  1. Encoding that exploits the data shape: dictionary encoding for low-cardinality columns, run-length encoding for repeated values, delta encoding for increasing numbers.
  2. Compression of the encoded pages with a general codec.

Common codecs:

Codec Size Speed When to use
snappy medium very fast default in Spark; good all-rounder
zstd small fast best size/speed trade-off; choose for cold or large data
gzip small slow legacy compatibility
none large fastest rarely, for data that is already compressed
Python Compare codecs and dictionary encoding

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Dictionary encoding is not always a win. In this data the session column is unique per row, so its dictionary is as large as the column itself and turning it off gives a smaller file. Writers normally decide per column and fall back automatically when a dictionary grows too big; the lesson is to measure on your own data rather than assume.

Schema evolution

Data changes. Parquet stores the schema in each file, and engines can merge schemas of many files. Safe, additive changes work well: adding a new nullable column, or reading an old file where the column simply does not exist (it comes back as null). Risky changes are renaming a column (old files still use the old name) and changing a type (int to string breaks readers).

Python Reading files with different schemas

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Table formats (next lessons) track schema changes centrally, so you do not need to hand-merge like this.

Practical rules

  • Aim for files of 100 MB to 1 GB; thousands of 1 MB files are slow because each needs its own open and footer read.
  • Sort or cluster by the column you filter on most (often a date or a customer id).
  • Use explicit types, including timestamp with a time zone policy and decimal for money.
  • Pick zstd unless a tool you use does not support it.
  • Never overwrite a file in place while readers are using it; write a new file and swap (table formats automate this).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Where does a Parquet reader find the schema and column statistics?

    The footer holds the schema and per-chunk statistics, so readers start at the end of the file.
  2. 2. A filter on id is not skipping any row groups. What is the most likely reason?

    Statistics only help when values are clustered. Sorting by the filter column makes the ranges narrow.
  3. 3. Which change is the safest schema evolution for existing Parquet files?

    Old files lack the new column and are read as nulls; renames and type changes break compatibility.
  4. 4. Why are thousands of tiny Parquet files a problem?

    Per-file overhead dominates when files are small; larger files amortize it.

Summary

  • A Parquet file has row groups, column chunks, pages and a footer with statistics.
  • Readers fetch only needed columns and row groups; clustering makes statistics useful.
  • Dictionary/run-length encoding plus a codec (snappy or zstd) give small files.
  • Add nullable columns for safe evolution; avoid renames and type changes.

Next: Partitioning and Compaction