Skip to content
Tutorials Center
My learning

Topics

Lesson 25 of 47 ยท Data Formats and Storage

Parquet and ORC Internals

Open a Parquet file: row groups, column chunks, pages, dictionary and run-length encoding, min/max statistics and predicate pushdown, with runnable stand-ins.

Analytical queries rarely need every column. A report that sums amount by country touches two columns out of fifty, and in a row format like CSV or Avro you must still read all fifty. Apache Parquet and Apache ORC are columnar formats: they store each column's values together, so a query reads only the columns it needs, and values of the same type sit next to each other, which makes them compress very well.

This lesson opens the files. You will learn how Parquet nests row groups, column chunks and pages, how dictionary and run-length encoding shrink data, and how per-chunk statistics let an engine skip whole blocks of a file. ORC is built on the same ideas with different names; we compare them at the end.

Parquet and ORC libraries run on a JVM or in native code, so they cannot run on this site. The code and commands marked read-only are real. The runnable examples are Python stand-ins that implement the ideas (encoding, statistics, pruning) with plain lists, not the real binary format.

The layout of a Parquet file

A Parquet file is divided horizontally first, then vertically:

Inside one Parquet file

Row group A horizontal slice of rows, often 128 MB.
Column chunk All values of one column in one row group.
Page The unit of encoding and compression, about 1 MB.
Footer Schema, offsets and statistics.

Readers start at the footer, then jump straight to the chunks they need.

1 / 2

The footer is written last (the writer does not know the offsets until the data is done), so a reader opens the file, reads the last few bytes to find the footer's length, loads the footer and then knows where every column chunk is. This is why Parquet files must be complete before they are readable, and why they are not a good format for appending record by record.

A row group is the unit of parallel work and of skipping: a Spark or Hive task usually processes one or more row groups. Row groups are sized to fit a whole number into an HDFS block (the idea from HDFS Architecture): a common setup is 128 MB row groups in 128 MB or 256 MB blocks, so a task reads local data.

Inside a row group, each column has one column chunk, and each chunk is split into pages. Pages are where values are encoded and compressed. Nested data (structs, lists, maps) is flattened into columns using repetition and definition levels, a technique from Google's Dremel paper; you do not need the details to use it, only to know that nested fields become separate columns that can be read independently.

Terminal
# Read-only: inspect a real file with parquet-tools or the parquet-cli
parquet-tools meta sales.parquet
parquet-tools schema sales.parquet

Encodings: dictionary and run-length

Because a chunk holds one column of one type, simple tricks work well.

Dictionary encoding replaces each distinct value by a small integer. A country column with millions of rows and 40 distinct values becomes a 40-entry dictionary plus a list of small integers. Parquet builds a dictionary per column chunk and falls back to plain encoding if the dictionary grows too large.

Run-length encoding (RLE) replaces a run of equal values by (value, count). It shines when the column is sorted or has long repeats, which is also why sorting data on a low-cardinality column before writing makes files dramatically smaller. Parquet's real encoding is a hybrid of RLE and bit-packing, but the intuition is the same.

After encoding, each page is compressed with a general codec: Snappy (fast, the default in many tools), ZSTD (smaller, still fast) or Gzip (smallest, slow). Because the data is already encoded, columnar files compress far better than CSV with the same codec.

Python Dictionary and run-length encoding

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In arrival order every value differs from its neighbour, so RLE finds 24 runs for 24 values and gains nothing. After sorting there are only 3 runs: each country is stored once with a count. That is the whole reason data engineers sort or cluster by common filter columns before writing.

Statistics and predicate pushdown

For every column chunk (and often every page) the writer records statistics: the minimum value, the maximum value and the number of nulls. They are stored in the footer, so a reader sees them without reading any data.

Predicate pushdown uses them. Take the query:

SQL
SELECT SUM(amount) FROM sales WHERE amount > 900;

The engine hands the predicate amount > 900 down to the file reader. For each row group the reader checks the footer: if the chunk's max(amount) is 900 or less, no row in it can match, so the whole row group is skipped without being read or decompressed. The same logic works for equality (value < min or value > max means no match) and for ranges.

Pushdown works best when the data is clustered on the filtered column. If amount values are scattered randomly, every row group has a small min and a large max, and nothing can be skipped. If rows are written in order_date order, a date range filter prunes almost everything. Some formats add bloom filters per chunk for exact-match lookups such as WHERE user_id = 42, which min/max cannot answer.

Python Min/max pruning across row groups

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For the threshold 900 only the last group is read, five of the six groups are skipped. With the threshold 10 nothing can be skipped, because every group contains a value above 10. Pushdown helps exactly when the filter is selective and the data is clustered.

Reading from Spark (read-only)

Engines apply column pruning and pushdown automatically. You can see it in a Spark plan:

Python
df = spark.read.parquet("hdfs:///warehouse/sales/")
result = df.where("amount > 900").select("country", "amount")
result.explain()
# ... FileScan parquet [country,amount]
#     PushedFilters: [IsNotNull(amount), GreaterThan(amount,900)]

The [country,amount] list is the pruned columns, and PushedFilters is the predicate handed to the Parquet reader.

ORC and how it compares

ORC (Optimized Row Columnar) came out of the Hive project and follows the same plan with other names:

Parquet ORC
Row group Stripe (default about 64 MB)
Column chunk Column streams inside a stripe
Page Row group of 10,000 rows (index unit)
Footer File footer plus a postscript
Min/max statistics Min/max and counts, at file, stripe and 10,000-row level

ORC keeps lightweight indexes and optional bloom filters inside each stripe and was designed with Hive's ACID tables in mind. Parquet has the broader ecosystem: Spark, Impala, Trino, Arrow, pandas, DuckDB and cloud warehouses all read it well. A simple rule of thumb: use ORC when your world is Hive, Parquet nearly everywhere else. Both are splittable at row group or stripe boundaries, and both support schema evolution of the "add columns at the end" kind, though less flexibly than Avro.

Neither format suits writing one record at a time. The usual pipeline lands raw events as Avro or JSON, then converts them in batches to Parquet or ORC for analysis. How you organize those files into directories is the next topic, in Partitioning, Bucketing and File Layout.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In a Parquet file, what is a column chunk?

    A file is split into row groups, and each row group stores one column chunk per column. Chunks are further divided into pages.
  2. 2. Why does sorting data by a low-cardinality column before writing help?

    Sorted data has long runs of the same value (cheap with RLE) and tight min/max ranges per row group (good for skipping).
  3. 3. A row group has min(amount)=100 and max(amount)=400. Can it be skipped for WHERE amount > 500?

    Statistics in the footer tell the reader that nothing in the group can satisfy the predicate, so the group is never read.
  4. 4. Why can random, unsorted data make predicate pushdown useless?

    When values are scattered, each group contains both small and large values, so the min/max test cannot prove that a group has no match.

Summary

  • Parquet and ORC are columnar: queries read only the columns they need.
  • Parquet nests row groups, column chunks and pages, with the footer at the end; ORC uses stripes with the same idea.
  • Dictionary and run-length encoding, then Snappy or ZSTD, make columnar files small, especially after sorting.
  • Min/max statistics let engines skip whole row groups; this is predicate pushdown, and it needs clustered data.
  • Use ORC mainly in Hive-centric stacks and Parquet almost everywhere else; write in batches, not row by row.

Next: Partitioning, Bucketing and File Layout.