Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 36 ยท File Formats and Compression

ORC and Parquet in Hive

Learn how ORC and Parquet store data, how stripes, row indexes and bloom filters let Hive skip data, and how to create, convert and compact ORC tables.

For analytics, two columnar formats dominate Hive: ORC (Optimized Row Columnar) and Parquet. Both store data column by column in compressed blocks, carry their schema, and keep statistics that let a query skip data it does not need. Choosing one of them, and laying out the data well, is often the largest single performance win in a Hive warehouse.

How ORC is organized

An ORC file is a sequence of stripes (by default about 64 MB each) followed by a footer.

flowchart TD
    F["ORC file"] --> S1["Stripe 1"]
    F --> S2["Stripe 2"]
    F --> FT["File footer: schema, stripe list, file-level statistics"]
    S1 --> IX["Index data: min, max, counts per 10,000 rows"]
    S1 --> RD["Row data: one compressed stream per column"]
    S1 --> SF["Stripe footer: stream locations"]

A reader looks at the footer first, then only reads the stripes and column streams it needs.

Three levels of statistics let Hive skip work:

  1. File and stripe statistics (min, max, null count per column). A stripe whose maximum is below your filter value is skipped without being read.
  2. Row index, one entry per 10,000 rows (orc.row.index.stride). Inside a stripe Hive skips groups of 10,000 rows whose min and max cannot satisfy the filter.
  3. Bloom filters, optional. They answer "could this value be here?" for equality filters on high-cardinality columns such as ids.

For skipping to work, the filter must be pushed down into the file reader (hive.optimize.index.filter=true, on by default in recent versions) and the data should be sorted on the filter column, so that each stripe covers a narrow value range. Random data has wide min/max ranges everywhere and nothing gets skipped.

Creating ORC tables

SQL
CREATE TABLE sales_orc (
  sale_id     BIGINT,
  customer_id BIGINT,
  amount      DECIMAL(10,2),
  sale_ts     TIMESTAMP
)
STORED AS ORC
TBLPROPERTIES (
  'orc.compress'            = 'ZSTD',               -- NONE, ZLIB (default), SNAPPY, LZO, LZ4, ZSTD
  'orc.bloom.filter.columns' = 'customer_id',
  'orc.bloom.filter.fpp'     = '0.05',
  'orc.stripe.size'          = '67108864'
);

Converting an existing table is a CREATE TABLE ... AS SELECT or an INSERT ... SELECT:

SQL
CREATE TABLE sales_orc STORED AS ORC AS
SELECT * FROM sales_text;

-- Sorting while loading gives tight min/max ranges
INSERT OVERWRITE TABLE sales_orc
SELECT * FROM sales_text SORT BY customer_id;

ORC is also required for full ACID tables (see ACID transactions) and it is the format with the deepest Hive integration: vectorized readers, LLAP cache and the best compression.

Small files and CONCATENATE

Frequent small inserts produce many small ORC files, which slows every query because each file costs a task. For ORC tables you can merge the files of a table or partition without rewriting through SQL:

SQL
ALTER TABLE sales_orc CONCATENATE;
ALTER TABLE sales_orc PARTITION (sale_date = '2026-01-15') CONCATENATE;

Concatenation merges at stripe level, so stripes are not re-sorted. For ACID tables use compaction instead (next chapter).

Parquet

Parquet is the columnar format of the wider ecosystem: Spark, Trino, Impala, Arrow, pandas and most cloud tools read it natively. A Parquet file has row groups, each holding column chunks split into pages, plus a footer with the schema and min/max statistics. Nested data (structs, arrays, maps) is stored using the Dremel record-shredding technique with repetition and definition levels.

SQL
CREATE TABLE sales_pq (
  sale_id BIGINT, customer_id BIGINT, amount DECIMAL(10,2)
)
STORED AS PARQUET
TBLPROPERTIES ('parquet.compression' = 'SNAPPY');   -- UNCOMPRESSED, GZIP, ZSTD, LZ4 ...

Parquet also supports predicate pushdown through row-group statistics and, in newer versions, bloom filters, but Hive's support is less deep than for ORC (no ACID, weaker vectorization and cache).

ORC or Parquet

Question ORC Parquet
ACID tables (UPDATE, DELETE, MERGE) Yes Insert-only transactional tables only
Best fit Hive and LLAP centered warehouses Shared by Spark, Trino, Impala, cloud tools
Built-in indexes Row index, bloom filters Row-group statistics, optional bloom filters
Hive vectorization and cache Strongest Supported
Typical choice Hive-first Multi-engine lake

If several engines read the same files, Parquet is the safe neutral choice. If Hive is your main engine and you need updates, use ORC. Newer table formats like Iceberg (STORED BY ICEBERG in Hive 4) can sit on top of either file format and add snapshots and time travel.

A stand-in: min/max skipping

The editors here are not Hive. This JavaScript model shows why sorted data lets a reader skip stripes using only min and max statistics.

JS Stripe skipping with min/max

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Sorted data needs 1 stripe of 10; scattered data needs all of them. In Hive the same effect comes from SORT BY, from CLUSTERED BY ... SORTED BY buckets and from partitioning.

The same thinking in SQL

You can feel the benefit of narrow ranges with a simple range query too. This SQLite stand-in compares a range filter before and after sorting the access path with an index:

SQL Range filter plan (SQLite stand-in) ยท database: shop

Press Run (or Ctrl + Enter) to see the result here.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What do ORC row indexes store?

    The row index lets a reader skip row groups whose value range cannot match the filter.
  2. 2. Which format is required for full ACID tables in Hive?

    Updates and deletes on managed tables use ORC delta files; other formats only support insert-only transactional tables.
  3. 3. Why do queries skip more data when the table is sorted by the filter column?

    Min/max pruning only works when values are clustered; scattered values give every stripe a wide range.
  4. 4. You need Spark, Trino and Hive to share the same files. Which format is the most neutral choice?

    Parquet is read natively by almost every engine in the ecosystem.

Summary

  • ORC and Parquet are columnar, compressed and self-describing; use them instead of text for analytics.
  • ORC has stripes, row indexes and bloom filters, and is required for full ACID tables.
  • Sorted or clustered data makes min/max skipping effective; random data defeats it.
  • CREATE TABLE ... AS SELECT/INSERT ... SELECT converts formats; ALTER TABLE ... CONCATENATE merges small ORC files.
  • Pick Parquet for multi-engine lakes, ORC for Hive-centered warehouses.

Next: Compression Codecs and SerDes