Skip to content
Tutorials Center
My learning

Topics

Lesson 25 of 36 ยท File Formats and Compression

Compression Codecs and SerDes

Choose compression codecs for Hive output and intermediate data, understand splittability, and learn what a SerDe does with LazySimpleSerDe, OpenCSV, JSON and Regex.

Compression makes Hive jobs faster in two ways. Smaller files mean less disk to read and less data to shuffle across the network. The price is CPU time to compress and decompress, plus a constraint that matters in Hadoop: whether a compressed file can be split so many tasks can read it in parallel.

This lesson covers codecs, how to switch compression on, and the SerDe layer that turns bytes into typed columns.

Codecs compared

Codec Ratio Speed Splittable (plain text) Notes
Snappy moderate very fast no (yes inside ORC, Parquet, SequenceFile, Avro) Default choice for speed
LZ4 moderate very fast no (yes in containers) Similar to Snappy
Zstandard (ZSTD) high fast no (yes in containers) Great balance, modern default for new tables
Gzip / Deflate / ZLIB high slower no Good ratio, common for archives
Bzip2 highest slowest yes Rare; very slow
LZO moderate very fast only with an index Needs extra libraries; legacy

The important column is splittable. A single gzip-compressed text file of 10 GB is read by one task, however large the cluster. The same data in ORC or Parquet with Snappy is split internally into many stripes or row groups, each compressed on its own, so the file is still read in parallel. That is the strongest argument for using a container format and compressing inside it, instead of compressing text files.

flowchart LR
    A["10 GB .csv.gz"] --> B["One task reads it all"]
    C["10 GB ORC, Snappy inside"] --> D["Many tasks, one per stripe"]

Compress inside a splittable container, not around a text file.

Compressing table data

For formats with internal compression, set it in table properties, as shown in the ORC and Parquet lesson (orc.compress, parquet.compression). For text and SequenceFile output, use session settings:

SQL
SET hive.exec.compress.output = true;
SET mapreduce.output.fileoutputformat.compress = true;
SET mapreduce.output.fileoutputformat.compress.codec = org.apache.hadoop.io.compress.SnappyCodec;

INSERT OVERWRITE TABLE logs_compressed
SELECT * FROM logs;

You rarely need to configure input: Hive recognizes .gz, .bz2 and .snappy extensions and decompresses automatically.

Intermediate data

Between stages, Hive writes temporary data (shuffle output on the way to reducers). Compressing this often helps, and a fast codec is ideal:

SQL
SET hive.exec.compress.intermediate = true;
SET hive.intermediate.compression.codec = org.apache.hadoop.io.compress.SnappyCodec;
SET hive.intermediate.compression.type = BLOCK;

On Tez the equivalent shuffle compression is handled by Tez settings (tez.runtime.compress, tez.runtime.compress.codec), usually on by default.

Choosing

  • New analytic table: ORC or Parquet with ZSTD or Snappy. Snappy if CPU is scarce; ZSTD if storage and I/O dominate.
  • Archive, rarely read: a higher ratio (ZSTD high level or Gzip).
  • Raw text you must keep as text: prefer bzip2 if splitting matters, or split files yourself to a few hundred MB each and gzip them.
  • Shuffle data: Snappy, LZ4 or ZSTD.

Compression inside a columnar format

Columnar formats compress better than row formats because similar values sit next to each other, and before any codec runs they apply lightweight encodings: run-length encoding (repeated values), dictionary encoding (replace repeated strings by small integers) and delta encoding (store differences between neighbours). The stand-in below shows two of them in JavaScript. It is a model, not the real ORC code.

JS Dictionary and run-length encoding

Press Run (or Ctrl + Enter) to see the console output here.

No output.

If the column were sorted, the runs would be longer still, which is another reason to sort data on load.

SerDes

A SerDe (serializer/deserializer) is the code that turns a file's bytes into column values on read, and column values into bytes on write. The file format decides how records are found; the SerDe decides how a record is parsed. You pick one with ROW FORMAT SERDE:

SQL
CREATE TABLE web_events (...)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
WITH SERDEPROPERTIES ('field.delim' = '\t')
STORED AS TEXTFILE;

Built-in SerDes you will use:

SerDe For Notes
LazySimpleSerDe delimited text default for ROW FORMAT DELIMITED; parses columns lazily, only when accessed
OpenCSVSerde quoted CSV every column is STRING; properties separatorChar, quoteChar, escapeChar
JsonSerDe one JSON object per line org.apache.hive.hcatalog.data.JsonSerDe; keys match column names
RegexSerDe log lines input.regex with one capture group per column
AvroSerDe, OrcSerde, ParquetHiveSerDe binary formats chosen by STORED AS

Parsing logs with RegexSerDe

SQL
CREATE EXTERNAL TABLE access_log (
  ip STRING, ts STRING, request STRING, status STRING, bytes STRING
)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe'
WITH SERDEPROPERTIES (
  'input.regex' = '^(\\S+) \\S+ \\S+ \\[([^\\]]+)\\] "([^"]*)" (\\d{3}) (\\S+)$'
)
LOCATION '/data/raw/access_log/';

Rows that do not match the pattern come back as NULL columns, so check a sample. Like OpenCSV, RegexSerDe produces strings, so cast in a view or a conversion step.

Cast after reading text

Because text SerDes return strings, a common pattern is a thin typed view over the raw table. The SQL stand-in below does the same in SQLite:

SQL Typing raw text columns

Press Run (or Ctrl + Enter) to see the result here.

Note the third row: 'oops' casts to 0 in SQLite. Hive is stricter in a helpful way, giving NULL for a failed CAST, so bad values are easy to count with WHERE CAST(amount AS DECIMAL(10,2)) IS NULL AND amount IS NOT NULL.

Custom SerDes

If you have a proprietary binary format you can write your own SerDe in Java, package it as a JAR and use ADD JAR plus ROW FORMAT SERDE 'your.Class'. The UDF lessons show the same packaging steps.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is a single 10 GB gzip text file read by one task?

    A gzip stream must be decompressed from its start, so it cannot be divided among tasks.
  2. 2. How is ORC with Snappy still read in parallel?

    Compression is applied per stream inside the container, so the container stays splittable.
  3. 3. What type do all columns have when read by OpenCSVSerde?

    OpenCSVSerde returns strings, so cast to numbers or dates in a view or conversion step.
  4. 4. What does hive.exec.compress.intermediate affect?

    It compresses data written between stages, saving I/O and network traffic at a small CPU cost.

Summary

  • Compression saves I/O at a CPU cost; splittability decides whether a compressed file can be read in parallel.
  • Compress inside ORC or Parquet (Snappy or ZSTD) instead of gzipping text files.
  • Use hive.exec.compress.output for text output and hive.exec.compress.intermediate for stage data.
  • A SerDe parses bytes into columns: LazySimpleSerDe, OpenCSVSerde, JsonSerDe, RegexSerDe and the binary-format SerDes.
  • Text SerDes return strings; add a typed view with casts.

Next: Built-in Functions and UDF Basics