Skip to content
Tutorials Center
My learning

Topics

Lesson 23 of 47 ยท Data Formats and Storage

Row vs Column Formats

Compare text, CSV, JSON, SequenceFile and columnar layouts for big data: splittability, compression codecs and why column formats scan so much less data.

In a big data system, the file format matters as much as the engine. Pick the wrong one and every query reads ten times more data than needed, jobs cannot be split across machines, or you cannot read your own files two years from now. The format decides how many bytes leave the disk, how well data compresses and whether many workers can read one file at the same time.

This lesson starts with the formats you already know (text, CSV, JSON), introduces Hadoop's SequenceFile, and then explains the central idea of this chapter: row-oriented versus column-oriented layout. It also covers splittability and compression codecs. Later lessons go deeper into Avro, Parquet and ORC and partitioning.

Real Parquet or ORC files cannot be written on this site. The examples below are Python stand-ins that lay out the same data in rows and in columns and count how much would be read. They model the idea, not the actual file bytes.

Text formats: CSV, TSV and JSON Lines

Plain text (CSV, TSV, log files) is the default for a reason: any tool can read it, you can inspect it with head, and every language has a parser. The costs are real, though:

  • No types. Is 007 a number or a zip code? Every reader guesses.
  • No schema. Column order and meaning live in someone's head or a wiki page.
  • Bulky. Numbers are stored as characters (1234567 takes 7 bytes instead of 4).
  • Fragile parsing. Embedded commas, quotes and newlines need escaping rules.

JSON Lines (one JSON object per line) adds self-describing field names and nested data, which is good for logs and event streams, but every row repeats every field name, which makes it even bulkier. Both formats are splittable when the file is uncompressed: a reader can start in the middle and skip to the next newline.

SequenceFile: Hadoop's own container

A SequenceFile is a binary file of key-value records, one of Hadoop's earliest formats. It stores the types of key and value in a header, supports compression at record or block level and, importantly, contains periodic sync markers. A reader starting at an arbitrary byte can scan forward to the next sync marker, so SequenceFiles stay splittable even when compressed.

They are used for intermediate data and as a container for many small files (key = file name, value = file contents), which helps with the small-files problem from NameNode HA, Federation and Operations. Today they have mostly been replaced by Avro, Parquet and ORC, but you will still meet them in older pipelines.

Row-oriented vs column-oriented

Consider a table of one million page views with columns user_id, url, country, duration_ms, user_agent, referrer.

Same data, two layouts

Row format All fields of record 1, then record 2, and so on.
Column format All user_ids, then all urls, then all countries.
1 / 2

In a row format (CSV, JSON, Avro, SequenceFile), a record's fields are stored together. To compute the average duration_ms, you must read every byte of every row, including the six columns you do not care about. In a column format (Parquet, ORC), each column is stored contiguously, so the same query reads only the duration_ms bytes. Analytical queries typically touch 3 of 100 columns, so the saving is large.

Columnar layout also compresses better because neighbouring values are alike: a country column is mostly a few repeated strings, which dictionary and run-length encoding shrink dramatically. The price is that reconstructing one full record means reading from many places, and writing is more work because values must be buffered and regrouped by column.

Python Row vs column scan cost

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real columnar files are smaller still, thanks to encoding, and they keep statistics so entire sections can be skipped. That is the topic of Parquet and ORC Internals.

Compression and splittability

Big data is stored compressed: it saves disk, and it often makes jobs faster because reading fewer bytes matters more than the CPU spent decompressing. But the choice of codec interacts with parallelism.

A MapReduce or Spark job splits a large file into pieces so many workers can read it at once (see The MapReduce Programming Model). A worker must be able to start reading in the middle of a file. A format is splittable if that is possible.

Codec Ratio Speed Splittable alone? Typical use
gzip good medium No archives, small files
bzip2 very good slow Yes rarely, cold archives
LZO fair fast Only with an index older clusters
Snappy fair very fast No default inside Parquet/ORC/Avro
LZ4 fair very fast No shuffle data, streaming
Zstandard (zstd) very good fast No modern default for new data

A gzip file of 10 GB is a single stream: only one worker can read it, no matter how big the cluster. The usual solution is not to use a splittable codec but to put compression inside a splittable container. Parquet, ORC, Avro and SequenceFile all compress blocks internally (each block or row group independently), so the container stays splittable while using a fast codec like Snappy or zstd. Compressing the whole file with gzip is the thing to avoid for large inputs.

Python Why one gzip file limits parallelism

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The block version is slightly larger in total because each block is compressed without the context of the others. That small loss buys parallelism.

Choosing a format

Situation Good choice
Quick exchange, small data, human inspection CSV or JSON Lines
Streaming records, schema evolution, row-by-row access Avro (next lesson)
Analytical queries over many rows and few columns Parquet or ORC
Intermediate Hadoop key-value data, old pipelines SequenceFile
Archived raw logs text compressed with gzip, in many small-to-medium files

A common architecture lands raw data as JSON or CSV (the raw zone), then converts it to Parquet or ORC (the curated zone) for fast querying; see Data Lakes, Warehouses and Lakehouses.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A query needs 2 of 40 columns from a huge table. Which layout reads the least data?

    A columnar file stores each column contiguously, so the query reads only the two columns it needs.
  2. 2. What does it mean that a file format is splittable?

    Splittability lets many workers read different parts of one file at the same time.
  3. 3. Why is a single 10 GB gzip text file a problem for a cluster?

    gzip cannot start decompressing in the middle, so the whole file goes to one worker.
  4. 4. How do Parquet and ORC stay splittable while using Snappy?

    Compression is applied to each internal chunk, and the container's metadata marks where chunks begin.
  5. 5. Which workload favours a row-oriented format?

    Rows keep a record's fields together, which helps writes and full-record lookups.

Summary

  • Text formats are universal but untyped, bulky and schema-less; JSON Lines is self-describing but repeats field names.
  • SequenceFile is a binary key-value container with sync markers that stays splittable when compressed.
  • Row formats keep a record together (good for writes and full records); column formats keep a column together (good for analytics, better compression).
  • Splittability decides how many workers can read one file; whole-file gzip is not splittable.
  • Use fast codecs (Snappy, zstd) inside splittable containers such as Parquet, ORC and Avro.

Next: Avro and Schema Evolution.