Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 47 ยท Data Formats and Storage

Avro and Schema Evolution

Learn how Avro stores data with a schema, how its compact binary encoding works, and which schema changes are safe, with a runnable encoder and resolver.

In Row vs Column Formats you saw that text files like CSV and JSON are easy to read but wasteful and vague: they repeat field names, they carry no types, and nothing stops a producer from changing a column and breaking every consumer. Apache Avro fixes this for row-oriented data. Every Avro file carries its schema, the data is stored in a tight binary form without field names, and there are clear rules for changing the schema over time.

Avro is the usual choice for records that flow through a pipeline (Kafka messages, ingestion files, intermediate data) rather than for analytical scans. In this lesson you will read a schema, see exactly how a record becomes bytes, learn which changes are safe, and compare Avro with Protobuf.

Avro libraries (Java, Python fastavro, and others) cannot run on this site. The schema and library code below is read-only. The runnable examples are small Python stand-ins that follow the real Avro encoding rules for a few types, so you can see the idea working. They are not the real library and support only a subset of Avro.

What an Avro schema looks like

An Avro schema is plain JSON. A record has a name and a list of fields, each with a type:

JSON
{
  "type": "record",
  "name": "Click",
  "namespace": "com.example.web",
  "fields": [
    {"name": "user_id", "type": "long"},
    {"name": "url",     "type": "string"},
    {"name": "referrer", "type": ["null", "string"], "default": null},
    {"name": "tags",    "type": {"type": "array", "items": "string"}, "default": []}
  ]
}

Avro types

Primitives null, boolean, int, long, float, double, bytes, string.
Complex record, enum, array, map, fixed.
Union A list of types, written as a JSON array.

An optional field is a union of null and another type.

1 / 2

An optional field is written as a union, ["null", "string"]. The first type in a union is the type of its default value, so "default": null needs "null" first.

Avro files (.avro, the object container file) store the schema once in the header, then the records in blocks separated by a random sync marker. The sync marker is what makes Avro files splittable: a MapReduce or Spark task can start at any byte offset, scan forward to the next marker and begin reading whole blocks. Blocks can be compressed with Snappy, Deflate and other codecs without losing splittability.

How a record becomes bytes

The binary encoding is surprisingly small because it stores only values, in the exact order the schema lists the fields. There are no field names, no tags and no delimiters. The reader needs the schema to know what the bytes mean.

The rules for the common types:

  • int and long use zigzag encoding (so small negative numbers stay small), then a variable-length integer: 7 bits per byte, the high bit says "more bytes follow".
  • string and bytes are a long length followed by the bytes (UTF-8 for strings).
  • boolean is one byte, 0 or 1.
  • A union is a long giving the index of the chosen branch, then the value.
  • An array is one or more blocks: a long count, then that many items, ending with a block of count 0.
  • A record is just its fields one after another.

Zigzag maps signed to unsigned like this: 0 becomes 0, -1 becomes 1, 1 becomes 2, -2 becomes 3, and so on. The formula is (n << 1) ^ (n >> 63).

Python A tiny Avro-style encoder

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the output: the long 42 is zigzagged to 84, which is hex 54, one byte. The string /home is a length byte (0a, which is 5 zigzagged) plus five characters. The union index 0 (null) costs a single byte. The whole record fits in 8 bytes, while the same record as JSON text needs 49 bytes, because JSON repeats "user_id", "url" and "referrer" in every record.

The price is that the bytes are meaningless without the schema. That sounds like a weakness, but it leads straight to the main feature.

Reader schema and writer schema

When Avro reads data it uses two schemas:

  • The writer schema is the schema the data was written with. It is stored in the file header (or looked up in a schema registry for Kafka messages).
  • The reader schema is the schema your code wants to see now.

They do not have to be identical. Avro resolves the differences field by field, by name:

  1. A field in both schemas is read normally (with simple type promotions, such as int to long).
  2. A field only in the writer schema is skipped and ignored.
  3. A field only in the reader schema gets its default value. If it has no default, resolution fails.

Because fields are matched by name, you may reorder fields freely. Renaming needs an aliases entry in the reader schema.

Safe and unsafe changes

These two words matter. A change is backward compatible if a new reader can read old data. It is forward compatible if an old reader can read new data. Full compatibility means both.

Change Backward (new reader, old data) Forward (old reader, new data)
Add a field with a default safe safe (old reader skips it)
Add a field without a default breaks safe
Remove a field with a default safe safe (old reader uses its default)
Remove a field without a default safe breaks
Promote int to long safe breaks
Rename a field (no alias) breaks breaks

The practical rule: give every field you add a default, and only remove fields that have one. Then your changes stay fully compatible, and producers and consumers can upgrade in any order.

Python Writer and reader schema resolution

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first line shows backward compatibility: old data gains country: unknown. The second shows forward compatibility: the old reader simply never sees country. The third is the mistake to avoid: adding device with no default makes old data unreadable by the new schema.

Schema registries

Putting the full schema in every Kafka message would waste space. Instead, a schema registry stores schemas and gives each a small integer ID. Each message starts with that ID (in Confluent's format, one magic byte plus a 4-byte ID), followed by the Avro-encoded body. The reader fetches the writer schema by ID, caches it and resolves it against its own. The registry can also enforce a compatibility mode (backward, forward, full or none) and reject a new schema version that breaks it, before any bad data is written.

Avro versus Protobuf

Protocol Buffers (Protobuf) solves the same problem differently.

Avro Protobuf
Schema in data Needed to read; stored in file header or registry Not stored; compiled into code
Field identity Name (order in the schema) Field number (tag in each field)
Missing fields Need explicit defaults Always optional, defaults by type
Code generation Optional (generic readers work) Normally required
Typical home Hadoop files, Kafka, Spark RPC (gRPC), service messages

Protobuf writes a tag number with each field, so a reader can skip fields it does not know without any schema. Avro's records are smaller and need no tags, but both sides must have a schema to decode. In the Hadoop world, Avro won because files carry their own schema and splittable blocks, which suits batch jobs.

Remember what Avro is not: it is a row format. To read three columns out of fifty, you still read every byte of every record. For that, use the column formats in Parquet and ORC Internals.

Real usage, read-only, with the Python fastavro library:

Python
import fastavro

schema = fastavro.parse_schema(click_schema)
with open("clicks.avro", "wb") as out:
    fastavro.writer(out, schema, records, codec="snappy")

with open("clicks.avro", "rb") as src:
    for record in fastavro.reader(src, reader_schema=new_schema):
        print(record)

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is an Avro record smaller than the same record as JSON?

    The schema defines field order and types, so the binary data holds only values. Field names are never repeated per record.
  2. 2. You add a new field to a schema and give it a default value. Which statement is true?

    A new reader fills in the default for old data, and an old reader skips the field it does not know. That makes the change fully compatible.
  3. 3. What happens when the reader schema has a field that the writer schema lacks and that has no default?

    Without a default there is no value to supply, so resolution fails. That is why new fields should always have defaults.
  4. 4. How does Protobuf identify a field in the data, compared with Avro?

    Protobuf writes a field number with each value. Avro writes values in schema order and matches reader and writer fields by name.

Summary

  • An Avro schema is JSON; optional fields are unions with null.
  • Records are encoded as values only, in schema order: zigzag varints, length-prefixed strings, union indexes.
  • Files store the schema in the header and use sync markers, so they are splittable and compressible.
  • A reader resolves writer and reader schemas by field name: extra writer fields are skipped, missing reader fields use defaults.
  • Add fields with defaults and remove only fields that have defaults to stay fully compatible.
  • Protobuf uses numbered tags instead of names; Avro is a row format, so analytics prefer columnar files.

Next: Parquet and ORC Internals.