Skip to content
Tutorials Center
My learning

Topics

Lesson 17 of 39 ยท Data Formats and the Ecosystem

Schemas, Avro, Protobuf and JSON

Why Kafka topics need schemas, how a schema registry works, what backward, forward and full compatibility mean, and how to evolve a schema safely, with runnable stand-ins.

Kafka stores bytes. It has no idea whether a value is an order, a click or a photo. That freedom is a strength, but it means the contract between producers and consumers lives entirely outside the broker. If a producer renames a field from total to amount, every consumer reading total breaks at runtime, possibly days later, and nobody was warned.

A schema writes that contract down: field names, types and which fields are optional. A schema registry stores schemas centrally and checks that each new version is compatible with the old ones before any producer can use it. This lesson covers the common formats, the registry idea, the compatibility modes and how to evolve a schema without breaking anyone. It builds on Writing a Producer and Reading with a Consumer.

Real versus stand-in. The schemas and configuration below are real formats, shown read-only because the site cannot run a registry or a Kafka client. The TryIt blocks are plain JavaScript stand-ins: a tiny field-by-field compatibility checker and a toy wire format. Real Avro and Protobuf resolution has more rules (unions, aliases, nested records).

Why schemas

A contract Producers and consumers agree on fields.
Early failure Bad changes are rejected at publish time.
Smaller messages Binary formats skip repeating field names.
1 / 2

The three common formats

JSON is human-readable and every language can parse it. It repeats field names in every message, has no integer-versus-float distinction and has no built-in schema. You can add JSON Schema to describe and validate it.

Avro keeps data in a compact binary form without field names. The reader needs the writer's schema to decode, so Kafka setups pair Avro with a registry. Avro schemas are JSON documents:

JSON
{
  "type": "record",
  "name": "Order",
  "namespace": "shop.events",
  "fields": [
    { "name": "id", "type": "long" },
    { "name": "total", "type": "double" },
    { "name": "currency", "type": "string", "default": "USD" }
  ]
}

Protobuf uses a .proto file and numbers every field. The number, not the name, identifies the field in the encoded bytes:

PROTOBUF
syntax = "proto3";

message Order {
  int64 id = 1;
  double total = 2;
  string currency = 3;
}

Whichever you choose, serialization happens in the client through a serializer and a deserializer (see Writing a Producer). The broker still sees only bytes.

The schema registry

A schema registry is a separate service (Confluent Schema Registry is the best known; others such as Apicurio exist) that keeps every schema version. Its role in the flow:

  1. The producer's serializer sends the schema to the registry and gets back a numeric schema id. The registry has already checked the schema against earlier versions of the same subject (by default <topic>-value and <topic>-key).
  2. The serializer writes the message as: a magic byte, then the four-byte schema id, then the encoded data. This is the Confluent wire format; the message carries an id, not the whole schema.
  3. The consumer's deserializer reads the id, fetches (and caches) the schema from the registry, and decodes.
sequenceDiagram
    participant P as Producer
    participant R as Schema Registry
    participant K as Kafka topic
    participant C as Consumer
    P->>R: register schema, check compatibility
    R-->>P: schema id 7
    P->>K: magic byte + id 7 + data
    C->>K: poll
    K-->>C: magic byte + id 7 + data
    C->>R: get schema 7 (cached afterwards)
    R-->>C: schema

The registry is consulted on the first use of each schema id; later messages hit the client's cache.

A toy wire format

This stand-in builds the same shape with a JSON payload: byte 0, a four-byte big-endian id, then the data. A decoder reads the id first and looks the schema up in a tiny registry.

JS Magic byte, schema id, payload

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Compatibility modes

When you evolve a schema, two questions decide who must upgrade first:

  • Can a consumer using the new schema read data written with the old schema? That is backward compatibility.
  • Can a consumer still using the old schema read data written with the new schema? That is forward compatibility.
Mode Guarantee Upgrade order
BACKWARD New schema reads old data Consumers first, then producers
FORWARD Old schema reads new data Producers first, then consumers
FULL Both directions Any order
NONE No checks You are on your own

BACKWARD is the usual default in Confluent Schema Registry. Each of the first three also has a _TRANSITIVE variant that checks against all earlier versions, not just the latest. Pick the mode that matches how you deploy: if you can never coordinate consumers, choose FULL.

The practical rules for Avro, in short: adding a field with a default is safe in both directions, removing a field that had a default is safe in both directions, adding a field without a default breaks backward compatibility, and removing a field without a default breaks forward compatibility. Changing a field's type, or renaming it, breaks both unless you use aliases. Protobuf has its own rules, built around never reusing or renumbering a field number.

A compatibility checker you can run

The checker below treats a schema as a map of field name to {type, default?}. A reader can read a writer's data when every field the reader expects either exists with the same type or has a default.

JS Check schema evolution

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The output shows the rules in action: the default makes the first change fully compatible; the required new field is backward-incompatible; removing a required field is forward-incompatible; and the type change breaks both.

Headers and envelopes

Not everything belongs in the value. Kafka records can carry headers, small key-value pairs (key is a string, value is bytes) that travel with the record. Common uses are a trace id, the producing service, a content type or a schema version hint. Consumers and tools can filter or route on headers without decoding the value.

Another design choice is an envelope: a stable outer shape that wraps the changing payload, for example { "type": "OrderPlaced", "version": 2, "occurredAt": "...", "data": { ... } }. Envelopes make topics with several event types possible and give consumers one place to look for metadata. Keep the envelope itself small and change it rarely, since every consumer depends on it.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a Confluent-style message contain besides the encoded data?

    The serializer writes a magic byte, then the schema id, then the data. The consumer looks the id up in the registry and caches the schema.
  2. 2. You add a field with a default value to an Avro schema. Which compatibility holds?

    New readers use the default for old data, and old readers ignore the extra field in new data, so the change is fully compatible.
  3. 3. With BACKWARD compatibility, which upgrade order is safe?

    Backward means the new schema can read old data, so upgraded consumers handle both old and new messages before producers start writing the new version.
  4. 4. Where should a trace id usually go?

    Headers carry per-record metadata that tools can read without decoding the value.

Summary

  • Kafka stores bytes, so the data contract needs a schema; JSON, Avro and Protobuf are the usual choices.
  • A schema registry stores versions, checks compatibility on registration and lets messages carry a small schema id instead of the schema.
  • BACKWARD, FORWARD and FULL (plus _TRANSITIVE) decide who must upgrade first; adding fields with defaults is the safe way to evolve.
  • Headers and envelopes keep metadata out of the business payload.

Next, move data in and out of Kafka without writing code in Kafka Connect.