Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 39 ยท Testing and Versioning

Protobuf Schema Evolution and Compatibility

Change protobuf schemas without breaking clients: safe and unsafe edits, field numbers, reserved, wire format compatibility and API versioning, with a runnable wire-format demo in Go.

Servers and clients are almost never deployed at the same moment. A mobile app from last year still talks to today's backend; one microservice upgrades on Tuesday, its caller on Friday. If a schema change breaks the older side, you get outages, or worse, silently wrong data. Protobuf was designed for this: if you follow a few rules, old and new code can talk to each other indefinitely.

The key to the rules is understanding what is actually on the wire. This lesson shows the wire format with a small program you can run, then lists the safe and unsafe changes and how to version a whole API.

protoc and the generated code cannot run on this site. The .proto snippets are read-only, and the wire-format programs implement the real protobuf encoding (varints, tags, length-delimited fields) in plain Go, so what they print is what real protobuf libraries would put on the wire.

Field numbers are the identity

On the wire a protobuf message is a sequence of (tag, value) pairs. The tag is (field_number << 3) | wire_type. Field names never appear on the wire, only numbers. Wire types that matter most: 0 varint (integers, bools, enums), 2 length-delimited (strings, bytes, nested messages, packed repeated fields), 1 and 5 fixed 64/32-bit.

That has two consequences:

  1. You can rename a field freely (on the wire); you cannot change its number or its wire type.
  2. A reader that finds a field number it does not know skips it (using the wire type to know how many bytes), and keeps it as an unknown field in modern libraries. That is why adding fields is safe.

Runnable: look at the bytes

Go Protobuf wire format by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

The output starts with 0a 03 41 64 61: tag 0x0a is field 1 with wire type 2, then the length 3, then the bytes of "Ada". Then 10 24: tag 0x10 is field 2 with wire type 0, and 0x24 is 36. Seven bytes in total, with no names and no whitespace.

Safe changes

These are compatible (old and new code keep working):

  • Add a new field with a new, never-used number. Old readers skip it; new readers see the default value when reading old data.
  • Remove a field, as long as you reserve its number and name (see below).
  • Rename a field on the wire (but not for the JSON mapping or for generated code consumers, where names matter; see REST transcoding).
  • Add an RPC to a service or an enum value (old clients may get an unknown enum number; handle it).
  • Change between compatible integer types like int32, uint32, int64, uint64 and bool (values may be truncated), or sint32/sint64 with each other.

Unsafe changes

  • Change a field number. It is a different field now.
  • Reuse a field number for something else (even after deleting the old one). Old data and old clients will interpret it as the old meaning.
  • Change a field's wire type, for example int32 to string.
  • Change optional/repeated semantics carelessly, or move a field into or out of a oneof (partly unsafe).
  • Rename a package, service or message in a way that changes the fully qualified name; it appears in the HTTP/2 path /users.v1.UserService/GetUser and in Any type URLs.
  • Change the meaning of an existing field without changing its name or number. Compatible bytes can still mean something new.

Runnable: why reusing a number hurts

Go Skipping unknown fields, and reuse gone wrong

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

reserved: protect removed fields

When you delete a field, reserve its number and its name so nobody reuses them:

PROTO
message User {
  reserved 2, 15, 9 to 11;
  reserved "age", "legacy_flag";

  string id = 1;
  string name = 3;
  string email = 4;
}

The compiler rejects any later attempt to define a field with a reserved number or name. For enums use the same syntax. A cheaper alternative to deleting is to leave the field in place and mark it [deprecated = true], then remove it after all clients have moved.

Defaults and presence

In proto3, a scalar field set to its default (0, empty string, false) is not sent, and a reader cannot tell "zero" from "missing". That matters for evolution: if you add int32 retries = 5; old clients never set it, so the server sees 0. Design for it: treat the default as "not specified", or use optional int32 retries = 5; which tracks presence explicitly, or wrap in a message type (google.protobuf.Int32Value). Similarly, always make the zero value of an enum mean UNSPECIFIED:

PROTO
enum Status {
  STATUS_UNSPECIFIED = 0;
  STATUS_ACTIVE = 1;
  STATUS_SUSPENDED = 2;
}

A new enum value sent to an old client appears as its number; old code must have a default branch.

Versioning the whole API

Compatible changes need no new version. For breaking changes, create a new package version: users.v1 and users.v2 can be served side by side, with different gRPC paths, so both generations of clients keep working. A typical lifecycle:

  1. Release v2 in parallel with v1; v1 handlers can convert to the new internals.
  2. Announce deprecation; measure traffic per version with the method label from your metrics.
  3. Remove v1 when traffic is gone.

Within a version follow the rules above; tooling to enforce them is the subject of linting and breaking-change detection.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What identifies a field on the protobuf wire?

    The tag is (field number << 3) | wire type. Names never appear on the wire.
  2. 2. Which change is NOT backward compatible?

    Old data and clients still interpret that number with its old meaning, causing wrong data or parse failures.
  3. 3. What does reserved do?

    The compiler rejects new definitions that use reserved numbers or names.
  4. 4. Why should the zero value of an enum be named UNSPECIFIED?

    Defaults are not sent, so a missing value decodes as 0. UNSPECIFIED keeps that from looking like a real choice.

Summary

  • Field numbers and wire types define compatibility; names do not appear on the wire.
  • Adding fields, renaming, adding RPCs and enum values are safe; changing or reusing numbers and changing wire types are not.
  • reserved numbers and names when removing fields; prefer deprecation first.
  • Remember proto3 defaults are not sent: design for "unset", use optional when presence matters.
  • Make breaking changes in a new package version (v2) served alongside the old one.

Next: Linting and Breaking-Change Detection with Buf