Skip to content
Tutorials Center
My learning

Topics

Lesson 7 of 39 ยท Protobuf Language

Field Numbers and the Protobuf Wire Format

Understand how protobuf encodes messages: field numbers, wire types, varints and ZigZag, with runnable Go code that encodes and decodes real protobuf bytes.

Why does protobuf need a number on every field? Because the binary message does not contain names. It contains a stream of tag, value pairs, and the tag holds the field number. The number is the field's permanent identity: rename the field, and old data still decodes; reuse the number for something else, and data is corrupted.

This lesson explains the real protobuf wire format. The Go programs here implement it by hand using only the standard library, so what you see is genuinely how protobuf encodes data. (They handle only a few types; the real libraries handle them all.)

The tag byte

Every field on the wire starts with a tag, a varint computed as:

Text
tag = (field_number << 3) | wire_type

There are a handful of wire types. The ones you will meet constantly:

Wire type Value Used for
VARINT 0 int32, int64, uint*, sint*, bool, enums
I64 1 fixed64, sfixed64, double
LEN 2 string, bytes, embedded messages, packed repeated fields
I32 5 fixed32, sfixed32, float

(Types 3 and 4 were the deprecated "group" start and end markers.)

For field 1 with wire type 0, the tag is (1 << 3) | 0 = 8, which is the byte 0x08. For field 2 with wire type 2, it is (2 << 3) | 2 = 18 = 0x12.

Varints: small numbers take few bytes

A varint stores an integer in 7-bit groups, least significant group first. The high bit of every byte is a "more bytes follow" flag.

  • 1 is 0x01 (one byte).
  • 150 is binary 1001 0110. Split into 7-bit groups: 0010110 and 0000001. Emit low group first, with the continue bit set on all but the last: 0x96 0x01.

So message { int32 a = 1; } with a = 150 encodes to 08 96 01. That famous example comes straight from the protobuf documentation, and our code will reproduce it.

Stand-in 1: encode a varint and a message by hand

Go Stand-in: hand-written protobuf encoder

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

You can verify the first line by hand: tag 08, then varint 96 01. The second line begins with 0a 03 (field 1, length 3) followed by the bytes of "Ada".

ZigZag: negative numbers

A negative int32 is encoded as a very large unsigned 64-bit value, which takes 10 bytes. sint32 and sint64 use ZigZag: map signed integers to unsigned ones so small magnitudes stay small.

Text
0 -> 0, -1 -> 1, 1 -> 2, -2 -> 3, 2 -> 4, ...
zigzag(n) = (n << 1) ^ (n >> 63)
Go Stand-in: ZigZag versus plain varint

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Plain -1 takes 10 bytes (its 64-bit two's complement is huge), while ZigZag makes it 1.

Length-delimited fields

Strings, bytes and embedded messages use wire type 2: tag, then a varint length, then the payload. A nested message is simply another encoded message inside that payload, which is why you can add fields to nested types without breaking parents.

flowchart LR
  T["tag: field 2, type LEN (0x12)"] --> L["length (varint)"] --> P["payload: encoded inner message"]

Embedded messages are just length-delimited blobs. The reader recursively decodes the payload.

Decoding and unknown fields

A decoder reads a tag, extracts the field number and wire type, and then knows how many bytes to consume, even for a field it has never heard of. That is how old software ignores new fields and keeps working.

Stand-in 2: a decoder that skips unknown fields

The decoder below knows only fields 1 (name) and 2 (id). The message it receives also has a field 3 that a newer sender added. The decoder skips it cleanly.

Go Stand-in: forward-compatible decoding

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Rules for field numbers

  • Numbers 1 to 15 take one tag byte; 16 to 2047 take two. Give the numbers 1 to 15 to your most frequently used fields.
  • The legal range is 1 to 536,870,911, excluding 19000 to 19999 (reserved for the implementation).
  • Never change or reuse a number once data using it exists anywhere (files, queues, other services). To retire a field, use reserved:
PROTO
message Product {
  reserved 6, 9 to 11;
  reserved "old_price", "legacy_code";
  int64 id = 1;
  string name = 2;
}

The compiler now refuses to let anyone reuse those numbers or names.

  • Renaming a field is safe on the wire (names are not encoded), but it breaks JSON mapping and generated code. Changing a type is only safe among compatible types, for example int32 to int64.

Field order

Protobuf does not guarantee canonical encoding. Fields are usually written in ascending number order, but readers must accept any order. If the same non-repeated field appears twice, the last value wins. Do not use serialized bytes as hash keys unless you control the serializer.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. How is the tag for field number 3 with wire type 2 computed?

    tag = (field_number << 3) | wire_type, so (3<<3)|2 = 26 (0x1a).
  2. 2. What does a varint's high bit mean?

    Each byte holds 7 bits of data; the high bit is set on all but the last byte.
  3. 3. Why is it dangerous to reuse a field number?

    The number is the wire identity. Use reserved to retire numbers permanently.
  4. 4. How does an old decoder handle a field number it does not recognize?

    Wire types tell the decoder how many bytes to consume, enabling forward compatibility.
  5. 5. Which fields should get numbers 1 to 15?

    Numbers up to 15 encode in a one-byte tag, which saves space on hot fields.

Summary

  • A protobuf message is a sequence of tag/value pairs; a tag is (field_number << 3) | wire_type.
  • Varints use 7 bits per byte; ZigZag keeps small negative numbers small.
  • Strings, bytes and nested messages are length-delimited.
  • Decoders can skip unknown fields, which makes adding fields safe.
  • Never change or reuse numbers; mark retired ones reserved.

Next: Enums, Repeated Fields and Maps