Skip to content
Tutorials Center
My learning

Topics

Lesson 28 of 133 ยท Buffers and Binary Data

Working with Binary Protocols

Learn to design and parse binary protocols in Node.js - length-prefixed framing, a streaming frame parser, bitfields, zlib.crc32 checksums, packing and unpacking, and endianness bugs, with a TCP demo.

Text protocols such as HTTP/1.1 and JSON lines are easy to read, but they are not the most compact or the fastest. Databases, game servers, message brokers, file formats and network equipment mostly speak binary protocols: every field has a fixed meaning, position and size, and numbers travel as raw bytes instead of digits. Writing a parser for one is a classic Node.js job, and it combines everything from the previous lessons: buffers, typed arrays and text encodings.

This lesson builds a small, realistic protocol step by step. You will learn why a stream of bytes has no message boundaries and how length-prefixed framing restores them, how to write a parser that survives chunks split at any byte, how to pack several flags into one byte with bit operations, how to protect messages with a CRC32 checksum, and how endianness bugs look and how to prevent them. At the end the parser runs on a real TCP connection on loopback. Sockets themselves are covered in TCP Servers and Clients with net.

Why you need framing

TCP, pipes, files and most other byte channels are streams: they deliver bytes in order, but they do not remember where one message ended and the next began. If a client sends two messages of 5 and 7 bytes, the server may receive them as 12 bytes at once, as 3 + 9 bytes, or as 12 separate one-byte chunks. A data event is not a message. Treating it as one works on your laptop and fails under load, which is one of the most common beginner bugs in network code.

The fix is framing: the sender adds enough information for the receiver to find the boundaries again. The three standard techniques are:

  1. Delimiters. End each message with a special byte, such as \n in JSON lines. Simple, but the delimiter must never appear inside the payload, so binary data needs escaping.
  2. Fixed-size messages. Every message has the same length. Trivial to parse and wasteful when sizes vary.
  3. Length prefix. Start each message with its length. The receiver reads the length, then exactly that many bytes. This works for any payload, including arbitrary bytes, and is used by TLS records, PostgreSQL, Kafka, WebSocket and many more.

We will use a length prefix. Our frame has a header of 8 bytes followed by the payload:

TEXT
offset  size  field
0       2     magic number 0xCAFE (a sanity check: "this is our protocol")
2       1     message type (1 = text, 2 = ping, 3 = data)
3       1     flags (bit field, explained later)
4       4     payload length, unsigned 32-bit big-endian
8       n     payload bytes

Protocol designers call this layout the wire format. Writing it down like this table before coding is the most useful habit in this whole topic, because the parser and the writer must agree on it byte for byte.

Encoding a frame

Encoding is the easy half: allocate a buffer of the right size and write each field at its offset, using the typed read and write methods you met in the Buffers lesson.

Node.js Encoding one frame

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The header begins with ca fe, then type 01, flags 00, and the length 00 00 00 03 for the three payload bytes 48 69 21. Use Buffer.byteLength(text) or Buffer.from(text) to get the byte length of text, never text.length, which counts UTF-16 code units. A payload such as 'hรฉllo' has 5 characters but 6 bytes, and a frame that announces 5 would corrupt every following message.

Write the format table in a comment next to your encoder and decoder. When a bug appears, you compare bytes against the table instead of against your memory.

Decoding: the partial-data problem

A naive decoder reads one buffer and assumes it holds exactly one frame. A correct decoder handles three situations:

  • The buffer holds less than a header: wait for more bytes.
  • The buffer holds a header but only part of the payload: wait for more bytes.
  • The buffer holds several frames, perhaps followed by the start of another: extract them all and keep the remainder.

The standard solution is an internal buffer that accumulates incoming chunks, plus a loop that peels off complete frames. We also validate the magic number and reject absurd lengths early, because a malicious or buggy peer could announce a 4 GB payload and make you allocate memory for it.

Node.js A streaming frame parser

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The output is identical for the three chunk sizes: one chunk with everything, chunks of seven bytes (which cut headers and multi-byte characters in half), and one byte at a time. This property, the parser must not care how the stream is chopped, is the single most important test for any protocol code. Always include a "feed it one byte at a time" test.

Some design notes professionals care about:

  • Buffer.concat on every chunk copies the pending data again. For small messages this is fine. For high throughput, keep a list of chunks and a byte count, and concatenate only when a full frame is available.
  • subarray creates a view without copying, so payload above is decoded while the memory still exists. If you store a subarray of a large buffer for a long time, it keeps the whole buffer alive; copy it with Buffer.from(view) in that case.
  • The MAX_PAYLOAD check is a security measure. Without it, one malicious header makes your server wait for gigabytes while holding connection memory.
  • After a protocol error you usually cannot resynchronise, because you no longer know where the next frame starts. The usual response is to close the connection.

Bitfields and flags

Our header has one flags byte. A byte has eight bits, so it can carry eight independent yes/no values. Packing booleans like this is common in binary formats because it costs one bit per flag instead of one byte, and sometimes because the format is older than your project.

You work with bits using the operators & (AND), | (OR), ^ (XOR), ~ (NOT) and the shifts << and >>. Give every bit a name with a constant so the code stays readable:

Node.js Packing flags and small numbers into a byte

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Things to remember:

  • 0b0000_0101 is a binary literal; underscores are just visual separators (numeric separators, ES2021).
  • To test a bit, use (byte & FLAG) !== 0. To set it, use |=. To clear it, use &= ~FLAG. To toggle it, use ^=.
  • To extract a multi-bit field, shift it down first and then mask: (byte >> 4) & 0b1111.
  • JavaScript bit operators work on 32-bit signed integers. 1 << 31 is -2147483648, and >>> 0 converts the result back to an unsigned number. That matters for 32-bit flag words; for 64-bit values use BigInt.

Checksums with CRC32

Networks and disks flip bits. A checksum lets the receiver detect that a frame was damaged. CRC32 is a fast checksum used by Ethernet, ZIP, PNG and gzip. It detects accidental corruption very well, but it is not a security feature: anyone can compute it, so an attacker can change the payload and fix the checksum. For tamper protection use an HMAC (see Hashing and HMAC).

Node.js has zlib.crc32(data, initialValue) built in since Node 22.2. It accepts a string or a buffer and returns an unsigned 32-bit number. Passing the previous result as the second argument continues the calculation, which lets you checksum data piece by piece.

Node.js zlib.crc32 and a trailing checksum

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The classic test value is cbf43926 for the nine ASCII digits. If your implementation prints something else, the problem is in your code, not in CRC32. If you target a Node.js version older than 22.2, there is no built-in function; write a table-based version or use a library.

Many formats store the checksum last (as above) so the sender can compute it while writing. Others store it in the header. Whichever you choose, document exactly which bytes it covers: the payload only, or the header too.

Endianness bugs

A multi-byte number can be written with its most significant byte first (big-endian, also called network byte order) or last (little-endian, the native order of x86 and ARM processors). The protocol must pick one. Reading with the wrong one does not throw; it silently produces a different number, which is why these bugs are so nasty.

Node.js The same bytes, two different numbers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Rules that prevent most endianness bugs:

  • Choose one byte order for the whole protocol and state it in the format table. Big-endian is the convention for network protocols.
  • Do not rely on the machine's native order. Always call the explicit BE or LE method; os.endianness() is rarely the answer.
  • Mind signedness. A length is unsigned, a temperature can be signed. Using readInt32BE for a length turns lengths above 2 GB into negative numbers.
  • A 53-bit limit applies to normal JavaScript numbers. readUInt32BE is safe, but 64-bit ids and timestamps need readBigUInt64BE and a BigInt.

A protocol on a real socket

Now we connect the pieces. The server listens on loopback with port 0 (the operating system picks a free port, so we read it from server.address() and never print it), uses our FrameParser on every connection and answers each text frame with an upper-cased reply frame. The client deliberately writes bytes in awkward pieces, to prove that the framing works when TCP chops the stream. Do not worry about every detail of the networking code; the net lesson explains it. Focus on how little the protocol code depends on how bytes arrive.

Node.js Binary frames over a loopback TCP connection

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The server receives the bytes in pieces whose boundaries have nothing to do with the message boundaries, and still answers ALPHA, BETA and GAMMA in order. TCP guarantees order and delivery, not boundaries, which is exactly the gap that framing fills.

Design checklist for your own protocol

When you design a protocol, or read one someone else designed, check these points:

  • Version field. Put a version number near the start so you can evolve the format later.
  • Magic number. A fixed start marker catches wrong-protocol connections and desynchronisation early.
  • Explicit byte order and signedness for every number.
  • Maximum sizes for every length field, enforced before allocating.
  • Unknown types. Decide what to do with a frame type you do not understand: skip it using its length (forward compatible) or close the connection.
  • Checksums or authentication, depending on whether you fear accidents or attackers.
  • Text inside binary. Say which encoding strings use (almost always UTF-8) and whether they are length-prefixed or zero-terminated.

For ready-made formats, Protocol Buffers, MessagePack, CBOR or BSON give you schemas and parsers, so you do not reinvent the wheel. Hand-written framing is the right choice for small protocols, for learning, and for gluing to existing systems, and knowing it makes those libraries easier to understand.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A client sends two 10-byte messages over TCP. What can the server's data events look like?

    TCP is a byte stream without message boundaries. The data event sizes depend on the network and timing, so the receiver must find boundaries itself, for example with a length prefix.
  2. 2. What should a length-prefixed parser do when the buffer holds a full header but only half of the payload?

    Incomplete data is normal on a stream. The parser keeps what it has and tries again when the next chunk arrives.
  3. 3. Which expression clears the FLAG_URGENT bit in the variable flags without touching the other bits?

    ~FLAG_URGENT has every bit set except the urgent one, so AND-ing with it keeps all other bits and clears that one.
  4. 4. Why is CRC32 not enough to protect a message against deliberate tampering?

    CRC32 detects accidental corruption. Tamper protection needs a keyed construction such as an HMAC.
  5. 5. The bytes 00 00 01 00 are read with readUInt32BE and with readUInt32LE. What happens?

    Big-endian puts the most significant byte first, so 00 00 01 00 is 0x0100 = 256. Little-endian reverses the order and reads 0x00010000 = 65536. Neither call throws.

Summary

  • Byte streams have no message boundaries; a data event is not a message. Frame your messages, most often with a length prefix.
  • Write the wire format as a table (offsets, sizes, byte order) and implement the encoder and decoder from it.
  • A good parser accumulates chunks, extracts every complete frame, keeps the remainder, validates magic numbers and rejects oversized lengths. Test it by feeding the same bytes in every chunk size, including one byte at a time.
  • Use &, |, ^, ~ and shifts for bitfields; mask after shifting, and remember that JavaScript bit operators work on signed 32-bit integers.
  • zlib.crc32 (Node 22.2+) detects accidental corruption; use an HMAC when attackers are a concern.
  • Always call the explicit BE or LE read and write methods, choose signed or unsigned deliberately, and use BigInt methods for 64-bit values.

Next: Reading and Writing Files