Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 36 ยท Integration with Frameworks

Binary Data, Chunking and Compression

Send Buffers and ArrayBuffers with Socket.IO, understand binary placeholders, limit payload size, upload files in chunks and tune compression.

Socket.IO events can carry more than JSON. You can send Buffer, ArrayBuffer, typed arrays, Blob (in browsers) and even objects that contain them. Images, audio chunks, files, game state in compact binary form: all fit. This lesson shows how binary events are used, how they are encoded on the wire, and how to keep large transfers safe with size limits, chunking and compression.

Socket.IO is not available in the runner on this site. Real code is read-only. The runnable examples are plain Node.js stand-ins that implement the same ideas (placeholder extraction, chunked reassembly) so you can see what happens.

Sending binary

JavaScript
// server (Node.js): Buffer
socket.emit("avatar", { userId: 7, image: fs.readFileSync("a.png") });

// client (browser): ArrayBuffer, typed array, Blob or File
fileInput.addEventListener("change", () => {
  const file = fileInput.files[0];
  socket.emit("upload", { name: file.name, type: file.type, data: file });
});

// receiving: Buffer in Node.js, ArrayBuffer in browsers
socket.on("upload", ({ name, data }, ack) => {
  fs.writeFile(`uploads/${path.basename(name)}`, data, (err) => ack({ ok: !err }));
});

You do not have to flag anything: Socket.IO detects binary values anywhere in the arguments, including nested in objects and arrays. Note the path.basename(name) in the handler: never use a client-supplied name directly in a file path, or ../../etc/passwd becomes your problem.

How it travels on the wire

JSON cannot hold bytes. When a packet contains binary parts, the encoder splits it into several frames:

  1. A text frame with the JSON structure, where every binary value is replaced by a placeholder like {"_placeholder":true,"num":0}.
  2. One binary frame per attachment, in order.

The receiver collects the text frame, then reads as many binary frames as the packet announces, and puts them back into the placeholders. WebSocket supports binary frames natively. With HTTP long-polling, binary is base64-encoded inside text, which costs about a third more bytes, one reason to prefer WebSocket for binary-heavy apps.

Node.js Placeholders: pulling binary out of a packet (stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The structure is plain JSON, and the heavy parts travel as raw bytes. This is also why a client that does not understand binary events never confuses them with text: the placeholders are explicit.

Size limits

Servers refuse packets above maxHttpBufferSize, 1 MB (1e6 bytes) by default. A client that exceeds it is disconnected. This protects memory; raise it only deliberately:

JavaScript
const io = new Server(httpServer, {
  maxHttpBufferSize: 5e6, // 5 MB, per message
});

Even at 5 MB, one connection can tie up memory while the message is assembled, and many clients can do it together. Keep limits as low as the feature needs.

Chunking large uploads

For files bigger than a few megabytes, slice them and send the pieces with acknowledgements. Benefits: bounded memory, progress bars, retry of just the failed chunk, and no giant packet. Do not use Socket.IO for huge files at all if you can avoid it: a presigned URL upload to object storage is better. Chunking is the right tool when the file must flow through your own server (for example, to stream into a processing pipeline).

Node.js Chunked upload with progress and ack (stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Each chunk carries an index and the total, so the order of arrival does not matter and a retried chunk is idempotent (delivery guarantees). The final checksum catches corruption. In real code, also limit the number of concurrent uploads per user and delete incomplete ones after a timeout.

Compression

There are two different layers:

  • WebSocket per-message deflate (perMessageDeflate): compresses each WebSocket message. On the Socket.IO server it is disabled by default, because it costs CPU and memory on every connection. Enable it when messages are large text, and measure.
  • HTTP compression for polling (httpCompression): enabled by default for the long-polling transport, with a threshold (1024 bytes) under which messages are not compressed.
JavaScript
const io = new Server(httpServer, {
  perMessageDeflate: {
    threshold: 1024,       // do not compress small messages
    zlibDeflateOptions: { level: 3 },
  },
  httpCompression: { threshold: 2048 },
});

Compression does not help already-compressed data (JPEG, PNG, MP4, ZIP): you burn CPU for nothing. It helps repeated JSON text. For repeated structure, sending a compact format (arrays instead of objects with long keys, or a binary encoding) often beats compression with less CPU.

Node.js When does compression pay off? (zlib)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Repeated JSON shrinks a lot, random bytes do not, and the tiny message can even grow. That is why the default threshold exists.

Binary best practices

  • Send binary directly rather than base64 text; base64 adds about 33 percent.
  • Prefer WebSocket for binary-heavy traffic.
  • Validate sizes and types of everything the client sends.
  • For media, use signed URLs to storage and send only the URL over Socket.IO.
  • For frequent small updates (positions, audio frames), pack numbers into a Float32Array instead of JSON, and consider volatile emits.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. How does Socket.IO encode an event that contains a Buffer?

    The JSON part has placeholders with an index, and the binary parts follow as binary frames that are put back on arrival.
  2. 2. What is the default maxHttpBufferSize?

    The default is 1e6 bytes. Larger packets make the server close the connection.
  3. 3. Why upload a big file in chunks with acknowledgements?

    Chunking avoids one huge packet and allows retrying only the chunk that failed.
  4. 4. When is perMessageDeflate least useful?

    Compressed formats have little redundancy left, so deflate costs CPU and gains nothing.

Summary

  • Buffers, ArrayBuffers, typed arrays and Blobs can be emitted directly, even nested in objects.
  • The wire format is a JSON frame with placeholders plus binary frames; polling falls back to base64.
  • maxHttpBufferSize (1 MB by default) protects the server; raise it with care.
  • Chunk large files with index, total, acks and a checksum, or upload to object storage instead.
  • perMessageDeflate is off by default on the server; enable it only when it measurably helps.

Next: Testing Socket.IO Apps