Skip to content
Tutorials Center
My learning

Topics

Lesson 81 of 133 ยท Compression, CLI and Tooling

Compression with zlib

Learn Node.js compression with zlib - gzip, deflate and brotli in sync, async and streaming form, compression levels, CompressionStream, HTTP compression and protecting against zip bombs.

Text, JSON, HTML and logs are full of repetition, and compression exploits it: it replaces repeated patterns with short references so the same information takes fewer bytes. Smaller data means faster downloads, cheaper storage and less network traffic. Nearly every web page you load travels compressed, and log archives, backups and packages (.tgz, .zip, .gz) rely on the same ideas.

Node.js includes the node:zlib module, which implements three formats: gzip and deflate (both built on the DEFLATE algorithm from the zlib library) and Brotli, a newer algorithm that usually compresses text better. This lesson shows each of them in one-shot and streaming form, how to choose a level, how compression works in HTTP, and how to avoid the classic danger of decompressing untrusted data. You should be comfortable with Buffers and have seen Streams Fundamentals and Transform Streams.

The three formats

Format Functions Typical use
gzip gzip, gunzip Files (.gz), HTTP (Content-Encoding: gzip)
deflate deflate, inflate (zlib wrapper), deflateRaw, inflateRaw Inside other formats (PNG, ZIP, PDF), HTTP
Brotli brotliCompress, brotliDecompress HTTP over HTTPS, static assets

All three work on the same data, but they wrap it differently. gzip adds a header and a CRC-32 checksum, deflate adds a smaller header and an Adler-32 checksum, raw deflate adds neither, and Brotli has its own framing. Because compressed output is binary, you always work with Buffer values, never plain strings.

Compress and decompress in one call

The simplest API is the synchronous one: pass a string or Buffer, get a Buffer. Compression only pays off when there is something to squeeze, so the example uses repetitive text, like the JSON you might send from an API.

Node.js gzip, deflate and Brotli side by side

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The exact byte counts depend on the zlib and Brotli versions bundled with Node, so do not hard-code them in tests; check that the ratio is good instead, and compare against the original. Notice that raw deflate is smaller than deflate by the wrapper (2-byte header plus 4-byte checksum), and gzip is bigger by its header and trailer.

Tiny inputs get bigger

Compression has overhead. Compressing very short or already-random data often makes it larger. Images (JPEG, PNG), video, and encrypted data are already compressed or look random, so running them through gzip wastes CPU for no benefit.

Node.js When compression does not help

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The random-data line varies by a few bytes between runs because the input differs, but it will always be slightly larger than 5000. If you encrypt and compress, compress first: ciphertext cannot be compressed (see Encryption: AES, RSA and Keys). One caution: compressing secrets together with attacker-controlled text in the same response can leak information (the CRIME and BREACH attacks), so be careful when compressing responses that mix both.

Compression levels and options

Every compressor trades time for size. For gzip and deflate, level runs from 0 (store, no compression) to 9 (smallest, slowest), with -1 meaning the library default (6). For Brotli, quality runs from 0 to 11 through a parameter object.

Node.js Choosing a compression level

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Practical guidance:

  • On-the-fly HTTP responses: use a low or middle level (gzip 1 to 6, Brotli 4 to 5). Higher levels cost much more CPU for tiny gains, and that CPU time delays every response.
  • Static assets compressed at build time: use the maximum (gzip 9, Brotli 11) once, then serve the saved files forever.
  • Level 0 is useful only for formats that need a gzip container without compression.
  • Brotli at high quality is slow to compress but fast to decompress, so it suits content that is compressed once and read many times.

Async and promise APIs

The sync functions block the event loop, which is fine in a script and a mistake in a server handling many requests. The callback versions run the work on the libuv thread pool; wrap them with promisify to use await. Details are in The util Module.

Node.js Non-blocking compression with promisify

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Streaming: compress without loading everything

For large files you never want the whole input in memory. zlib.createGzip(), createGunzip(), createBrotliCompress() and friends are transform streams: bytes in, compressed bytes out. Connect them with pipeline, which handles errors and cleanup, as explained in Pipelines and Backpressure.

Node.js Compress a file with a pipeline, then read it back

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

You can also compress while data is being produced, with no file at all, and process the output chunk by chunk. For example, streaming gzip works with async iteration:

Node.js Gzip as a stream of chunks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

(The number and size of the compressed chunks are not something to rely on: gzip buffers internally and decides when to emit output. Always concatenate or pipe onward.) The same idea applies to File Streams.

CompressionStream: the Web API

Node also implements the browser's CompressionStream and DecompressionStream, which work with Web Streams. They support gzip, deflate and deflate-raw (Brotli support depends on the runtime version, so check before relying on it). This is the way to write compression code that also runs in browsers and other runtimes. See Web Streams.

Node.js CompressionStream and DecompressionStream

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Errors: corrupt and truncated data

Decompressing damaged or truncated input fails. Always handle the error, because the data often comes from a file or a network peer you do not control.

Node.js Decompressing damaged data (fails on purpose)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The first two messages are caught and printed; the third throws and ends the program with a stack trace (the exact text for a damaged stream, such as an invalid block or a failed check, depends on where the damage lands). For streams, errors arrive as 'error' events, and pipeline forwards them to the awaiting code. A related tool is zlib.crc32(data) (added in Node.js 22.2), which computes the CRC-32 checksum that gzip and ZIP use, handy for binary protocols.

Zip bombs: decompress untrusted data safely

A zip bomb is a tiny file that expands to an enormous size. Because compression removes repetition, a few kilobytes of gzip can expand to gigabytes of zeros and exhaust your memory (a denial-of-service attack). Never decompress user-supplied data without a limit on the output size. For one-shot calls use maxOutputLength; for streams count the bytes yourself and abort.

Node.js Limiting output size

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The compressed size (about 50 KB here, a ratio of roughly 1,000 to 1) varies slightly by zlib version. What matters: the error code ERR_BUFFER_TOO_LARGE for the sync call, and the explicit counter for the stream. Also protect against many files in an archive and against path traversal in archive entry names (an entry called ../../x that writes outside your folder), as described in Web Application Security.

HTTP compression

Browsers send Accept-Encoding: gzip, deflate, br to say what they understand. A server picks one, compresses the body, and replies with Content-Encoding: br (or gzip). The client decompresses transparently. Compress text responses (HTML, CSS, JavaScript, JSON, SVG), not images or videos, and add Vary: Accept-Encoding so caches store a separate copy per encoding. In production a reverse proxy (nginx, a CDN) usually does this for you, and you cache pre-compressed static files. Here is the mechanism in a tiny server and client. fetch sends Accept-Encoding automatically and decompresses the body.

Node.js A server that negotiates Content-Encoding

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

For incoming compressed request bodies (an upload with Content-Encoding: gzip), decompress with the same safeguards as above: a size limit, always.

Archives: what tar and zip add

Compression squeezes one stream of bytes. An archive bundles many files and their names into one stream. The classic Unix pairing is tar (bundle) plus gzip (compress), which is why you see .tar.gz or .tgz files. .zip combines both ideas but compresses each file separately, which allows random access. Node has no tar or zip reader built in (npm packages such as tar and fflate exist, shown read-only below), but the concept is simple enough to build, which makes it clear what the format does.

Node.js A toy archive: bundle, then compress

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Compressing the whole bundle lets repetition across files shrink together (a tar.gz advantage), while zip's per-file compression lets you extract one file without reading the rest. In real projects use a maintained library, for example (read-only):

JavaScript
// Read-only: needs npm packages, which our runner does not have
import * as tar from 'tar';
await tar.create({ gzip: true, file: 'backup.tgz' }, ['src', 'package.json']);
await tar.extract({ file: 'backup.tgz', cwd: 'restore' });   // validates paths for you

Choosing wisely

  • Web responses: Brotli for static assets (compressed ahead of time) and gzip or low-level Brotli for dynamic responses.
  • Files and logs: gzip for compatibility; Brotli or other formats if you control both ends and size matters most.
  • Data inside other formats (PNG-like containers, custom protocols): raw deflate.
  • Large data: always streams. Untrusted data: always a size limit. Server code: never the *Sync functions in a request handler.
  • Do not compress images, video, archives or encrypted data again.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which zlib function pair compresses and decompresses data in the gzip format synchronously?

    gzipSync and gunzipSync handle the gzip format. deflate and inflate use the zlib wrapper, Brotli has its own functions, and there is no zipSync in node:zlib.
  2. 2. Why should a busy server avoid zlib.gzipSync inside a request handler?

    The synchronous calls run on the main thread. The async and streaming versions use the thread pool and keep the server responsive.
  3. 3. What does the maxOutputLength option protect against?

    With a maximum output size, decompression stops with ERR_BUFFER_TOO_LARGE instead of allocating gigabytes.
  4. 4. Which data is the WORST candidate for gzip?

    JPEG and encrypted data already look random: there are no repeated patterns left to remove, so compression adds CPU cost and slightly increases the size.
  5. 5. How does a client tell a server which compression formats it understands?

    The client lists supported encodings in Accept-Encoding. The server picks one and states it in Content-Encoding (and adds Vary: Accept-Encoding for caches).

Summary

  • node:zlib provides gzip, deflate (zlib-wrapped and raw) and Brotli; results are binary Buffer values.
  • Use *Sync functions in scripts, promisified or callback versions for concurrency, and createGzip()/createGunzip() with pipeline for large or streaming data.
  • Levels trade CPU time for size: low levels for dynamic responses, maximum levels for assets compressed once.
  • Compression helps repetitive text, not already-compressed or encrypted data; compress before you encrypt.
  • CompressionStream and DecompressionStream give a portable Web Streams API for gzip, deflate and deflate-raw.
  • Always handle decompression errors, and always limit decompressed size (maxOutputLength or a byte counter) for untrusted input.
  • HTTP uses Accept-Encoding, Content-Encoding and Vary; tar bundles files, gzip compresses the stream, and zip does both per file.

Next: Interactive CLIs with readline