Skip to content
Tutorials Center
My learning

Topics

Lesson 25 of 133 ยท Buffers and Binary Data

Buffers

Learn Node.js Buffers - creating them with from, alloc and allocUnsafe, hex and base64 encodings, subarray memory sharing, concat, compare, reading and writing integers, and Buffer versus Uint8Array.

JavaScript strings are great for text, but files, network packets, images and encrypted data are bytes: numbers from 0 to 255. When Node.js reads a file without telling it an encoding, or receives data from a socket, it does not give you a string. It gives you a Buffer, a fixed-length chunk of raw memory with convenient methods for reading it, slicing it and converting it to and from text.

Understanding buffers makes many "mysterious" things clear: why console.log(await readFile('a.png')) prints <Buffer 89 50 4e 47 ...>, why concatenating chunks of a stream as strings can corrupt a character, and what base64 and hex really are. Buffer is a global, so no import is needed (though import { Buffer } from 'node:buffer' is fine and explicit). Technically a Buffer is a subclass of Uint8Array, the standard JavaScript typed array covered in Typed Arrays, ArrayBuffer and DataView, which means that everything you learn about typed arrays works on buffers too.

Creating buffers

There are three families of constructors: from existing data, zero-filled, and uninitialised. Never use the old new Buffer(...) constructor; it is deprecated because its behaviour depended on the argument type and could leak memory.

Node.js Buffer.from, alloc and allocUnsafe

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

What to take from this:

  • Buffer.from(string) encodes the text as UTF-8. The five characters of Hรฉllo take six bytes because รฉ needs two.
  • Buffer.from(array) stores each number modulo 256, so 289 becomes 33 and -1 becomes 255.
  • Buffer.from(buffer) copies; changing the copy does not affect the original.
  • Buffer.alloc(n) returns n zero bytes. The optional second argument fills the buffer, repeating a string if needed.
  • Buffer.allocUnsafe(n) skips the zeroing. It is faster for big buffers you are about to overwrite completely, but it may expose leftover memory from earlier operations, so use it only when you write every byte before anyone reads it. If in doubt, use alloc.

The printed form <Buffer 48 c3 a9 ...> shows each byte as two hexadecimal digits. console.log shows at most 50 bytes and then ... 950 more bytes.

Reading and writing single bytes

A buffer behaves like an array of small integers. You can index it, loop over it with for...of and compare values. Assigning a number outside 0 to 255 wraps around, and assigning to an index beyond the length is silently ignored.

Node.js Indexing a buffer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Indexing out of range returns undefined (not an error), a classic source of silent bugs; check length when parsing untrusted data.

Encodings: text, hex, base64

A buffer holds bytes; an encoding says how to convert between bytes and a string. Node supports these:

Encoding What it is
utf8 (default) Unicode text, 1 to 4 bytes per character
utf16le / ucs2 Two or four bytes per character, little endian
latin1 / binary One byte per character, code points 0 to 255 only
ascii 7-bit ASCII (mostly the same as latin1 when decoding)
hex Two hexadecimal digits per byte
base64 Text-safe encoding, 4 characters per 3 bytes
base64url Like base64 but URL-safe (- and _, no padding)
Node.js Converting between encodings

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Three lessons hide in that output:

  1. Bytes are not text. The same four bytes can be one emoji (UTF-8), a few strange Latin-1 characters, or nothing meaningful. Always know which encoding applies.
  2. Invalid UTF-8 is replaced, not rejected. toString() turns undecodable bytes into the replacement character ๏ฟฝ (shown as ๏ฟฝ in the JSON). Round-tripping binary data through a UTF-8 string destroys it. To carry arbitrary bytes in text, use hex or base64.
  3. Base64 is not encryption. It is just a different spelling of bytes (and costs about 33% more space). Anyone can decode it. Hashing and encryption are in Hashing and HMAC and Encryption.

base64url replaces + and / with - and _ and drops the = padding, which is what you need in URLs and in JSON Web Tokens.

Byte length is not string length

'Node ๐Ÿš€'.length is 7 (UTF-16 code units, the emoji counting as two), the UTF-8 byte length is 9. When you need the number of bytes, for a Content-Length header or a size limit, use Buffer.byteLength:

Node.js Buffer.byteLength

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The columns are: the text, its .length, its UTF-8 bytes and its UTF-16 bytes. This difference between characters and bytes gets its own treatment in Text, Encodings and Unicode.

Slicing: subarray shares memory

A very important and surprising rule: subarray does not copy. The new buffer is a view onto the same memory, so writing through either one changes both. The old slice method behaves the same way on buffers (unlike on arrays and strings), which is why subarray is the preferred name. To get an independent copy, use Buffer.from(view) or copy.

Node.js subarray is a view, not a copy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Sharing is a performance feature (slicing a huge buffer costs nothing) and a hazard: a tiny view keeps the entire underlying memory alive, and a function that returns a view lets the caller modify your data. When you keep a small piece of a big buffer for a long time, copy it.

Joining, comparing and searching

Node.js concat, compare, equals, indexOf

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Buffer.compare(x, y) returns -1, 0 or 1 and is ideal for sort. Use equals for equality; the == operator compares object identity, not contents. For comparing secret values such as signatures, use crypto.timingSafeEqual, which does not leak information through timing differences.

Reading and writing numbers

Binary formats store numbers in fixed sizes. The read* and write* methods name the type and, where it matters, the byte order (endianness): BE for big endian (most significant byte first, the "network order") and LE for little endian (least significant first, what most CPUs use). Methods exist for 8, 16, 32 and 64-bit integers (BigInt for 64), unsigned and signed, and 32-bit float and 64-bit double.

Node.js Reading and writing integers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Notice what happens with endianness: 0x1234 is stored as bytes 12 34 in big endian and 34 12 in little endian, and reading with the wrong order silently gives a different number (0x3412). That is the number-one bug in hand-written binary parsers. There is also readUIntBE(offset, byteLength) for sizes like 3 or 6 bytes. Unlike plain indexing, these methods throw a RangeError: ERR_OUT_OF_RANGE when the value does not fit the type, ERR_BUFFER_OUT_OF_BOUNDS when the buffer is too short. We build a complete parser on top of them in Working with Binary Protocols.

More Buffer methods: a quick tour

A few methods did not fit the sections above. They are small, and knowing they exist saves you from writing loops by hand.

Node.js swap, doubles, odd-sized integers, fill and write

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Some details to keep: the swap methods change the buffer in place and return it, and they throw ERR_INVALID_BUFFER_SIZE if the length is not a multiple of 2, 4 or 8. write(string, offset, length, encoding) never writes half a UTF-8 character: it stops before a character that would not fit, so the return value can be smaller than you expect. fill with a string repeats it to the end of the range. JSON.stringify turns a buffer into {"type":"Buffer","data":[...]}, which is why you should send binary data as base64 text instead, and convert it back with Buffer.from(text, 'base64'). The small shared pool behind Buffer.allocUnsafe is sized by Buffer.poolSize (8 KiB by default); buffers larger than half of it are allocated on their own.

Two more details complete the picture. Buffer.allocUnsafeSlow(size) is the unpooled twin of allocUnsafe: it always gets its own memory block, which is the right choice for a small buffer you keep for a long time, because a tiny pooled buffer would otherwise keep its whole shared pool alive. And the IntBE/IntLE/UIntBE/UIntLE methods exist in read and write pairs with an explicit byte length from 1 to 6.

Node.js allocUnsafeSlow and 3-byte integers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Buffer is a Uint8Array

Because Buffer extends Uint8Array, you can pass a buffer anywhere a Uint8Array is accepted, and use the typed-array API.

Node.js Buffer versus Uint8Array

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The differences are small but real. Buffer#toString takes an encoding while Uint8Array#toString joins numbers with commas. JSON.stringify(buffer) gives {"type":"Buffer","data":[...]}. And small buffers made by from or allocUnsafe are cut out of a shared 8 KiB pool, so buf.buffer can be bigger than the buffer, and buf.byteOffset is usually not zero. That is why code that needs the underlying ArrayBuffer must always pass byteOffset and length, as in the example. New library code often accepts Uint8Array and only uses Buffer where encodings are needed, which keeps it working in browsers too.

Buffers in the wild

Files and chunks

Reading a file without an encoding gives a buffer; with an encoding it gives a string.

Node.js Chunked data and multi-byte characters

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

If you decode each chunk separately, a character that straddles the boundary turns into replacement characters. Either collect the buffers and Buffer.concat them before calling toString, or use a StringDecoder (or a TextDecoder in streaming mode), which remembers incomplete bytes between calls. Streams that set an encoding with setEncoding() do this for you; see Streams Fundamentals.

Equality, size and memory

  • Binary data is cheaper as a buffer than as a string: the string is stored with 1 or 2 bytes per character plus overhead.
  • Large buffers live outside the JavaScript heap (as external memory), so process.memoryUsage().external or arrayBuffers grows when you allocate them, not heapUsed.
  • buffer.constants.MAX_LENGTH is the largest allowed buffer. It has changed between Node and V8 versions, so read it instead of hard-coding a number.
  • Avoid creating many tiny buffers in a hot loop; each is an object with overhead. Preallocate and write at offsets instead.

Common pitfalls

  • Concatenating buffers with +. buf1 + buf2 converts both to UTF-8 strings first. Use Buffer.concat.
  • Using == to compare contents. Use equals.
  • Forgetting that subarray shares memory. Copy if the data must be independent.
  • Wrong endianness when reading numbers written by another system.
  • allocUnsafe without overwriting every byte.
  • Assuming one string character is one byte.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is Buffer.from('รฉ').length?

    The default encoding is UTF-8 and the character รฉ needs two bytes (c3 a9), although the string has length 1.
  2. 2. You change a byte in buf.subarray(0, 4). What happens to buf?

    subarray returns a view onto the same memory. Use Buffer.from(view) when you need a copy.
  3. 3. Which approach safely carries arbitrary binary data inside a text-only channel such as JSON?

    Hex and base64 can represent every byte value. Decoding arbitrary bytes as UTF-8 replaces invalid sequences and loses data.
  4. 4. What does Buffer.alloc(4).writeUInt8(300) do?

    The write methods validate their input. Only plain indexing assignment (buf[0] = 300) wraps around silently.
  5. 5. Why should you avoid decoding each stream chunk with chunk.toString() separately?

    UTF-8 characters can use up to four bytes. Collect the chunks and concat them, or use a StringDecoder that keeps incomplete bytes.

Summary

  • A Buffer is a fixed-size chunk of raw bytes and a subclass of Uint8Array; it is a global.
  • Create buffers with Buffer.from, Buffer.alloc (zero-filled) and, carefully, Buffer.allocUnsafe; never new Buffer().
  • Encodings convert bytes and text: utf8, hex, base64, base64url, latin1, utf16le. Use Buffer.byteLength for byte sizes.
  • subarray (and slice) share memory; copy with Buffer.from(view) or copy.
  • concat, equals, compare and indexOf replace the string operators that do not work on buffers.
  • readUInt16BE, writeInt32LE and friends handle numbers; choose the right endianness and expect RangeError when values do not fit.
  • Decode streamed data after concatenating chunks or with a StringDecoder.

Next: Typed Arrays, ArrayBuffer and DataView