Node.js Reference Β· Buffers and Encoding
The StringDecoder class
Reference for string_decoder.StringDecoder in Node.js: write(), end(), multi-byte characters split across chunks, and how it differs from toString().
A StringDecoder turns a stream of byte chunks into strings without breaking characters that are split across chunk boundaries. A character like β¬ takes three bytes in UTF-8. If a network packet or file chunk ends after the second byte, calling chunk.toString() on it produces a replacement character, and the next chunk starts with a stray byte that decodes badly as well.
The decoder remembers the incomplete tail of each chunk and prepends it to the next one. It is what readable.setEncoding() uses internally.
import { StringDecoder } from 'node:string_decoder'; const decoder = new StringDecoder('utf8');
Try it
Press Run (or Ctrl + Enter) to see the result here.
Program input Β· what you'd type while the program runs
Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.
Press Run (or Ctrl + Enter) to see the result here.
Program input Β· what you'd type while the program runs
Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.
Syntax
import { StringDecoder } from 'node:string_decoder'; new StringDecoder(encoding = 'utf8') decoder.write(buffer: Buffer | TypedArray | DataView | string): string decoder.end(buffer?: Buffer | TypedArray | DataView): string decoder.encoding // string
| Name | Description |
|---|---|
encoding |
One of utf8, utf16le (alias ucs2), latin1, ascii, hex, base64, base64url. Defaults to utf8. |
write(buffer) |
Decodes as much as forms complete characters and keeps the rest. Returns the decoded string, possibly empty. |
end([buffer]) |
Optionally decodes one last chunk, then returns anything still buffered and resets the decoder. |
Returns: a string from write() and end().
Errors: an unknown encoding throws ERR_UNKNOWN_ENCODING; passing something that is not a buffer-like value to write() throws ERR_INVALID_ARG_TYPE. Passing a string to write() returns it unchanged.
Usage notes
- Always call
end(). A truncated sequence stays in the decoder untilend()runs, then comes out asU+FFFD. Forgetting it silently loses the last bytes. - Invalid bytes (not just incomplete ones) become
U+FFFDimmediately, as withbuf.toString(). - One decoder per stream. The decoder holds state, so never share an instance between two independent streams.
- Single buffers do not need it. If you already have all the bytes,
buf.toString()ornew TextDecoder().decode()is simpler.TextDecodersupports the same idea throughdecode(chunk, { stream: true }). - Streams do it for you.
readable.setEncoding('utf8')makes the stream emit strings decoded with aStringDecoder, so adatahandler never sees half a character. - Binary-to-text encodings.
hex,base64andbase64urlalso need state: base64 encodes three bytes at a time, and the decoder holds back the remainder. - Concatenating buffers first with
Buffer.concat(chunks).toString()also works, but keeps the whole input in memory.