Node.js Reference Β· Streams and readline
Node.js TextDecoderStream and TextEncoderStream
Reference for TextDecoderStream and TextEncoderStream in Node.js: turn byte streams into strings and back, handle split characters, BOMs and bad input.
TextDecoderStream is a transform stream that converts chunks of bytes into strings. TextEncoderStream does the opposite: strings in, UTF-8 Uint8Array chunks out. They are globals in Node.js, and they wrap TextDecoder and TextEncoder with streaming behavior.
The important feature is correct handling of characters that straddle chunk boundaries. A multi-byte UTF-8 character can arrive split across two chunks, and decoding each chunk separately would produce garbage. The decoder stream keeps the incomplete bytes and finishes the character when the next chunk arrives.
const text = response.body.pipeThrough(new TextDecoderStream());
Try it
Press Run (or Ctrl + Enter) to see the result here.
Program input Β· what you'd type while the program runs
Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.
Press Run (or Ctrl + Enter) to see the result here.
Program input Β· what you'd type while the program runs
Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.
Syntax
new TextDecoderStream(label = 'utf-8', options?) new TextEncoderStream() decoderStream.encoding // string decoderStream.fatal // boolean decoderStream.ignoreBOM // boolean stream.readable stream.writable
| Name | Description |
|---|---|
label |
Encoding name such as 'utf-8', 'utf-16le' or 'latin1'. Supported labels depend on the ICU build of your Node.js. |
options.fatal |
When true, invalid input errors the stream with a TypeError. Default false: invalid bytes become U+FFFD. |
options.ignoreBOM |
When true, a leading byte order mark stays in the output. Default false: it is stripped. |
Returns: a transform stream pair. The decoder's writable side accepts ArrayBuffer, typed arrays and Buffers; its readable side emits strings. The encoder's writable side accepts strings; its readable side emits Uint8Array.
Errors: an unsupported label throws a RangeError (ERR_ENCODING_NOT_SUPPORTED). With fatal: true, bad bytes produce ERR_ENCODING_INVALID_ENCODED_DATA.
Usage notes
- Never call
toString()per chunk. It cannot know a character continues in the next chunk. The decoder stream, orStringDecoderfor Node streams, can. TextEncoderStreamis UTF-8 only. It takes no label. Lone surrogates in input are replaced by U+FFFD.- Lines are not chunks. Decoding gives you text pieces with arbitrary boundaries; use a TransformStream to split lines.
- The BOM is stripped once. Only a BOM at the very start of the stream is removed.
- Pairs well with
fetch.response.body.pipeThrough(new TextDecoderStream())streams a response as text. - Added in Node.js 18.