Skip to content
Tutorials Center
My learning

Topics

Lesson 16 of 37 ยท Standard Library and Web APIs

URL, Streams and Encoding

Work with URL and URLPattern, ReadableStream and TransformStream pipelines, text encodings, base64 and hex, Blob and compression streams in Deno using web standard APIs.

Three groups of web standard APIs show up in almost every Deno program. URLs describe where things are. Streams move data in pieces instead of all at once. Encoding converts between text and bytes. They are standardised, so they behave the same in Deno, Node.js and browsers, and every example on this page is runnable.

URL and URLSearchParams

The URL class parses an address into parts. Do not slice URL strings by hand.

Node.js Parsing and building URLs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Note how '2 & more' was percent-encoded automatically. Building query strings by concatenation is a classic source of bugs and injection problems.

Deno also gives you file URLs. A path and a file: URL convert both ways, which matters on Windows:

TypeScript
import { fromFileUrl, toFileUrl } from "@std/path";

toFileUrl("/home/me/data.txt").href; // "file:///home/me/data.txt"
fromFileUrl(import.meta.url);         // the current file's path

URLPattern

URLPattern matches URLs against patterns with named parts, and is the core of simple routers:

Node.js Matching URLs with URLPattern

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Streams

A stream delivers data in chunks over time. Reading a 5 GB file as a stream uses a few kilobytes of memory, because only one chunk lives in memory at a time. There are three kinds:

Class Role
ReadableStream A source you read from (file, response body, stdin)
WritableStream A destination you write to (file, socket, stdout)
TransformStream Sits in between and changes chunks

You connect them with pipeTo and pipeThrough, and the runtime handles backpressure: a slow consumer automatically slows a fast producer.

flowchart LR
  A["Readable: file or fetch body"] --> B["TransformStream: decode text"]
  B --> C["TransformStream: split lines"]
  C --> D["Writable: output"]

A pipeline connects a source, transformations and a destination.

Build a small pipeline you can run:

Node.js A stream pipeline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Deno's own APIs plug into this model. Deno.stdin.readable, file.readable, file.writable, Response.body and Request.body are all web streams, so they chain together:

TypeScript
// copy a file, uppercasing the text on the way
using src = await Deno.open("in.txt");
using dst = await Deno.open("out.txt", { write: true, create: true });
await src.readable
  .pipeThrough(new TextDecoderStream())
  .pipeThrough(new TransformStream({ transform: (s, c) => c.enqueue(s.toUpperCase()) }))
  .pipeThrough(new TextEncoderStream())
  .pipeTo(dst.writable);

Streams are async iterable, so you can read them with for await (Async Code):

Node.js Reading a stream with for await

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Text and bytes

Computers store text as bytes. UTF-8 is the standard encoding, and the web APIs default to it.

Node.js TextEncoder and TextDecoder

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The difference between characters and bytes matters when you compute sizes or slice binary data: "๐Ÿฆ•".length is 2 in JavaScript but the emoji takes 4 bytes in UTF-8.

Base64, hex and other encodings

Binary data inside text formats (JSON, URLs, headers) is usually written in Base64 or hex. The @std/encoding package has the full set:

TypeScript
import { decodeBase64, encodeBase64 } from "@std/encoding/base64";
import { encodeHex } from "@std/encoding/hex";
import { encodeBase64Url } from "@std/encoding/base64url";

encodeBase64(new TextEncoder().encode("hi")); // "aGk="
encodeHex(new Uint8Array([255, 0, 16]));      // "ff0010"

The global btoa and atob handle Base64 for Latin-1 text only. Practise with a safe wrapper for any Unicode text:

Node.js Base64 for any text

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Blob and File

A Blob is an immutable chunk of binary data with a type; a File adds a name and modification time. They read lazily and can be streamed:

TypeScript
const blob = new Blob(["hello ", "world"], { type: "text/plain" });
console.log(blob.size, blob.type);
console.log(await blob.text());
const part = blob.slice(0, 5);

Compression streams

CompressionStream and DecompressionStream compress data with gzip, deflate or brotli in a pipeline, with no extra libraries:

Node.js Gzip round trip

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why use URL and URLSearchParams instead of string concatenation?

    They handle encoding and edge cases, avoiding bugs and injection problems.
  2. 2. What is backpressure in a stream pipeline?

    Pipes coordinate speed so memory does not fill with unconsumed chunks.
  3. 3. Which stream type changes chunks in the middle of a pipeline?

    A TransformStream has a writable side and a readable side and modifies data between them.
  4. 4. How many UTF-8 bytes does the emoji "๐Ÿฆ•" use?

    Characters outside the Basic Multilingual Plane take 4 bytes in UTF-8, although string.length counts it as 2 UTF-16 units.

Summary

  • URL, URLSearchParams and URLPattern parse, build and match addresses.
  • Streams process data in chunks: ReadableStream, WritableStream, TransformStream, joined with pipeThrough and pipeTo.
  • Deno files, fetch bodies and stdin are all web streams.
  • TextEncoder and TextDecoder convert between strings and UTF-8 bytes; characters and bytes differ.
  • Use @std/encoding for Base64, hex and friends; CompressionStream handles gzip.

Next: Path, FS and Data Formats