Skip to content
Tutorials Center
My learning

Topics

Node.js Reference ยท child_process, worker_threads and cluster

Node.js transferring and sharing memory between threads

How worker threads move or share memory in Node.js: transfer lists, detached ArrayBuffers, SharedArrayBuffer with Atomics, MessagePort transfer and Buffer pool pitfalls.

By default postMessage() copies its payload. That is safe, but copying a 100 MB buffer takes time and doubles memory. Worker threads offer two ways around it. Transferring moves ownership of an ArrayBuffer (or a MessagePort) to the receiver without copying; the sender's copy becomes empty. Sharing uses a SharedArrayBuffer, whose memory is visible to both threads at the same time.

Transfer is simple and race-free because only one thread owns the data at any moment. Sharing is the fastest way to exchange data repeatedly, but you are then responsible for synchronization, usually with Atomics. Both work with postMessage(), workerData and MessagePort.

NODE
worker.postMessage({ pixels: buffer }, [buffer]);   // transfer: second argument lists what to move
worker.postMessage({ counters: shared });           // share: a SharedArrayBuffer is never copied

Try it

Node.js Transferring an ArrayBuffer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Node.js SharedArrayBuffer with Atomics

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

How it works

Transfer lists

The second argument of postMessage(value, transferList) (or options.transferList for new Worker()) names objects to move instead of copy. Transferable objects in Node.js include ArrayBuffer, MessagePort, FileHandle (from fs/promises), web stream objects and several others. After the call:

  • The sender's ArrayBuffer is detached: byteLength is 0, and every typed array over it reports length 0.
  • The receiver gets an ArrayBuffer with the same bytes, in constant time.
  • Passing the same object twice, or putting something in the list that is not transferable, throws a DataCloneError.

A typed array is not itself transferable; transfer its .buffer. The typed array in the received message is rebuilt over the moved buffer.

SharedArrayBuffer

A SharedArrayBuffer posted to a worker is not copied or detached: both threads see the same memory through their own typed arrays. Writes by one thread become visible to the other, but without synchronization the order is unpredictable and updates can be lost. Use Atomics for counters, flags, locks and for waiting with Atomics.wait() and Atomics.notify(). Only integer typed arrays (Int32Array, BigInt64Array, and so on) work with Atomics.

Pitfall: the Buffer pool

Small Buffer objects created with Buffer.from() or Buffer.allocUnsafe() are slices of a larger shared pool. Their buffer.buffer is the whole pool, which cannot be transferred: the call throws a DataCloneError. Copy the bytes first. Use Buffer.alloc(), or create a standalone Uint8Array, for data you intend to transfer.

Usage notes

  • Choose by access pattern. One-way hand-off of big data: transfer. Many small updates seen by several threads: SharedArrayBuffer plus Atomics.
  • Detached buffers are unusable. Reading a detached typed array gives length 0, and creating a view on it throws a TypeError. Do not keep using a buffer after transferring it.
  • Structured data cannot be shared. Objects, strings and arrays are always copied. Share numbers by storing them in a typed array over a SharedArrayBuffer.
  • Plain writes to shared memory are racy. counters[1]++ is a read, an add and a write; two threads can interleave and lose updates. Atomics.add() is one indivisible step.
  • Memory is released with the last reference. A SharedArrayBuffer lives as long as any thread holds it.
  • MessagePort transfer moves one end of a MessageChannel to another thread, which is how workers talk to each other directly.
  • Main-thread blocking. Atomics.wait() blocks the calling thread and should only be used in workers (browsers forbid it on the main thread; Node.js allows it, but it freezes your event loop).