Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 36 ยท Performance and Alternatives

node_modules Layouts, Hoisting and the pnpm Store

Understand how npm hoists dependencies into a flat node_modules, why phantom dependencies happen, and how pnpm uses a content-addressable store with hard links and symlinks.

The three package managers agree on the protocol (registry, tarballs, package.json) but disagree on what ends up on disk. That choice decides how much space an install uses, how fast it is, and, less obviously, which imports accidentally work. This lesson explains the layouts: npm's flat tree, pnpm's store with links, and Yarn's Plug'n'Play.

Everything runs as plain-Node stand-ins that build small layouts on a temporary disk folder; the real tools do the same thing at scale.

How Node finds a package

When your code runs require('ms') or import 'ms', Node looks for node_modules/ms in the current file's folder, then in each parent folder, up to the filesystem root. The first match wins. Every layout below is a different way of arranging folders so that this walk finds the right version.

npm and Yarn Classic: the hoisted tree

Suppose app needs express and chalk, and both need ms. A naive layout nests copies:

TEXT
node_modules/express/node_modules/ms
node_modules/chalk/node_modules/ms

That wastes space, so npm hoists: it moves shared packages up to the top, leaving one copy where versions agree, and nests only on conflicts:

TEXT
node_modules/
โ”œโ”€โ”€ express/
โ”œโ”€โ”€ chalk/
โ”œโ”€โ”€ ms/                โ† hoisted, shared by both
โ””โ”€โ”€ qs/                โ† needed by express, also hoisted

Hoisting makes installs smaller and require fast, but it has a side effect: your app can import ms and qs even though its package.json never mentions them. These are phantom dependencies. They work today; if express drops qs next month, your app breaks without any change on your side.

Node.js Stand-in: hoisting and phantom dependencies

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The order in which packages are visited affects which copy is hoisted, which is why two machines could produce different trees without a lockfile.

pnpm refuses to make phantom dependencies possible, and also avoids storing the same file twice across your whole machine. It works in two layers:

  1. A content-addressable store, one per machine (for example ~/.local/share/pnpm/store). Every file of every package version is saved once, named by its content hash. A package that exists in a hundred projects is on disk once.
  2. Per-project node_modules made only of links:
    • node_modules/.pnpm/<name>@<version>/node_modules/<name>/ holds that package's files as hard links into the store (the same bytes on disk, a second name).
    • The same .pnpm/.../node_modules/ folder also contains symlinks to the package's own dependencies.
    • The top-level node_modules/<name> is a symlink that exists only for the project's direct dependencies.
TEXT
node_modules/
โ”œโ”€โ”€ express  โ†’ .pnpm/[email protected]/node_modules/express        (symlink)
โ””โ”€โ”€ .pnpm/
    โ”œโ”€โ”€ [email protected]/node_modules/
    โ”‚   โ”œโ”€โ”€ express/        (hard links to the store)
    โ”‚   โ”œโ”€โ”€ qs  โ†’ ../../[email protected]/node_modules/qs                (symlink)
    โ”‚   โ””โ”€โ”€ ms  โ†’ ../../[email protected]/node_modules/ms                 (symlink)
    โ”œโ”€โ”€ [email protected]/node_modules/qs/
    โ””โ”€โ”€ [email protected]/node_modules/ms/

The app can import express (it declared it) but not qs, because there is no node_modules/qs at the top. Phantom dependencies fail loudly. express itself finds qs through its sibling symlink. Node's resolution is happy because every package sees exactly its declared dependencies.

Real consequences:

  • Disk: the second project that uses [email protected] adds nearly zero bytes (only links).
  • Speed: installing from a warm store is mostly creating links, not copying or extracting.
  • Correctness: a missing declaration is caught the first time the code runs.

On Windows, pnpm uses junctions for the directory links, and hard links need the store and the project on the same drive. If they are on different drives, pnpm falls back to copying (use --package-import-method to change the method).

This stand-in puts a "package file" into a store folder, hard-links it into two projects, then compares inode numbers and link counts. Identical inodes mean one copy on disk.

Node.js Stand-in: one file, many names (hard links)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Yarn Plug'n'Play (PnP)

Yarn Berry can skip node_modules altogether. Instead of unpacking, it keeps compressed zip files in a cache and writes a single .pnp.cjs file: a map from (package, dependency name) to a zip location. Node is started with a loader (yarn node) that consults this map. Benefits: very fast installs (nothing to unpack), strict dependencies (no phantoms), and a repository that can commit its cache for "zero installs". Costs: some tools that crawl node_modules directly need patches, and you must run Node through Yarn. When a tool cannot cope, Yarn offers nodeLinker: node-modules (classic layout) and nodeLinker: pnpm (a pnpm-style layout).

Layout Disk use Phantom deps Compatibility
npm hoisted tree One copy per project Possible Best; everything expects it
pnpm store + links Shared across projects Blocked Very good; a few tools need public-hoist-pattern
Yarn PnP Zip cache, shared Blocked Good, with some friction

Fixing "Cannot find module" under pnpm

If code imports something it never declared, pnpm shows an error such as Cannot find module 'qs'. The right fix is to add the dependency (pnpm add qs). Escape hatches exist (public-hoist-pattern[]=*eslint* for tools that expect root packages, or shamefully-hoist=true for a fully flat layout), but they bring phantom dependencies back, so use them only for stubborn tooling.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is a phantom dependency?

    Hoisting puts transitive packages at the top level, so undeclared imports can work by accident and break later.
  2. 2. How does pnpm avoid storing the same package version twice on one machine?

    Files live once in the global store, named by content hash, and project folders link to them.
  3. 3. In a pnpm project, why can app code not import a transitive dependency such as qs?

    Top-level node_modules contains symlinks only for declared dependencies; transitive ones live under .pnpm.
  4. 4. What replaces node_modules in Yarn Plug'n'Play?

    PnP records where each package lives in a map and loads from zip archives instead of unpacked folders.

Summary

  • Node resolves packages by walking up node_modules folders; layouts arrange them differently.
  • npm and Yarn Classic hoist into a flat tree: compact and compatible, but they allow phantom dependencies.
  • pnpm stores each file once and builds node_modules from hard links and symlinks, which saves space and blocks phantoms.
  • Yarn PnP uses a resolution map and zip cache instead of folders.
  • Fix missing-module errors by declaring the dependency, not by flattening everything.

Next: Speeding Up Installs and Caching