Skip to content
Tutorials Center
My learning

Topics

Lesson 120 of 133 ยท Projects

Project: Directory Snapshot and Diff Tool

Build a tool that snapshots a folder with SHA-256 hashes, finds added, changed, moved and deleted files, speeds up rescans with a quick check, and syncs a mirror safely.

"What changed in this folder?" is a surprisingly common question. A backup program asks it to decide what to upload. A build tool asks it to decide what to rebuild. A deploy script asks it to decide what to copy to the server. A synchronization tool such as rsync is built around it.

In this project you build the core of such a tool with the standard library alone. You will walk a directory tree, fingerprint every file with a SHA-256 hash read through a stream, save the result as a snapshot, compare two snapshots to produce a report of added, modified, moved and deleted files, rescan quickly by trusting file sizes and timestamps, and finally apply the differences to a mirror folder, with a dry-run mode and protection against writing outside the target. Every example creates its own temporary folder, so it is safe to run and leaves nothing behind.

Plan: snapshots are plain data

The key design decision is to turn a directory into data first and compare data afterwards. A snapshot is a plain object:

JSON
{
  "README.md": { "size": 6, "mtimeMs": 1767225600000, "sha256": "5891b5b5..." },
  "src/app.js": { "size": 20, "mtimeMs": 1767225600000, "sha256": "6f4c113f..." }
}

Keys are paths relative to the root, written with forward slashes on every operating system. Values hold the size, the modification time and the content hash. Because it is plain JSON, a snapshot can be stored in a file, sent over the network or compared with another one without touching the disk again. That gives the tool three clean layers:

  1. Scanning (walk, hashFile, takeSnapshot): talks to the file system.
  2. Comparing (diffSnapshots): pure function from two objects to a report. Easy to test.
  3. Applying (planSync, applySync): turns a report into file operations, with a safe dry run.

Why hash the content instead of comparing timestamps? Because timestamps lie. A git checkout or a file copy changes the modified time of a file whose content is identical, and some tools restore old timestamps over changed content. A cryptographic hash such as SHA-256 changes whenever the content changes, and the chance that two different files have the same hash is, for practical purposes, zero. We will still use timestamps later, but only as a speed-up and with eyes open.

Stage 1: walking a tree and hashing files

Walking a directory tree is a natural fit for an async generator: walk yields file paths one at a time, recursing into subdirectories with yield*, so a tree with a million files is never held as one giant list by the walker. Four details matter.

  • readdir(dir, { withFileTypes: true }) returns directory entries that already know whether they are files or folders, so there is no extra stat call per entry.
  • The entries are sorted. Directory listing order depends on the file system, and a tool whose output order changes between machines is painful to test and to diff.
  • Only regular files are yielded. entry.isFile() is false for symbolic links, so the walker never follows a link out of the tree or around in a loop. Empty directories produce no output.
  • Names such as node_modules and .git are skipped through an ignore set.

hashFile reads the file through a stream and feeds each chunk to a crypto hash. A file of any size is hashed with a few kilobytes of memory, the same constant-memory idea as in the log processor project. See crypto and hashing for more about hash functions.

takeSnapshot combines the two. It collects the file list, then starts a small number of workers (4 by default) that each take the next file, stat it and hash it. This is bounded concurrency: hashing is a mix of disk reads and CPU work, so a few files at once keeps both busy, while hashing ten thousand files at once would open ten thousand file descriptors and fail with EMFILE (see concurrency control). Sharing a counter between the workers is safe, because JavaScript only switches tasks at an await.

Node.js Stage 1: walk a folder and take a snapshot

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The walk found four files and nothing else: node_modules and .git were skipped, and empty-folder has no files. The snapshot lists them in sorted order with their sizes and the first 12 characters of the hash. README.md contains hello and a newline, six bytes, and its SHA-256 starts with 5891b5b522d5, which you can check with printf 'hello\n' | sha256sum on a Unix machine. The empty file has the famous hash that begins with e3b0c44298fc: the SHA-256 of zero bytes.

The helper functions at the top (makeTree, writeTree) create a temporary folder and files and set every modified time to a fixed date. Fixed times keep the output identical on every run, which is also how you would make a test for a tool like this one.

Stage 2: comparing two snapshots

diffSnapshots(before, after) is a pure function, and its algorithm is short:

  1. A path that exists in after but not in before is added.
  2. A path in both with different hashes is modified; with equal hashes it is unchanged (only counted, not listed).
  3. A path in before but not in after is removed.
  4. Move detection: for each removed file, look for an added file with the same hash. If you find one, the file was renamed or moved: report one from -> to line instead of a deletion plus an addition.

Step 4 is what makes the report useful. A backup tool that sees a renamed 2 GB video as "delete one file, upload another" wastes hours, and a tool that knows it was a move does almost nothing.

Move detection has traps, and the code handles each one:

  • Empty files all share one hash, so any two empty files look like the same file. The code never treats size-0 files as moves.
  • Each added file can be the target of only one move. The code removes a matched file from the "still added" list, so two removed files with the same content cannot both claim it.
  • A copy is not a move. If a.txt still exists and copy.txt appears with the same content, a.txt is not in the removed list, so nothing is wrongly reported as moved.

The program below scans a small site, then changes it: edits one file, adds one, deletes one and moves one into a new folder. It prints the report and then runs four checks that use small hand-made snapshots, with no files at all.

Node.js Stage 2: the diff report and its tests

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The report uses one symbol per kind of change, like a compact git status: + added, ~ modified, > moved, - removed. It shows the added contact.html, the edited index.html, the move of old/notes.txt into archive/, and the deleted tmp.log. The three files nobody touched (about.html, css/site.css, img/logo.svg) are only counted.

Because diffSnapshots takes plain objects, the tests do not need a file system. This is the payoff of the "data first" design. The four checks run in a few milliseconds and cover the tricky cases (empty files, copies), which would be tedious to set up with real folders. Use node:test for real projects; plain assert keeps this example small.

Stage 3: faster rescans, and the trade-off

Hashing every byte of a big folder on every run is slow. The standard trick, used by rsync, Git and many build tools, is the quick check: if a file has the same size and the same modified time as in the previous snapshot, assume it did not change and reuse the old hash. Only files that fail the check are read again.

takeSnapshot accepts a previous snapshot for exactly this, and fills a stats object with the number of files hashed, reused and vanished. The example saves a snapshot as JSON, loads it again as a later run would, and shows the numbers:

  • first run: all 4 files hashed;
  • nothing changed: 0 hashed, 4 reused (this is the fast path);
  • two files edited: only those 2 are hashed again;
  • a deliberate trap: a.txt gets new content of the same size, and its modified time is set back to the old value.
Node.js Stage 3: the quick check and its blind spot

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The last part is the lesson. The quick check missed the change to a.txt (false) because size and timestamp both matched, while a full rescan without a previous snapshot found it (true). The final report even lists all three changed files.

This is not a bug in the code, it is the price of the shortcut, and every tool that uses it has the same blind spot. In practice it is rare: ordinary editing always updates the modified time. But it means you should offer both modes. Use the quick check for daily runs and a full rescan (a --verify flag in a command-line tool) from time to time, or whenever you must be sure, for example before deleting a backup. It also explains why some tools compare the timestamp and the size: it makes accidental matches less likely.

Stage 4: files that vanish while you scan

Real folders change while you work. A build deletes temporary files, an editor saves through a temporary name, a user drags a file to the bin. If a file shows up in the listing and is gone a moment later, calling stat or opening it fails with ENOENT. The example below reproduces this by deleting a file between the listing and the stat call, with no protection:

Node.js A file vanishes mid-scan (fails on purpose)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The first line shows the listing, which still contains gone.txt. Then the program crashes with Error: ENOENT: no such file or directory, stat '/tmp/snapshot-.../gone.txt' (the part in the folder name is random on every run), and code: 'ENOENT' tells you exactly what went wrong. The code that checks for this in takeSnapshot is a try/catch around each file:

JavaScript
try {
  // stat + hash the file
} catch (error) {
  if (error.code !== 'ENOENT') {
    throw error; // permissions, I/O errors and so on are real problems
  }
  stats.vanished++; // the file is gone: not part of this snapshot
}

Two rules are visible here. First, catch only the error you expect (ENOENT) and rethrow everything else, so a permission error or a failing disk is never hidden. Second, a vanished file is not an error for a snapshot, but you still count it, so the report can say "2 files vanished during the scan", because a busy folder may need a second pass. See file system errors and paths and permissions for the codes you will meet.

Stage 5: applying the diff to a mirror

The last piece turns a report into action: make a mirror folder equal to a source folder. The plan is a list of simple operations built from the diff of target against source: move, then copy for added and modified files, then delete. Moves go first so that a renamed file is not copied again and its old copy is not deleted twice.

Three safeguards are part of the design, not extras:

  • Dry run. applySync(plan, source, target, { dryRun: true }) returns the list of actions it would perform and touches nothing. Always show the plan before the first real run.
  • Path containment. The plan comes from data, and data can be wrong or hostile: a snapshot loaded from a JSON file could contain a path like ../outside.txt. inside(root, relative) resolves the final path and refuses anything that does not stay inside the root. This is the classic path traversal bug, discussed in web security.
  • Verification. After syncing, the program snapshots the mirror again and diffs it against the source. If the sync worked, the answer is "everything unchanged".

Copies use fs.cp with preserveTimestamps: true, so the next quick check sees the same modified time on both sides.

Node.js Stage 5: dry run, sync and verification

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The dry run printed four planned actions: about.html is moved into pages/ instead of being deleted and copied, pages/team.html and index.html are copied, and old.log is deleted. The line mirror untouched: stale proves that the dry run changed nothing. After the real run the diff says (4 unchanged): mirror and source are identical, and the mirror's index.html has the new content. The last line shows the containment guard refusing ../outside.txt.

This sync is a teaching tool. It does not handle empty directories (a moved file can leave an empty folder behind), file permissions, symbolic links, very large files being modified while they are copied, or a crash in the middle of a copy. Real tools copy to a temporary name and rename it into place, so a half-copied file is never visible. Do not point it at data you cannot afford to lose without a backup.

Ideas for taking it further

  • A command-line interface. snapshot <dir> > state.json, diff old.json new.json, sync <src> <dest> --dry-run, using parseArgs and exit codes (see CLI tools).
  • Ignore files. Read patterns from a .snapignore file, with glob support through fs.glob or path.matchesGlob.
  • Watching. Combine fs.watch with the quick check to rescan only directories that reported a change (see directories).
  • Chunked hashing. Hash big files in blocks so that only the modified blocks of a large file are transferred. This is how rsync saves bandwidth.
  • Streams for huge trees. Write the snapshot as one JSON object per line (NDJSON) through a stream instead of one big object, as in streaming large data.
  • Progress. Emit events (EventEmitter) such as file and done so a user interface can show a progress bar.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why does the snapshot store a content hash instead of relying only on modified times?

    Copies and checkouts change timestamps for identical content, and some tools restore old timestamps over new content. A cryptographic hash changes if and only if the content changes (for practical purposes).
  2. 2. How does the diff recognize that a file was moved or renamed?

    Identical content under a new path is the signature of a move. Empty files are excluded, because every empty file has the same hash and would be matched wrongly.
  3. 3. What does the quick check (same size and same mtime means unchanged) trade away?

    The quick check never reads files that look unchanged, so a change that preserves both size and mtime goes unnoticed. A periodic full rescan closes that gap.
  4. 4. Why does takeSnapshot hash files with a small, fixed number of workers instead of Promise.all over all files?

    Starting one hash per file at once can exceed the limit of open files (EMFILE) and thrash the disk. A handful of workers keeps the disk and CPU busy without that risk.
  5. 5. What is the purpose of the inside() function in the sync code?

    A snapshot or plan may come from a file that someone edited. Resolving the final path and requiring that it stays inside the root prevents path traversal.

Summary

  • Turn a directory into plain data (a snapshot), compare data, then act. Each layer is easy to test.
  • Walk trees with an async generator, sort entries, yield regular files only, and skip names you do not want.
  • Hash files through a stream so memory use stays small, and use bounded concurrency for many files.
  • A diff reports added, modified, removed and moved files. Detect moves by hash, but never for empty files and never twice for one target.
  • The size and mtime quick check is fast and has a known blind spot; offer a full verify mode.
  • Catch ENOENT for files that vanish mid-scan and rethrow everything else.
  • Sync from a plan: dry run first, contain every path inside the target, and verify afterwards with another diff.

Next: Node.js Cheat Sheet