Skip to content
Tutorials Center
My learning

Topics

Lesson 17 of 36 ยท Partitioning and Storage

Distributed File Systems

How distributed file systems work: file access models, client caching and validation, sharing semantics, and chunked, replicated storage in the style of GFS and HDFS.

A distributed file system (DFS) lets programs on many machines use files stored on other machines with the same operations they use for local files: open, read, write, close. Network file shares in an office, the shared home directories of a university cluster, and the huge chunked file systems behind big-data platforms are all distributed file systems. They apply nearly every idea in this tutorial at once: replication, partitioning, caching, consistency and failure handling.

This lesson explains the main design questions: how clients access files, how caching makes remote files fast and what it costs in consistency, what "sharing" a file means when several clients write, and how large-scale systems split files into replicated chunks.

What a DFS must hide

A DFS aims for the transparency goals from What Is a Distributed System?: users should not care where a file is (location transparency), whether it is replicated, or whether the server it lived on crashed and was replaced. Everything else follows from three components: clients that issue file operations, a naming and metadata service that maps paths to the places data lives, and storage servers that hold the bytes.

File access models

There are two ways a client can touch remote data.

  • Remote access model: every operation is sent to the server. There is exactly one copy, so consistency is easy, but every read pays network latency and the server carries all the load.
  • Download/upload (data-caching) model: the client fetches the file or a part of it, works on the local copy, and sends changes back. Reads and writes are fast and the server load drops, but copies can go stale and writes can conflict.

Almost every real DFS uses the caching model, because latency to a remote server is orders of magnitude larger than local memory or disk. A related choice is the unit of transfer: a whole file (simple, good for sequential access, bad for a huge file when you need one byte) or blocks and chunks (flexible, more protocol chatter).

Caching and consistency

Caches can live in the client's memory or on its local disk. They create the central problem: when another client changes the file, the cached copy is wrong. There are three broad answers.

  • Validate on use: before using cached data, ask the server whether the version is still current. Always correct, but each use costs a round trip (a short one, no data).
  • Time-based expiry: trust the cache for a few seconds, then revalidate. Cheap, but clients can see stale data inside the window.
  • Server callbacks: the server remembers who caches what and notifies clients when the file changes. Few round trips, but the server must keep state and handle clients that crash or vanish.

On the write side, write-through sends every write to the server immediately (safe, slow), while write-back (delayed write) batches changes and sends them later or on close (fast, and a client crash can lose recent writes).

Python Stale reads without validation, extra round trips with it

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The fast client never asks the server again and so keeps showing v1 after the file changed. The safe client pays a round trip every time and sees v2. Real systems choose a point between these: short expiry times, callbacks, or leases like those in Distributed Locks, Leases and Fencing.

File-sharing semantics

What should a read return when someone else has written the file? Systems define one of several semantics:

Semantics Rule Notes
UNIX semantics A read sees the effect of every earlier write, immediately Natural, but needs a single copy or strong coordination; costly across a network
Session semantics Writes become visible to others only when the writer closes the file Used by AFS-style systems; simple, but concurrent writers produce a last-close-wins file
Immutable files Files are never modified, only created and replaced Sharing becomes trivial; versions replace updates
Transaction semantics A group of operations appears atomic Strongest, needs locking or transactions

Choose according to the workload. If files are mostly read, or are written once and appended to, relaxed semantics give large performance gains. If many clients edit the same file at once, you need locks or a different design.

Chunked, replicated storage

Systems built for huge files (logs, datasets, media) split each file into large fixed-size chunks (commonly 64 to 128 MB) and store each chunk on several machines. A central metadata server (called the master or name node) holds only the namespace and the chunk map. Data nodes store the chunks and serve data directly to clients, so the metadata server never becomes a bandwidth bottleneck.

Python Splitting a file into chunks and replicating them

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The simulation uses 8-byte chunks to keep the output short and puts every chunk on three of five nodes, so losing any two nodes loses no data. This design has well-known consequences:

  • Fault tolerance by replication. When a data node dies, the metadata server notices (missed heartbeats, see Failure Detection) and instructs other nodes to copy the lost replicas until each chunk has enough copies again.
  • Rack awareness. Replicas are spread over different racks, so a switch failure does not take out every copy.
  • Append-friendly. Large sequential reads and appends are fast; small random updates are slow, so these systems suit batch analytics, not databases.
  • The metadata server is critical. It is replicated and its operation log is persisted, or its state is kept in a consensus-backed store (Raft).
  • Small files hurt. Millions of tiny files overload the metadata server; systems pack them into bigger containers.

For where files, blocks and objects fit among other storage choices, see SQL vs NoSQL and storage, and for how data is spread over nodes, Sharding.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the main downside of the remote access model, where every operation goes to the server?

    With one copy there is no staleness, but each read and write crosses the network. Caching trades that cost for consistency problems.
  2. 2. Under session semantics, when do other clients see a writer's changes?

    Changes are published at close, so the system avoids per-write coordination. Two clients writing at once end with whichever closes last.
  3. 3. Why do chunked file systems keep the metadata server out of the data path?

    Clients ask the metadata server only where a chunk lives, then read the bytes directly from data nodes.
  4. 4. A cache that trusts its data for 30 seconds before revalidating can show:

    Time-based expiry is cheap but allows staleness inside the window.

Summary

  • A distributed file system hides location, replication and failures behind ordinary file operations.
  • Clients almost always cache; caches need validation, expiry or callbacks to stay reasonably fresh.
  • Write-through is safe and slow; write-back is fast and can lose recent writes.
  • Sharing semantics (UNIX, session, immutable, transactional) define what readers see; pick the weakest that fits the workload.
  • Large-scale systems split files into big replicated chunks with a small metadata service and many data nodes.

Next: CAP, PACELC and Real Trade-offs.