Skip to content
Tutorials Center
My learning

Topics

Lesson 10 of 47 ยท HDFS

The HDFS Read and Write Path

Follow a file through HDFS: how a client asks the NameNode, streams blocks down a DataNode pipeline, verifies checksums and how heartbeats keep metadata fresh.

Every HDFS operation is a conversation between three parties: the client (your program), the NameNode (which knows where everything is) and the DataNodes (which hold the bytes). The design rule is simple: the NameNode handles metadata only, and the data itself flows directly between the client and the DataNodes. If all bytes passed through the NameNode, it would become the bottleneck for the whole cluster.

This lesson follows one write and one read step by step. You will see the replication pipeline, acknowledgements, checksums, short-circuit reads, and the heartbeats and block reports that keep the NameNode's picture of the cluster correct. It builds on Replication and Block Placement and HDFS Architecture.

The real HDFS cannot run on this site. The commands and settings below are read-only, and the two runnable examples are Python stand-ins that model the protocol ideas with ordinary objects. They do not speak the real wire protocol.

The write path

When a client creates a file, the following happens.

  1. The client asks the NameNode to create the file. The NameNode checks permissions and that the path does not exist, then records the file in its namespace. No data exists yet.
  2. The client splits what it writes into packets (64 KB by default), grouped into blocks (128 MB by default).
  3. For each new block the client asks the NameNode to allocate it. The NameNode picks DataNodes using the placement policy and returns an ordered list, for example [dn1, dn4, dn7].
  4. The client opens a connection to the first DataNode only. That node connects to the second, and the second to the third. This chain is the write pipeline.
  5. The client sends packets to dn1. Each DataNode writes the packet to disk and forwards it downstream at the same time.
  6. Acknowledgements travel back up the pipeline: dn7 acks to dn4, dn4 to dn1, dn1 to the client.
  7. When the block is full, the client asks for the next block, and the cycle repeats. At the end the client calls complete, and the NameNode commits the file once the minimum number of replicas has been reported.

One write, three roles

Client Splits data into packets.
NameNode Picks nodes, records metadata.
DataNodes Store and forward the bytes.

Data never flows through the NameNode.

1 / 2

The pipeline is the clever part. A client sending three copies itself would use three times its own bandwidth. Chaining sends one copy out of the client, and the cluster's internal network does the rest, in parallel with the disk writes.

sequenceDiagram
    participant C as Client
    participant N as NameNode
    participant D1 as DataNode 1
    participant D2 as DataNode 2
    participant D3 as DataNode 3
    C->>N: allocate block
    N-->>C: pipeline [D1, D2, D3]
    C->>D1: packet
    D1->>D2: packet
    D2->>D3: packet
    D3-->>D2: ack
    D2-->>D1: ack
    D1-->>C: ack

Packets flow down the pipeline, acknowledgements flow back up.

When a DataNode fails mid-write

If a DataNode in the pipeline dies, the client does not give up. It stops, removes the bad node from the pipeline, and resumes with the remaining nodes. The NameNode notices that the block is under-replicated and arranges a new copy later. The write succeeds as long as the minimum number of replicas (dfs.namenode.replication.min, default 1) is written. This is why a failing disk does not fail your job.

Here is a stand-in that models the pipeline: three DataNodes, packets flowing forward, acks flowing back, and one node that dies halfway through.

Python Pipeline write with a node failure

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that dn4 holds only two packets: a replica that missed data is stale and must not be trusted. Real HDFS tracks this with a generation stamp on each block, so the partial replica is discarded when dn4 comes back, and the NameNode schedules a fresh copy.

Checksums: trusting the bytes

Disks and networks silently corrupt data now and then. HDFS therefore stores a CRC checksum for every 512 bytes of data (dfs.bytes-per-checksum) in a small metadata file next to each block file. The checksums are computed by the client when writing and verified at the end of the pipeline. On every read, the client recomputes them and compares.

If a replica fails verification, the client reports it to the NameNode, reads the block from another replica, and the NameNode orders a good copy to replace the corrupt one. Each DataNode also runs a background block scanner that re-verifies old blocks, because data that is never read could rot unnoticed.

Python Detecting a corrupt replica with checksums

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The read path

Reading is simpler than writing.

  1. The client calls open; the NameNode returns, for the first few blocks, the list of DataNodes holding each one, sorted by network distance to the client (same node, same rack, then other racks).
  2. The client connects to the closest DataNode for block 1 and streams it, verifying checksums.
  3. At the block boundary it closes that connection and opens one to the best node for block 2. The NameNode is asked again for further block locations when needed.
  4. If a node fails or a checksum mismatches, the client moves to the next replica in the list and remembers the bad node for the rest of that file.

The client reads in parallel across files and sequentially within a file, which is exactly what large scans need. Because many replicas exist, many readers spread over different nodes.

Short-circuit local reads

When the client runs on the same machine as the DataNode (typical for jobs scheduled next to their data), sending bytes through a TCP socket to yourself is wasteful. With short-circuit local reads (dfs.client.read.shortcircuit), the DataNode hands the client an open file descriptor over a Unix domain socket and the client reads the block file directly from disk, skipping the DataNode process. It is one of the most useful single settings for read-heavy workloads such as HBase.

XML
<property>
  <name>dfs.client.read.shortcircuit</name>
  <value>true</value>
</property>
<property>
  <name>dfs.domain.socket.path</name>
  <value>/var/lib/hadoop-hdfs/dn_socket</value>
</property>

Heartbeats and block reports

How does the NameNode know which DataNode holds which block, if it does not store block locations on disk? The DataNodes tell it, continuously.

  • Heartbeat: every 3 seconds (dfs.heartbeat.interval) each DataNode sends a small message saying "I am alive, here is my free space and load". The NameNode replies with commands: replicate this block to that node, delete that block, re-register. If heartbeats stop for about 10.5 minutes, the node is declared dead and its blocks are re-replicated.
  • Block report: when it starts, and then every 6 hours, a DataNode sends the full list of blocks it stores. Incremental reports are sent as blocks arrive or are deleted.

The NameNode's block map is therefore rebuilt from reports each time it restarts. This is why a restarted NameNode sits in safe mode (read-only) until enough blocks have been reported, so it does not start deleting "missing" replicas that have simply not checked in yet.

How the NameNode stays informed

Heartbeat Every 3 seconds, alive and load.
Block report Full block list, every 6 hours.
Commands Replies carry replicate and delete orders.
Safe mode Read-only until reports arrive.
1 / 2

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Who sends the data bytes of a block to the replica DataNodes?

    HDFS chains DataNodes into a write pipeline. The client uploads each packet once, and each node forwards it to the next one.
  2. 2. A DataNode in the write pipeline dies. What happens?

    The write continues with the surviving nodes as long as the minimum replica count is met, and the missing copy is made afterwards.
  3. 3. What are checksums used for in HDFS?

    A CRC is stored per 512 bytes. A mismatch makes the client use another replica and lets the NameNode replace the bad copy.
  4. 4. Why does a DataNode send block reports?

    Block locations are rebuilt from DataNode reports, which is also why a restarting NameNode waits in safe mode.

Summary

  • The NameNode handles metadata; file data flows directly between client and DataNodes.
  • Writes use a pipeline: one upload from the client, forwarding between DataNodes, acks flowing back.
  • A failed pipeline node is dropped and the block is re-replicated later; stale replicas are detected by generation stamps.
  • CRC checksums (per 512 bytes) detect corruption on reads and in background scans.
  • Reads pick the closest replica; short-circuit reads let a local client read block files directly.
  • Heartbeats (3 s) and block reports (6 h) keep the NameNode's view current; silent nodes are declared dead after about 10.5 minutes.

Next: Working with HDFS from the Shell.