Skip to content
Tutorials Center
My learning

Topics

Lesson 9 of 47 ยท HDFS

Replication and Block Placement

How HDFS keeps data safe: the replication factor, rack awareness, the default placement policy, re-replication after failures and an introduction to erasure coding.

Every HDFS block lives on several machines at once. That is how a cluster of cheap servers can lose disks, nodes and even whole racks without losing data. But where the copies go matters as much as how many there are: three copies on three servers under the same broken switch are no protection at all.

In this lesson you will learn the replication factor, how racks enter the picture, the default placement policy, what HDFS does when a node dies, and how erasure coding stores data more cheaply. You will run a placement and failure simulation in Python. It follows HDFS Architecture.

The real placement policy and settings shown here are read-only because they need a cluster. The TryIt examples are stand-ins: they apply the same rules to a made-up cluster of racks and nodes, so you can see the effects without Hadoop.

The replication factor

The replication factor is the number of copies of each block. The default is 3, set cluster-wide by dfs.replication and changeable per file.

XML
<!-- hdfs-site.xml (read-only example) -->
<property>
  <name>dfs.replication</name>
  <value>3</value>
</property>

Three copies mean raw storage is three times the size of your data: 100 TB of files need 300 TB of disk. In return you can survive node failures and many readers can use different copies in parallel. Important data may get a higher factor, temporary data a lower one.

Racks and rack awareness

Servers sit in racks, and each rack shares a switch and often a power supply. Traffic between two machines in the same rack is fast; traffic between racks goes through more switches and is slower and more limited. A rack can also fail as a whole when its switch or power breaks.

Why racks matter

Same rack Fast network, but shared switch and power.
Other rack Slower link, separate failure domain.

Good placement balances safety and write speed.

1 / 2

HDFS does not detect racks by itself. An administrator provides a topology mapping (a script or a class named in net.topology.script.file.name) that turns a node address into a rack path such as /dc1/rack2. This feature is called rack awareness. Without it, HDFS treats every node as being in one default rack.

The default placement policy

For replication factor 3, HDFS chooses locations like this:

  1. First replica: on the node where the writer runs (if the writer is a DataNode), otherwise on a random node. Writing locally is fastest.
  2. Second replica: on a node in a different rack.
  3. Third replica: on a different node in the same rack as the second.
flowchart LR
  subgraph R1["Rack 1"]
    A["Node A<br/>replica 1"]
    B["Node B"]
  end
  subgraph R2["Rack 2"]
    C["Node C<br/>replica 2"]
    D["Node D<br/>replica 3"]
  end
  A -->|"pipeline"| C
  C -->|"pipeline"| D

Copy one stays near the writer, copies two and three share another rack, so only one cross-rack transfer is needed.

Why this shape? It survives the loss of a whole rack, because the copies exist in two racks. And it needs only one transfer across racks per block (copy 2 travels between racks, copy 3 stays inside rack 2), which keeps writes quick. Putting all three copies in different racks would be safer still but would use much more inter-rack bandwidth. For more than three replicas, additional copies are placed randomly, with a limit on copies per rack.

The following stand-in implements that policy for a small cluster and prints where 6 blocks of a file end up. The "random" node uses a seeded generator, so the output is the same on every run.

Python Placing blocks with rack awareness

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that rack 1 always holds exactly one copy (on the writer's node n1), and the other two copies always share one other rack. This is the placement shape, even though the exact nodes depend on the random generator.

Failure and re-replication

The NameNode knows how many live replicas each block has from DataNode block reports and heartbeats. If a DataNode stops sending heartbeats for about 10.5 minutes (the default: roughly 10 minutes plus a margin derived from heartbeat settings) it is declared dead. All blocks that had a copy on it are now under-replicated, and the NameNode orders other DataNodes to copy those blocks until the target is met again. This is called re-replication, and it happens automatically, in the background, with limits so it does not flood the network.

If the dead node returns, it may hold surplus copies; the NameNode then marks blocks as over-replicated and tells DataNodes to delete the extras. Blocks whose copies are all gone are missing and the files containing them are corrupt; hdfs fsck reports them (Working with HDFS from the Shell).

Here is a simulation of failure and recovery. It builds a cluster, kills one node, counts under-replicated blocks, repairs them with the NameNode's rule (copy from a surviving replica to a node that does not yet hold the block), and reports the result. The placement is simplified (a seeded random choice of three nodes) because the focus is the repair.

Python Node failure and re-replication

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The number of copies made equals the number of blocks that had a replica on n3. Real HDFS also considers racks when it picks the new target, and spreads the copying work over many nodes, so recovery from a node holding many terabytes takes many parallel streams, not one.

Erasure coding

Three-way replication is simple and fast but expensive: 200% overhead (two extra copies per block). For old, rarely read ("cold") data that cost is hard to justify. Erasure coding (EC), available since Hadoop 3, protects data with mathematics instead of full copies.

The idea is similar to RAID. A group of data cells is combined to produce parity cells. The common policy RS-6-3 (Reed-Solomon) splits data into 6 data cells and computes 3 parity cells. Any 6 of the 9 cells are enough to rebuild everything, so you can lose any 3 cells and still recover.

Scheme Failures survived Storage used for 6 units of data
3x replication 2 18 units (200% overhead)
RS-6-3 erasure coding 3 9 units (50% overhead)

So EC survives more failures with half the space. The costs: reading and recovering data needs CPU and network work (rebuilding a lost cell reads several others), files are striped over many nodes, which means small files are a poor fit, and data locality is lost. Hence the rule of thumb: replication for hot data, erasure coding for cold data. You choose a policy per directory (with hdfs ec -setPolicy).

This stand-in shows the principle with the simplest possible code, a single XOR parity cell. Real Reed-Solomon uses more advanced mathematics to support several parity cells, but the "rebuild the missing piece from the rest" idea is the same.

Python Erasure coding idea: one XOR parity cell

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. With the default replication factor of 3, how much raw disk space does 10 TB of files need?

    Each block is stored three times, so raw usage is three times the size of the data.
  2. 2. Where does the default policy put the second replica?

    The second copy goes to another rack so that losing a rack cannot remove all copies. The third copy goes to another node in that second rack.
  3. 3. What happens when a DataNode is declared dead?

    Blocks that lost a replica become under-replicated and are copied in the background until the target factor is met again.
  4. 4. Which statement about RS-6-3 erasure coding is correct?

    6 data cells plus 3 parity cells tolerate any 3 losses at half the space cost of one extra copy, though reconstruction costs CPU and network.

Summary

  • The replication factor (default 3) sets the number of copies; raw storage is that multiple of your data.
  • Rack awareness needs a topology mapping that you supply.
  • Default placement: copy 1 near the writer, copy 2 in another rack, copy 3 on another node in copy 2's rack.
  • Dead DataNodes trigger automatic re-replication of under-replicated blocks.
  • Erasure coding (for example RS-6-3) gives similar safety for 50% overhead, best for cold, large files.

Next: The HDFS Read and Write Path.