Skip to content
Tutorials Center
My learning

Topics

Lesson 8 of 47 ยท HDFS

HDFS Architecture

Learn how HDFS stores huge files: the NameNode and its metadata, DataNodes, large blocks, the write-once model and why a 128 MB block makes sense.

The Hadoop Distributed File System, HDFS, is where Hadoop keeps its data. It looks like an ordinary file system (paths such as /data/logs/2026/01/app.log), but behind that familiar surface a single file may be split into thousands of pieces stored on hundreds of machines. HDFS was designed for one job: storing very large files reliably on cheap hardware and streaming them out fast.

In this lesson you will meet the two kinds of nodes, learn what blocks are and why they are so large, see what the metadata looks like, and understand the write-once model. A runnable stand-in splits files into blocks so you can count them yourself. It follows Hadoop: History, Role Today and Alternatives.

HDFS needs a cluster, so it cannot run on this site. The real configuration in this lesson is read-only. The TryIt examples are stand-ins in Python that reproduce how HDFS splits files and tracks metadata; the numbers (block size, sizes of records) are the real defaults.

Two roles: NameNode and DataNodes

HDFS uses a leader and workers design:

Two roles in HDFS

NameNode Knows the names, folders and where every block lives.
DataNodes Hold the actual block bytes on their disks.

Data never flows through the NameNode, only metadata.

1 / 2
  • The NameNode keeps the namespace: the directory tree, file names, permissions, and, for every file, the ordered list of blocks. It also knows which DataNodes currently hold each block. This metadata is kept in the NameNode's RAM, so lookups are fast.
  • DataNodes store the blocks as ordinary files on their local disks and serve read and write requests. They send the NameNode a regular heartbeat and a block report listing what they hold.

A client that wants to read a file first asks the NameNode "where are the blocks of this file?" and then talks directly to the DataNodes. The NameNode is a directory service, never a data pipe, which is why one NameNode can serve a large cluster. (The HDFS Read and Write Path covers the exact steps.)

flowchart LR
  C["Client"] -->|"1. where are the blocks?"| N["NameNode<br/>(metadata in RAM)"]
  N -->|"2. block locations"| C
  C -->|"3. read bytes"| D1["DataNode 1"]
  C -->|"3. read bytes"| D2["DataNode 2"]
  D1 -.->|"heartbeats, block reports"| N
  D2 -.->|"heartbeats, block reports"| N

Metadata goes through the NameNode; file bytes flow straight between client and DataNodes.

Blocks

HDFS does not store a file as one lump. It cuts the file into fixed-size blocks (128 MB by default, set with dfs.blocksize). A 500 MB file becomes three full 128 MB blocks plus a smaller fourth block holding the remaining 116 MB. The last block takes only as much disk space as it needs: HDFS does not pad it to 128 MB.

XML
<!-- hdfs-site.xml (read-only example) -->
<property>
  <name>dfs.blocksize</name>
  <value>134217728</value>   <!-- 128 MB in bytes -->
</property>

Each block is then replicated (three copies by default) across different DataNodes, as described in Replication and Block Placement. Blocks are the unit of storage, replication and parallel processing: when a job starts, it usually launches one task per block.

Let us split some files. This stand-in computes, for several file sizes, how many blocks a file needs and how big the last block is. It uses the real default block size.

Python Splitting files into HDFS blocks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

tiny.txt shows an important detail: a 3 MB file uses one block of 3 MB, not 128 MB of disk. The block size is a maximum, not a reserved size. But the file still costs the NameNode the same metadata as a larger file, a fact that causes the "small files problem" in NameNode HA, Federation and Operations.

Why blocks are so large

A normal local file system uses blocks of 4 KB. HDFS uses 128 MB, about 32,000 times bigger, on purpose:

  1. Seek time becomes negligible. A disk spends about 10 ms finding data and then reads at roughly 100 MB per second or more. With a 128 MB block, reading takes over a second, so the 10 ms seek is under 1 percent of the time. With tiny blocks, you would be seeking all the time.
  2. Less metadata. The NameNode keeps an entry per block in memory. Large blocks mean far fewer entries for the same amount of data.
  3. Fewer network connections. Clients open one connection per big block, not thousands.
  4. Natural parallel units. A block is a good size of work for one task.

The cost is waste for tiny files (many small files stress the NameNode) and coarse granularity for parallelism on small data.

Try the seek-time argument with the numbers above in this stand-in. It estimates the share of time spent seeking for different block sizes, assuming a 10 ms seek and 100 MB/s reading. These are typical figures, not measurements of your disks.

Python Seek time versus block size

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

At 4 KB (0.004 MB) almost all the time is seeking; at 128 MB it is under one percent.

Metadata: the fsimage and the edit log

The NameNode keeps the whole namespace in memory but must not lose it on a restart. It therefore persists it in two files:

  • fsimage: a snapshot of the whole namespace at some point in time.
  • edit log: an append-only list of every change since then (create, delete, rename...).

On startup the NameNode loads the fsimage, replays the edit log, and then waits for DataNodes to report their blocks. Notice that the block locations are not saved: they are rebuilt from the block reports each time, because DataNodes are the real truth about what they hold. During this period the NameNode is in safe mode and the file system is read-only. Periodically the edit log is merged into a fresh fsimage (a "checkpoint"), so it never grows without limit.

A rule of thumb says each file, directory and block object takes roughly 150 bytes of NameNode heap. This stand-in uses that estimate to show why file count matters.

Python Estimating NameNode memory

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The estimate is rough, but it shows the point: HDFS likes a modest number of big files, not a huge number of tiny ones.

Daemons and cluster types

A running HDFS and YARN cluster is a set of long-lived Java processes called daemons. Knowing which is which makes logs, dashboards and failures much easier to read:

Daemon Belongs to Runs on Job
NameNode HDFS master namespace and block map, in memory
Secondary NameNode (or checkpoint node, standby NameNode) HDFS master or a separate node merges the edit log into a new fsimage (it is not a hot standby)
DataNode HDFS every worker stores and serves blocks, sends heartbeats and block reports
ResourceManager YARN master owns cluster resources and the scheduler
NodeManager YARN every worker starts and watches containers on its machine
JobHistory Server MapReduce master keeps logs and counters of finished jobs

Worker machines usually run a DataNode and a NodeManager together, so computation can start where the data is (data locality). The master daemons (NameNode, ResourceManager) are the parts that need high availability (see NameNode HA, Federation and Operations). Start scripts, the jps check and the web UIs are in Installing Hadoop: Modes, Config Files and Daemons.

Cluster types. A single-node cluster runs all daemons on one machine; replication must be 1, and it is for learning and testing. A multi-node cluster spreads them: in a small one, one master and several workers; in a large one the NameNode, ResourceManager and standbys each get their own machines. Typical properties of a production cluster are:

  • Commodity hardware, planned for failure: disks and nodes die every week at scale.
  • Horizontal scaling: add nodes to get both more storage and more compute.
  • Rack awareness: each node's rack is known so replicas land on different racks (see Replication and Block Placement).
  • Homogeneous workers where possible, which keeps scheduling and capacity planning simple.
  • Long-running, shared use by many teams, which is why YARN queues and quotas matter.

Choosing and changing the block size

The earlier section explained why HDFS blocks are large. The block size is not fixed forever: dfs.blocksize is the default for new files, and each file records the block size it was written with, so a cluster can hold files with different block sizes. You can set it per command:

Terminal
hdfs dfs -D dfs.blocksize=268435456 -put big.log /data/        # this file uses 256 MB blocks
hdfs dfs -stat "%o %r %b" /data/big.log                        # block size, replication, file size in bytes

Changing the default does not rewrite existing files. To change old data you copy it (distcp or a rewrite job) with the new setting. Which size should you pick? A larger block (256 MB or 512 MB) halves the number of blocks, which means less NameNode memory and fewer map tasks, and is good for very large sequential files. A smaller block gives more parallelism for small data and more evenly spread work, but costs metadata. Since a map task usually reads one block, the block size also sets the default task size (see Writing a MapReduce Job in Java). Beyond that, the biggest lever on NameNode memory is not the block size but avoiding tiny files.

Python Block size versus number of blocks and NameNode memory

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Write once, read many

HDFS files follow a write-once, read-many model. You create a file, write it from start to end, and close it. Afterwards you can read it any number of times and append to the end, but you cannot change bytes in the middle. This sounds restrictive, but it removes most consistency problems (see Distributed Systems in Plain Words): there are no concurrent edits to sort out, replicas only ever have the same immutable content, and reading is a simple sequential stream. It matches the batch jobs from What Is Big Data?, which read a big dataset and write a new one.

HDFS is therefore a poor fit for low-latency random reads, many tiny files, or workloads that edit files in place. Systems like HBase build random access on top of it (HBase: Wide-Column Storage).

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does the NameNode store?

    The NameNode holds metadata only. The block data lives on DataNodes, and clients read it from them directly.
  2. 2. A 300 MB file is stored with the default 128 MB block size. How many blocks does it use, and how big is the last one?

    128 + 128 = 256 MB, leaving 44 MB for a third block. The last block uses only the space it needs.
  3. 3. Why are HDFS blocks so large?

    With big blocks the disk mostly streams data instead of seeking, and there are far fewer blocks to track in NameNode memory.
  4. 4. Which operation does HDFS NOT support on an existing file?

    HDFS follows write-once, read-many with append. In-place edits in the middle of a file are not supported.
  5. 5. After a NameNode restart, where do block locations come from?

    Block locations are not persisted. DataNodes report what they hold, and the NameNode rebuilds the map.

Summary

  • HDFS has one NameNode (metadata in RAM) and many DataNodes (block storage).
  • Files are cut into 128 MB blocks by default; the last block uses only the space it needs.
  • Large blocks keep seek overhead tiny and metadata small, but make small files costly.
  • The NameNode persists an fsimage plus an edit log; block locations are rebuilt from block reports.
  • Files are write-once with append, which keeps replicas simple and consistent.

Next: Replication and Block Placement.