Skip to content
Tutorials Center
My learning

Topics

Lesson 6 of 47 ยท Big Data Basics

Hadoop 2 vs Hadoop 3

Compare Hadoop 2 and Hadoop 3: erasure coding, multiple standby NameNodes, Java 8 baseline, intra-DataNode balancer, new ports, YARN features and cloud connectors.

Hadoop 2 (released in 2013) introduced YARN and made Hadoop a general platform. Hadoop 3 (generally available in late 2017) kept that architecture and improved the parts that hurt most in production: storage cost, availability, resource management and cloud use. You will meet both versions: older clusters and tutorials use Hadoop 2 conventions, while new deployments run the 3.x line. This lesson lists the differences that matter and explains why each exists.

Hadoop cannot run on this site, so configuration snippets are read-only. The runnable examples are small Python stand-ins that compute storage costs and translate Hadoop 2 settings to their Hadoop 3 equivalents. For background on terms such as NameNode and replication, see HDFS Architecture and Replication and Block Placement.

The short comparison

Area Hadoop 2.x Hadoop 3.x
Minimum Java Java 7 (later 8) Java 8 or newer
Storage protection 3x replication only (200% overhead) 3x replication or erasure coding (about 50% overhead)
NameNode HA one active + one standby one active + two or more standbys
Disk balancing balancer moves data between DataNodes adds an intra-DataNode disk balancer
YARN memory and CPU as resources adds resource types (GPU, FPGA), opportunistic containers, Docker containers, Timeline Service v2
Default web ports NameNode 50070, DataNode 50075 NameNode 9870, DataNode 9864
Shell scripts original scripts rewritten (--daemon options, fewer bugs, better classpath handling)
Cloud storage basic S3 and Azure connectors stronger S3A, ABFS (Azure Data Lake Gen2), Aliyun and others
Federation ViewFs only adds Router-based federation

Everything else (HDFS blocks, the MapReduce API, the YARN architecture, Hive and Spark on top) works the same way, so code written for Hadoop 2 usually runs unchanged on Hadoop 3. The upgrade work is mostly in operations: ports, Java and scripts.

Erasure coding: cheaper storage

Three-way replication is simple and fast but triples the disk space. Erasure coding (EC) stores data as k data blocks plus m parity blocks computed from them; any k of the k + m blocks can rebuild the data. The common policy is RS-6-3: 6 data and 3 parity units, so the file occupies 1.5 times its size instead of 3 times, and it survives the loss of any 3 units, the same fault tolerance as replication or better.

The trade-offs are real. EC spreads a file's blocks across many nodes, so reads and writes touch more machines; reconstruction after a failure needs network traffic and CPU; and small files do not benefit (a file smaller than one stripe still needs all the parity cells, so a 1 MB file stores 4 MB, worse than replication). EC is therefore used for cold, large data (archives, old partitions) while hot data stays replicated. You choose a policy per directory:

Terminal
hdfs ec -listPolicies
hdfs ec -setPolicy -path /archive/2022 -policy RS-6-3-1024k
hdfs ec -getPolicy -path /archive/2022
Python Storage cost: replication versus erasure coding

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Multiple standby NameNodes

In Hadoop 2 HDFS HA has one active NameNode and one standby; if the active fails and you are already repairing the standby, you are one failure from an outage. Hadoop 3 allows several standbys (typically 2 or 3), so an operator can take one down for maintenance and still have failover protection. The same idea applies to the YARN ResourceManager. See NameNode HA, Federation and Operations for how failover works.

Disk balancing inside a DataNode

The classic HDFS balancer evens out usage between DataNodes. A DataNode with 12 disks can still have a few full disks and several empty ones, for example after a disk replacement. Hadoop 3's intra-DataNode disk balancer (hdfs diskbalancer) plans and executes moves between the disks of one node, which removes hot spots and improves read speed.

YARN improvements

YARN in Hadoop 3 treats resources more generally: resource types let you request GPUs or FPGAs next to memory and CPU, which made Hadoop usable for machine-learning workloads. Opportunistic containers can start on a node even when it is full, running at low priority, to improve utilization. Docker support runs containers inside Docker images, isolating dependencies between jobs. Timeline Service v2 is a scalable replacement for the application history store. See YARN: Resources and Scheduling.

Java, ports and scripts

Hadoop 3 needs Java 8 or later, which lets it use newer language features and libraries (and later 3.x releases support newer Java versions; check the release notes). Several default ports moved out of the Linux ephemeral port range (32768 and up) where they could clash with outgoing connections, which is why the NameNode UI is on 9870 instead of 50070. If a tutorial tells you to open :50070 and you run Hadoop 3, that is why nothing answers.

Python Translating Hadoop 2 settings to Hadoop 3

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The rewritten shell scripts also changed commands: in Hadoop 2 you started a single daemon with hadoop-daemon.sh start namenode; in Hadoop 3 use hdfs --daemon start namenode. The slaves file was renamed workers.

Cloud connectors

Hadoop 3 improved access to object stores: S3A gained faster listing, committers designed for object stores (S3 has no cheap rename, see Filesystems Hadoop Can Use), and the ABFS driver for Azure Data Lake Storage Gen2 arrived. This is part of why Hadoop tools run well against cloud storage even when no HDFS cluster exists.

Should you upgrade?

If you run a supported distribution, you probably already have Hadoop 3. Move older clusters to the 3.x line for security fixes and for the features above, but test first: some old ecosystem components, scripts that hard-code ports or Java 7-era libraries need changes. A rolling upgrade path exists for HDFS, and you can adopt erasure coding gradually on cold directories.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A 600 TB dataset uses RS-6-3 erasure coding. Roughly how much raw disk does it need?

    RS-6-3 stores 9 units for every 6 units of data, a factor of 1.5, so 600 TB needs about 900 TB. Three-way replication would need 1,800 TB.
  2. 2. Which workload is erasure coding best suited for?

    EC saves space on big, rarely read data. It has more network and CPU cost on reads and recovery and does not help small files.
  3. 3. What does Hadoop 3 allow for HDFS high availability that Hadoop 2 did not?

    Hadoop 2 supports one standby. Hadoop 3 supports multiple standbys.
  4. 4. Why does a Hadoop 2 tutorial's http://master:50070 not work on Hadoop 3?

    Hadoop 3 moved several default ports out of the ephemeral range; the NameNode UI is on 9870.

Summary

  • Hadoop 3 keeps the Hadoop 2 architecture (HDFS, YARN, MapReduce) and improves storage cost, availability and resource management.
  • Erasure coding (for example RS-6-3) cuts overhead from 200% to 50% for cold data at the price of more network and CPU on reads and recovery.
  • HA supports multiple standby NameNodes; the intra-DataNode disk balancer evens out disks inside one node.
  • YARN gained resource types such as GPU, opportunistic containers, Docker support and Timeline Service v2.
  • Ports, scripts (--daemon, workers) and the Java baseline changed; stronger S3A and ABFS connectors help in the cloud.

Next: set up your own cluster in Installing Hadoop: Modes, Config Files and Daemons.