Skip to content
Tutorials Center
My learning

Topics

Lesson 7 of 47 ยท Big Data Basics

Installing Hadoop: Modes, Config Files and Daemons

Understand Hadoop's three modes, the core-site, hdfs-site, yarn-site and mapred-site files, formatting the NameNode, starting daemons and web UIs, on Linux and Windows.

You do not need a data center to learn Hadoop. The same software runs on one laptop and on a thousand machines; what changes is the mode and a handful of XML files. This lesson explains the three modes, the four configuration files you will edit most, the commands that format and start a cluster, and the web pages that show it is alive. It also covers what is different on Windows.

Hadoop cannot run on this site, so the commands and XML below are read-only. The runnable example is a small Python stand-in that reads a configuration like the ones shown and tells you which mode it describes and what is missing. Read The Hadoop Ecosystem at a Glance first if the names HDFS, YARN and MapReduce are new.

The three modes

Mode Daemons File system Use it for
Standalone (local) none: everything runs in one JVM the local disk (file:///) testing a MapReduce program in seconds
Pseudo-distributed all daemons on one machine, each in its own JVM HDFS on that machine learning, development, demos
Fully distributed daemons spread over many machines HDFS across the cluster real clusters

Standalone mode is Hadoop's default right after you unpack it, with no configuration at all. A job reads and writes ordinary local files, and there is no NameNode, no YARN and no web page. It is perfect for checking that your mapper and reducer logic is right.

Pseudo-distributed mode is "a cluster of one". You start the real NameNode, DataNode, ResourceManager and NodeManager as separate processes, so everything behaves like production: HDFS paths, the YARN scheduler, the web UIs. Only the scale is missing, and the replication factor must be 1 because there is just one DataNode.

Fully distributed mode adds a workers file that lists the machines running DataNodes and NodeManagers, passwordless SSH from the master so the scripts can start remote daemons, and identical configuration on every node.

Prerequisites

Hadoop needs a Java runtime (the 3.3 and 3.4 lines run on Java 8 and 11; check the release notes of your version before choosing), a dedicated Linux user, and SSH for the start scripts, even on one machine. Set JAVA_HOME in etc/hadoop/hadoop-env.sh, and HADOOP_HOME plus PATH in your shell profile. Check the install with hadoop version.

Terminal
hadoop version                       # prints the version and build
hadoop fs -ls /                      # in standalone mode this lists the local root
hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar wordcount input output

The four configuration files

All configuration lives in $HADOOP_HOME/etc/hadoop/. Each file is XML with <property> entries. The defaults are in the *-default.xml files inside the JARs; your *-site.xml files override them.

core-site.xml: settings shared by everything. The most important is the default file system:

XML
<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://localhost:9000</value>       <!-- the NameNode's address -->
  </property>
</configuration>

hdfs-site.xml: HDFS behavior:

XML
<configuration>
  <property><name>dfs.replication</name><value>1</value></property>   <!-- 1 for a single machine -->
  <property><name>dfs.namenode.name.dir</name><value>/data/hadoop/nn</value></property>
  <property><name>dfs.datanode.data.dir</name><value>/data/hadoop/dn</value></property>
</configuration>

yarn-site.xml: the resource manager and node managers:

XML
<configuration>
  <property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value></property>
  <property><name>yarn.resourcemanager.hostname</name><value>master</value></property>
</configuration>

mapred-site.xml: tells MapReduce to run on YARN instead of locally:

XML
<configuration>
  <property><name>mapreduce.framework.name</name><value>yarn</value></property>
</configuration>

Without mapreduce.framework.name=yarn, jobs keep running in the local runner even though a cluster is up, which is a classic "why is my job not on the cluster?" mistake. For a multi-node cluster also edit the workers file (called slaves in Hadoop 2), one host per line. Remember that data directories default to /tmp, which is wiped on reboot, so always set dfs.namenode.name.dir and dfs.datanode.data.dir to a durable path.

Python A configuration checker

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Format, start, check

The first time only, format the NameNode. This creates an empty namespace (the fsimage and edit log) in dfs.namenode.name.dir:

Terminal
hdfs namenode -format           # once, ONLY on a new cluster: reformatting an existing one destroys its metadata

Then start the daemons with the scripts in sbin/:

Terminal
start-dfs.sh                     # NameNode, DataNode(s), SecondaryNameNode (or checkpoint node)
start-yarn.sh                    # ResourceManager, NodeManager(s)
mapred --daemon start historyserver   # the JobHistory server (Hadoop 3 syntax)
jps                              # lists the running Java processes

jps should show NameNode, DataNode, SecondaryNameNode, ResourceManager and NodeManager. If one is missing, read its log in $HADOOP_HOME/logs/; the usual causes are a wrong JAVA_HOME, SSH to localhost not working, ports already in use, or the DataNode refusing a NameNode that was re-formatted (a cluster ID mismatch). Stop everything with stop-yarn.sh and stop-dfs.sh. To use Hadoop beyond the shell, see Working with HDFS from the Shell.

This stand-in reads a pasted jps listing and tells you which daemons are missing for a pseudo-distributed cluster:

Python Which daemons are missing?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Web UIs

Component Hadoop 3 port Hadoop 2 port
NameNode 9870 50070
DataNode 9864 50075
ResourceManager 8088 8088
NodeManager 8042 8042
JobHistory server 19888 19888

Open http://localhost:9870 to see live nodes, capacity and a file browser, and http://localhost:8088 to watch applications. The port change between versions is a frequent source of outdated tutorials, covered in Hadoop 2 vs Hadoop 3.

Linux, Windows and containers

Hadoop was written for Linux, and production clusters are Linux. On Windows, native support is limited: the Hadoop binaries need helper files (winutils.exe, hadoop.dll) that are not part of the official release, and the shell scripts assume a POSIX environment. The reliable choices are WSL 2 (a real Linux inside Windows, so you follow the Linux steps exactly) or Docker, where a ready-made image starts pseudo-distributed Hadoop in one command. On macOS, brew install hadoop works for standalone and pseudo-distributed use. Keep memory in mind: a pseudo-distributed cluster with its default heap settings needs a few gigabytes of RAM.

Real versus stand-in. The checker above only inspects configuration text. Real Hadoop also validates paths, ports, permissions and the cluster ID, and the daemons fail at start time with their own log messages. Use the checker's logic as a mental checklist, not as a substitute for the logs.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In which mode does Hadoop run with no daemons, using the local file system?

    Standalone (local) mode runs a job inside one JVM on local files. It is Hadoop's default and good for testing logic.
  2. 2. A cluster is running but MapReduce jobs do not show up in the ResourceManager UI. Which property is the likely cause?

    Without mapreduce.framework.name=yarn, jobs use the local runner and never go to YARN.
  3. 3. When should you run hdfs namenode -format?

    Formatting creates an empty namespace. Running it on an existing cluster discards the metadata that describes all stored files.
  4. 4. What is the recommended way to run Hadoop on a Windows laptop?

    Official support for native Windows is limited, so a Linux environment through WSL 2 or a container is the dependable option.

Summary

  • Standalone mode runs in one JVM on local files, pseudo-distributed runs every daemon on one machine, and fully distributed spreads them over a cluster.
  • core-site.xml (fs.defaultFS), hdfs-site.xml (replication, data directories), yarn-site.xml and mapred-site.xml (mapreduce.framework.name=yarn) override defaults; add a workers file for multiple nodes.
  • Format the NameNode once, then start-dfs.sh, start-yarn.sh and check with jps and the web UIs (9870 and 8088 in Hadoop 3).
  • Set durable data directories; the defaults point to /tmp.
  • On Windows prefer WSL 2 or Docker.

Next: see what changed between releases in Hadoop 2 vs Hadoop 3.