Skip to content
Tutorials Center
My learning

Topics

Lesson 14 of 47 ยท YARN and Resource Management

YARN: Resources and Scheduling

Understand YARN, Hadoop's resource manager: ResourceManager, NodeManagers, containers, the ApplicationMaster and the life of an application, with a simulation.

HDFS stores the data; something must decide which program runs where, and how much CPU and memory each one gets. In Hadoop that something is YARN, short for Yet Another Resource Negotiator. YARN turns a pile of machines into one shared pool of compute, so that MapReduce, Spark, Tez, Flink and other engines can all run on the same cluster at the same time.

This lesson covers the parts of YARN (ResourceManager, NodeManager, container, ApplicationMaster) and follows one application from submission to completion. It comes after NameNode HA, Federation and Operations and sets up Capacity and Fair Schedulers.

YARN cannot run here. The commands and configuration are read-only. The runnable examples are Python stand-ins that model the allocation logic, not the real daemons or protocols.

Why YARN exists

Hadoop 1 had a JobTracker that did two jobs at once: it managed cluster resources and scheduled every MapReduce task. It was a bottleneck at a few thousand nodes, it could only run MapReduce, and its fixed "map slots" and "reduce slots" often sat idle while the other kind was starved.

Hadoop 2 split the two concerns:

  • Resource management became a general, central service: YARN.
  • Application logic (how to split a job into tasks, retry them, track progress) moved into a per-application program called the ApplicationMaster.

The result: one cluster, many frameworks, and much better utilization because resources are flexible containers instead of rigid slots.

The parts

The cast of YARN

ResourceManager One per cluster. Owns all resources.
NodeManager One per machine. Runs containers.
Container A slice of CPU and memory.
ApplicationMaster One per job. Asks for containers.

The ResourceManager schedules; it does not run your code.

1 / 2

ResourceManager (RM)

The RM is the cluster-wide authority. It has two main parts: the Scheduler, which decides who gets which resources according to queue rules (next lesson), and the ApplicationsManager, which accepts job submissions and starts each application's first container. The scheduler is a pure scheduler: it does not monitor progress or restart failed tasks. Like the NameNode, the RM can run as an active/standby pair with ZooKeeper-based failover.

NodeManager (NM)

Each worker machine runs a NodeManager. At startup it tells the RM how many resources it offers, for example 64 GB of memory and 16 virtual cores (yarn.nodemanager.resource.memory-mb, yarn.nodemanager.resource.cpu-vcores). It sends heartbeats, launches and monitors containers, tracks their resource usage, and kills a container that exceeds its memory allowance.

Container

A container is an allocation of resources on one node: "2048 MB and 1 vcore on node-7". It is the unit YARN grants. Inside it a process runs (a map task, a Spark executor, the ApplicationMaster itself). On Linux, containers are isolated with cgroups or by monitoring, so a greedy job cannot eat its neighbours' memory.

ApplicationMaster (AM)

Every submitted application gets its own AM, running inside the first container. It is framework-specific code: the MapReduce AM knows about map and reduce tasks and splits; the Spark AM knows about executors. The AM:

  1. negotiates containers from the RM ("I need 20 containers of 2 GB, preferably on these nodes");
  2. talks to NodeManagers to start tasks in the granted containers;
  3. tracks task progress and retries failed tasks;
  4. unregisters when finished, releasing everything.

Because the AM is per application, a failed job's coordinator cannot bring down the cluster.

The life of an application

sequenceDiagram
    participant C as Client
    participant RM as ResourceManager
    participant NM1 as NodeManager 1
    participant NM2 as NodeManager 2
    C->>RM: submit application
    RM->>NM1: start ApplicationMaster container
    NM1-->>RM: AM running
    NM1->>RM: AM requests 2 containers
    RM-->>NM1: grants on NM1 and NM2
    NM1->>NM2: AM launches task container
    NM2-->>NM1: task progress
    NM1->>RM: finish and unregister

The AM asks the RM for containers, then launches tasks directly on NodeManagers.

In words:

  1. The client submits the application (jar, command to start the AM, resources it needs) to the RM.
  2. The RM picks a node and allocates the first container for the AM; that NodeManager starts it.
  3. The AM registers with the RM and sends resource requests: how many containers, how big, locality preferences (a specific node, a rack, or anywhere).
  4. The RM's scheduler grants containers as capacity is available, returned to the AM in its next heartbeat.
  5. The AM contacts the NodeManagers to launch tasks in the granted containers.
  6. The tasks run and report progress to the AM; the client can poll the AM or RM for status.
  7. When done, the AM unregisters, its containers are released, and logs are aggregated (often to HDFS).

The application states you will see in yarn application -list are NEW, SUBMITTED, ACCEPTED (waiting for the AM container), RUNNING, then FINISHED, FAILED or KILLED.

Terminal
yarn application -list -appStates RUNNING
yarn application -status application_1700000000000_0042
yarn logs -applicationId application_1700000000000_0042
yarn application -kill application_1700000000000_0042
yarn node -list -all

Resource requests and locality

A request has a size (memory, vcores), a count and a locality. The scheduler tries, in order: the node holding the data (node-local), another node in the same rack (rack-local), then anywhere (off-switch). If the preferred node is busy, YARN waits a short time hoping a slot opens before relaxing the preference (delay scheduling). Node-local tasks read from their own disk instead of the network, which is the whole reason Hadoop moves computation to the data.

Memory is a hard limit; CPU is usually a share. Asking for more than yarn.scheduler.maximum-allocation-mb is rejected, and requests are rounded up to a multiple of yarn.scheduler.minimum-allocation-mb. The next lesson shows how to choose these values.

A container allocation simulation

The stand-in below has three NodeManagers and a ResourceManager that grants containers on request, preferring the node holding the data. The ApplicationMaster asks for container after container, and when no node can fit one, the request has to wait.

Python The ResourceManager grants containers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

You can see the preferred node get filled up and later requests land elsewhere. A real scheduler would also wait a little before giving up on locality, which this model omits.

Lifecycle with failures

Failures are routine. If a task container dies, the AM asks for a new one and retries the task (MapReduce tries up to 4 times). If a NodeManager stops heartbeating (10 minutes by default), the RM marks the node lost and the AMs reschedule the work that ran there. If the AM dies, the RM restarts it (up to yarn.resourcemanager.am.max-attempts, default 2), and a MapReduce AM recovers finished tasks from its history instead of redoing them.

Python Application states and a failed container

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which YARN part runs on every worker machine and launches containers?

    NodeManagers offer a machine's resources, start containers and watch their usage. The ResourceManager runs once per cluster.
  2. 2. What does the ApplicationMaster do?

    The AM is per application. It asks the RM for containers, launches tasks on NodeManagers and retries failures.
  3. 3. A container is best described as...

    A YARN container is a resource allocation on a node, in which a task or executor process runs.
  4. 4. Why did Hadoop 2 introduce YARN?

    The Hadoop 1 JobTracker did both jobs and only supported MapReduce. YARN is a general resource layer.

Summary

  • YARN is Hadoop's cluster resource manager; many engines share one cluster through it.
  • The ResourceManager schedules, NodeManagers run containers, a container is memory plus vcores on a node, and each application has its own ApplicationMaster.
  • Applications go NEW, SUBMITTED, ACCEPTED, RUNNING, then FINISHED, FAILED or KILLED.
  • Requests carry size, count and locality (node-local, rack-local, off-switch); delay scheduling trades a short wait for locality.
  • Failed tasks are retried by the AM, lost nodes are rescheduled, and the AM itself can be restarted.

Next: Capacity and Fair Schedulers.