Skip to content
Tutorials Center
My learning

Topics

Lesson 15 of 47 ยท YARN and Resource Management

Capacity and Fair Schedulers

Compare the FIFO, Capacity and Fair schedulers in YARN, learn queues, preemption and how to size containers and tune memory settings, with a fair-share allocator.

A cluster is shared. A nightly ETL job, an analyst's ad-hoc query and a data-science experiment all want the same CPUs and memory at the same moment. Someone has to decide who goes first and how much each gets. In YARN that is the job of the scheduler, a pluggable component inside the ResourceManager.

This lesson compares three schedulers (FIFO, Capacity, Fair), explains queues and preemption, and shows how to size containers and set the memory options that cause most "container killed" errors. It builds on YARN: Resources and Scheduling.

The configuration below is real but read-only: there is no cluster on this site. The runnable examples are stand-in models of the scheduling arithmetic. Real schedulers also handle locality, priorities, user limits and many more details.

FIFO: the simplest scheduler

The FIFO scheduler puts applications in one queue and serves them in arrival order. Each job takes what it wants until it is satisfied, and the next job starts afterwards. It is trivial to understand and fine for a single user, but one huge job blocks everyone behind it, so it is almost never used on shared clusters.

Capacity Scheduler: guaranteed shares per team

The Capacity Scheduler (the default in Apache Hadoop and in many distributions) divides the cluster into a hierarchy of queues. Each queue gets a guaranteed capacity, a percentage of the cluster, and can borrow idle resources from others up to a maximum.

XML
<!-- capacity-scheduler.xml -->
<property>
  <name>yarn.scheduler.capacity.root.queues</name>
  <value>etl,analytics,adhoc</value>
</property>
<property><name>yarn.scheduler.capacity.root.etl.capacity</name><value>50</value></property>
<property><name>yarn.scheduler.capacity.root.analytics.capacity</name><value>30</value></property>
<property><name>yarn.scheduler.capacity.root.adhoc.capacity</name><value>20</value></property>
<property><name>yarn.scheduler.capacity.root.adhoc.maximum-capacity</name><value>40</value></property>
<property><name>yarn.scheduler.capacity.root.adhoc.user-limit-factor</name><value>1</value></property>
  • capacity is the guaranteed share. The capacities of sibling queues add up to 100.
  • maximum-capacity caps how much a queue may grow by borrowing idle resources (elasticity).
  • user-limit-factor stops one user from monopolizing a queue.
  • Inside a queue, applications are by default served FIFO, so each queue is its own small FIFO.
  • Access is controlled with acl_submit_applications, so only the right people can use a queue.

The idea is the one organizations like: "the ETL team has half the cluster no matter what, and if analytics is idle, ETL may use its share too." After editing the file you refresh without restarting: yarn rmadmin -refreshQueues.

Fair Scheduler: everyone gets an equal share over time

The Fair Scheduler (the default in Cloudera's older CDH) aims to give every running application, or every queue, an equal share of resources over time. When only one job runs, it gets the whole cluster; when a second arrives, the resources are gradually rebalanced so each holds half, and so on. Short jobs finish quickly even when a long job is running, which is great for interactive use.

It also supports queue hierarchies, minimum shares, weights (a queue with weight 2 gets twice a weight-1 queue's share) and policies per queue: fair, fifo or drf.

DRF (Dominant Resource Fairness) handles several resource types. A job that is memory-heavy and one that is CPU-heavy are compared by their dominant share, meaning whichever resource uses the largest fraction of the cluster. Equalizing dominant shares keeps both kinds of job treated fairly.

Three schedulers

FIFO First come, first served. Simple, unfair.
Capacity Guaranteed percent per team, can borrow.
Fair Equal shares over time.

Capacity and Fair converge; both have queues, weights and preemption.

1 / 2

Preemption: taking resources back

Elasticity has a catch. Suppose adhoc borrows the whole idle cluster, then etl submits a job. The borrowed containers are busy, so etl must wait for them to finish, which may take hours. Preemption solves it: the scheduler waits a configured time, then kills containers of queues that are over their share and gives the resources back to queues that are below their guarantee.

Preemption wastes the work in the killed containers, so schedulers first send the AM a warning so it can finish or save progress, and they choose the most recently started containers to lose the least work. It is enabled with yarn.resourcemanager.scheduler.monitor.enable (Capacity) or yarn.scheduler.fair.preemption (Fair). Short-lived tasks like MapReduce maps recover easily from preemption; long-running services may not.

A fair-share allocator

The core of fair sharing is max-min fairness: split resources equally, but if a queue needs less than its share, give the leftover to the others. The stand-in below computes it, optionally with weights.

Python Max-min fair share with weights

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that analytics asks for only 20, below its equal share of 33.3, so the surplus flows to the hungry queues.

The second stand-in compares FIFO with fair sharing on the same workload: a long job arrives first and a short one right after. It uses a simple model with a cluster that can run 4 tasks at once, in time steps.

Python FIFO versus fair: when does the short job finish?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Fair sharing lets the short job finish within a couple of steps, while FIFO makes it wait behind the long job. The long job finishes at nearly the same time either way, because the total work is the same.

Sizing containers and tuning memory

Most real-world scheduler trouble comes from container sizes. The key settings are:

Setting Meaning
yarn.nodemanager.resource.memory-mb Memory one node offers to YARN (leave room for the OS and DataNode)
yarn.scheduler.minimum-allocation-mb Smallest container; requests are rounded up to a multiple
yarn.scheduler.maximum-allocation-mb Largest container a single request may have
mapreduce.map.memory.mb / mapreduce.reduce.memory.mb Container size for map and reduce tasks
mapreduce.map.java.opts JVM options such as -Xmx1638m for the task

The classic mistake: asking for a 2 GB container but giving the JVM -Xmx2g. The JVM also needs memory outside the heap (metaspace, thread stacks, native buffers), so the process exceeds 2 GB and the NodeManager kills it with "Container killed by YARN for exceeding memory limits". A common rule is to set the heap to about 80% of the container, so a 2048 MB container gets -Xmx1638m.

Other practical guidance:

  • If you give a node 64 GB for YARN and use 4 GB containers, it runs at most 16 of them, fewer if cores run out. Pick sizes so memory and vcores are both used.
  • Containers that are too small cause out-of-memory errors and spills; too large waste capacity and reduce parallelism.
  • Check actual usage in the job history before tuning, and raise one thing at a time.
Python How many containers fit on a node?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does the Capacity Scheduler give each queue?

    Each queue has a guaranteed capacity and a maximum-capacity limit for elastic borrowing.
  2. 2. Why is preemption needed?

    Without preemption, a queue that borrowed idle capacity could hold on to it for hours while the owner waits.
  3. 3. A 4 GB container runs a JVM with -Xmx4g and gets killed. Why?

    The JVM uses memory outside the heap too. Set -Xmx to roughly 80% of the container size.
  4. 4. What does the Fair Scheduler aim for?

    Fair sharing rebalances resources as jobs come and go, so short jobs finish quickly next to long ones.

Summary

  • FIFO serves one job at a time in arrival order; it suits only single-user clusters.
  • The Capacity Scheduler gives queues guaranteed percentages with elastic borrowing and ACLs.
  • The Fair Scheduler shares resources equally (or by weight, or by dominant share with DRF) over time.
  • Preemption kills containers of over-using queues to restore guarantees.
  • Container memory must cover the JVM heap plus overhead: heap about 80% of the container size.
  • Size containers so both memory and vcores are used on each node.

Next: The MapReduce Programming Model.