Skip to content
Tutorials Center
My learning

Topics

Lesson 22 of 47 ยท MapReduce

Fault Tolerance, Tuning and Limits of MapReduce

How MapReduce survives failures with task retries and speculative execution, how to tune a job, and why disk-based iteration led to Spark replacing it.

A MapReduce job may run thousands of tasks on hundreds of machines for hours. On that scale, something always breaks: a disk fails, a node freezes, a network switch drops packets. MapReduce was designed around this fact. Its tasks are small, independent and deterministic, so any of them can be thrown away and run again.

This lesson covers the failure-handling mechanisms (retries, speculative execution, the ApplicationMaster), the first things to tune in a slow job, and the limits that made newer engines like Spark take over most workloads. It builds on Inside the Shuffle and Sort and YARN: Resources and Scheduling.

Real task failures cannot be reproduced on this site. The examples are Python simulations of retries and stragglers with fixed random seeds, so their results are repeatable. They model the policy, not Hadoop's code.

Failure handling

Who can fail?

Task A map or reduce attempt crashes.
Node The machine or its NodeManager dies.
App master The job's coordinator is lost.
ResourceManager Needs HA to be safe.
1 / 2

Task retries

When a task attempt fails (an exception, a crash, a timeout because it stopped reporting progress), the ApplicationMaster schedules a new attempt, preferably on a different node. Each task may fail up to mapreduce.map.maxattempts / mapreduce.reduce.maxattempts times (default 4). If one task fails four times, the whole job fails: that is a signal of a bug (bad record, code error), not bad luck.

Because a map task reads an immutable HDFS input split and writes only its own temporary output, running it twice is harmless. Reducers write to a temporary directory and the output committer promotes only the successful attempt's files, so a partially written result is never exposed.

Lost nodes and lost map output

Map output lives on the mapper node's local disk (see the shuffle lesson). If that node dies after the map finished but before all reducers have fetched, the output is gone, so the ApplicationMaster re-runs that map task on another node. Reducers that already fetched its output keep it; the rest wait for the new copy.

A node that repeatedly fails tasks of a job is blacklisted for that job, and a node that stops sending heartbeats is declared lost and its running containers are rescheduled.

Bad records

Sometimes one corrupt line crashes the parser on every attempt. Instead of failing the job, you can catch the exception in the mapper, count the record in a counter and skip it. The counters in the final job summary then show how many records were bad, so you can decide whether the result is trustworthy.

Python Task retries with a failing node

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Stragglers and speculative execution

A job finishes only when its slowest task does. A straggler is a task that runs far slower than its siblings because of a dying disk, a busy machine, a slow network card or simply bad luck. In a large job, one straggler can double the total time.

Speculative execution attacks this. When most tasks of a stage have finished and one is clearly slower than average, the ApplicationMaster starts a duplicate attempt on another node. Whichever attempt finishes first wins, and the other is killed. It costs some extra resources, but it cuts the tail.

XML
<property>
  <name>mapreduce.map.speculative</name>
  <value>true</value>
</property>
<property>
  <name>mapreduce.reduce.speculative</name>
  <value>false</value>   <!-- often disabled: duplicates can be costly for reducers -->
</property>

Disable it when tasks have side effects, such as writing to an external database. Two attempts running at the same time would write twice.

Python Straggler and speculative execution

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Job execution, from submission to commit

The previous section described what happens when something fails. To understand why recovery works, it helps to follow one job through its whole life:

  1. Submission. The driver calls job.waitForCompletion(true). The client asks the ResourceManager for a new application id, copies the job JAR, configuration and the computed input splits metadata into a staging directory in HDFS, then submits the application.
  2. ApplicationMaster start. The ResourceManager picks a node and starts a container running the MapReduce ApplicationMaster (MRAppMaster). It reads the split information, creates one map task per split and the configured number of reduce tasks.
  3. Task containers. The ApplicationMaster requests containers (memory and cores) for each task, preferring nodes that hold the split's data. When a NodeManager launches a container, it starts a new JVM process called YarnChild that runs exactly one task attempt.
  4. Progress. Each task reports progress to the ApplicationMaster over a heartbeat-like channel; the client polls the master (or the ResourceManager UI) and prints it.
  5. Commit. When the work is done, output is promoted by the output committer, and the job state becomes SUCCEEDED.

Why each task gets its own JVM

A separate YarnChild JVM per task attempt gives isolation. A bug, memory leak or crash in user code kills only that task's JVM, not the NodeManager and not other tasks. YARN can enforce memory limits per container and kill an offender (the "container killed by YARN for exceeding memory limits" message). The price is JVM startup, a second or two per task, which is acceptable for large tasks and painful for thousands of tiny ones; uber mode runs very small jobs inside the ApplicationMaster's own JVM to avoid that cost. Old Hadoop 1 had a "JVM reuse" setting to amortize startup, which no longer exists in this form.

The output commit protocol

If two attempts of the same task (a retry or a speculative duplicate) both wrote to the final directory, results would collide. So a task never writes there directly. Each attempt writes to its own temporary directory under the output path (_temporary/<attempt>/). When an attempt finishes, it asks the ApplicationMaster for permission to commit; only one attempt per task gets it, and its files are moved ("task commit") to the job's temporary area. When all tasks are committed, the job commit step moves everything into the final output directory and writes the empty _SUCCESS marker. If the job fails, the temporary directories are deleted, and consumers never see half-written output. This is why downstream jobs wait for _SUCCESS, and why object stores, where rename is a slow copy, need special committers (see Filesystems Hadoop Can Use).

Three kinds of task failure

  • Exception or non-zero exit. The JVM prints a stack trace and dies. The ApplicationMaster notices the failed attempt and schedules a new one, normally on another node.
  • JVM crash or kill. The process vanishes (out of memory, a native crash, the container killed for exceeding limits). The NodeManager reports the container as lost; the effect is the same: a new attempt.
  • Hang. The task stays alive but stops making progress (an infinite loop, a blocked call). Tasks must report progress or read or write records; if the ApplicationMaster hears nothing for mapreduce.task.timeout (10 minutes by default), it kills the attempt and starts another. Long computations without I/O should call context.progress() periodically, or they will be killed even though they are healthy.

A task can fail up to four attempts (mapreduce.map.maxattempts, mapreduce.reduce.maxattempts) before the job fails. You can also tolerate a percentage of failed map tasks (mapreduce.map.failures.maxpercent) for jobs where a few bad splits are acceptable, though silently dropping data is rarely wise.

Python Retries, commit and the _SUCCESS marker

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Tuning the first things

When a job is slow, check these in order, because they explain most problems:

  1. Too many or too few map tasks. Thousands of tiny files create thousands of tiny maps with startup overhead each (the small-files problem; see NameNode HA, Federation and Operations). Combine inputs (CombineFileInputFormat) or compact the data.
  2. Add a combiner when aggregating, and compress map output (Snappy/LZ4).
  3. Right-size the reducers: not one reducer for 500 GB, and not 10,000 for 1 GB.
  4. Check skew. If one reducer runs far longer than the rest, the data, not the cluster, is the problem.
  5. Container memory. If tasks are killed for exceeding memory, raise mapreduce.map.memory.mb and keep the JVM heap (-Xmx in mapreduce.map.java.opts) at about 80% of it. Too-small heaps cause spills and garbage collection storms; too-large containers waste capacity.
  6. Use the counters. Spilled Records far above Map output records means the sort buffer is too small; GC time elapsed large means memory trouble; Data-local map tasks low means the data is not where the compute is.
  7. Avoid MapReduce for the wrong shape of problem, as the next section explains.

Limits and why Spark replaced it

MapReduce is excellent at one thing: a single large, simple pass over huge data on cheap disks. Its weaknesses are structural.

  • Disk between every stage. Map output goes to local disk, reduce output to HDFS (three replicas). A pipeline of ten jobs reads and writes HDFS ten times.
  • Iteration is expensive. Machine learning and graph algorithms repeat a computation until it converges (PageRank: tens of passes). In MapReduce every iteration is a new job: scheduling, JVM startup, re-reading the same input.
  • High latency. Starting a job takes seconds to minutes. Interactive queries and streaming are out of reach.
  • Only two stages. Complex logic must be chained into several jobs, and you manage the intermediate files yourself.
  • Verbose code. A join or group-by is many lines of Java.
Python The cost of iterating on disk

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The numbers are made up to show the shape: a per-iteration cost that grows with the number of iterations versus a one-time cost. Spark keeps the working dataset in memory across steps and builds a whole DAG of operations instead of chaining separate jobs; see Spark, Flink and What Came After MapReduce.

MapReduce is not dead, though. Hive on Tez and Spark both borrow its ideas (partitioning, shuffle, combiners), and plain MapReduce still runs simple, stable, huge batch jobs where reliability matters more than speed. Learning it teaches you what every distributed engine does underneath.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A map task fails once with an out-of-memory error. What does the ApplicationMaster do by default?

    Tasks are deterministic and side-effect free, so they can be retried. The job fails only if one task fails four times.
  2. 2. What is speculative execution?

    It reduces the tail time caused by stragglers, at the price of some extra work.
  3. 3. When should you disable speculative execution?

    Two simultaneous attempts have the same side effects. Tasks with external side effects should not run speculatively.
  4. 4. Why is MapReduce slow for iterative algorithms?

    Intermediate results go to disk between jobs, so an algorithm with many passes repeats expensive I/O, which in-memory engines avoid.

Summary

  • Tasks are deterministic and independent, so failed attempts are simply retried (4 attempts by default).
  • Lost map output on a dead node is recomputed; map output is not replicated.
  • Speculative execution launches duplicates of stragglers; disable it for tasks with side effects.
  • Tune in order: input and map count, combiner and compression, reducer count, skew, memory, counters.
  • MapReduce pays disk and startup cost per stage, which is why Spark and Tez took over iterative and interactive work.

Next: Row vs Column Formats.