Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 48 ยท Under the Hood

Fault Tolerance and Lineage

See how Spark survives lost tasks and executors: lineage recomputation, task retries, speculative execution and checkpoints, with runnable stand-ins.

A Spark job can run on hundreds of machines for hours. On that scale something always breaks: a disk fills up, a node is reclaimed by the cloud provider, a network link drops for a minute. If every failure meant starting over, big jobs would almost never finish. Spark is designed so that a failure costs a little extra work, not the whole job.

The idea is simple. Spark does not copy your data again and again for safety. Instead, it remembers how each dataset was built, and when a piece is lost it rebuilds only that piece. This lesson explains that recipe (lineage), the retry rules around it, speculative execution for slow tasks, and checkpoints for very long plans. If you have not met caching yet, read Caching and Persistence first.

Lineage: the recipe for every dataset

Every RDD knows its parents and the function that turns them into it. That chain is its lineage, also called the lineage graph. It is the same plan you met in Lazy Evaluation: textFile, then map, then filter, and so on. DataFrames sit on top of RDDs, so they inherit the same property.

Because transformations are deterministic and the input is immutable, Spark can rerun a lineage at any time and get the same result. When an executor dies and takes some partitions with it, Spark looks at the lineage of only the lost partitions and recomputes them. The other partitions are untouched.

You can print a lineage in PySpark. This is real Spark code, shown read-only:

Python
rdd = (sc.textFile("logs/*.txt")
         .map(lambda line: line.split(","))
         .filter(lambda f: len(f) == 3))

print(rdd.toDebugString().decode())
# (3) PythonRDD[3] at RDD at PythonRDD.scala:53 []
#  |  logs/*.txt MapPartitionsRDD[1] at textFile at <unknown>:0 []
#  |  logs/*.txt HadoopRDD[0] at textFile at <unknown>:0 []

The indentation shows parents. A line such as ShuffledRDD with a different indent level marks a shuffle dependency, which matters for recovery in a moment.

Narrow and wide dependencies

Recovery cost depends on the kind of dependency:

  • Narrow dependency: each output partition depends on one (or a few fixed) input partitions, as in map or filter. Losing a partition means recomputing a single chain on one machine.
  • Wide dependency: an output partition depends on many input partitions, as in groupByKey or join. Losing one means fetching pieces from every upstream partition. Spark avoids repeating that work by keeping the shuffle files written by the map side on local disk. If the files are still there, only the reduce side reruns. If the executor holding them died, the map tasks that wrote them must run again.
flowchart LR
    A["Partition 1<br/>read"] --> B["Partition 1<br/>map + filter"]
    C["Partition 2<br/>read"] --> D["Partition 2<br/>map + filter"]
    E["Partition 3<br/>read"] --> F["Partition 3<br/>map + filter (LOST)"]
    B --> G["reduceByKey<br/>(shuffle)"]
    D --> G
    F -.->|"recompute only this chain"| E
    F --> G

Only the lost partition's narrow chain is rebuilt from its source; the other partitions keep their results.

The stand-in below records the steps applied to each partition. We delete one partition's result and ask for it again: the log shows that only the lost partition is recomputed, from the source.

Python Recomputing one lost partition from its lineage

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Task failures and retries

Most failures are small: one task hits a corrupt block, a timeout, or a lost connection. The driver's task scheduler reacts in layers:

  1. Retry the task. A failed task is resubmitted, usually on another executor. The setting is spark.task.maxFailures (default 4 in cluster mode). In local mode the default is 1, unless you write local[4, 8] to allow 8 failures.
  2. Fail the stage and the job. If one task fails that many times, the stage is aborted, and so is the job. You then see Job aborted due to stage failure: Task 7 in stage 3.0 failed 4 times.
  3. Re-run upstream stages. If a reduce task cannot fetch shuffle data (FetchFailedException), Spark assumes the map output is gone. It marks those map outputs as lost and re-runs the earlier stage for the missing partitions, then continues. A stage may be retried a limited number of times (spark.stage.maxConsecutiveAttempts, default 4).
  4. Executor loss. When an executor disappears, its running tasks fail (they count as retries, but not against maxFailures if the loss was not the task's fault, for example a preempted node), its cached partitions disappear, and its shuffle files may be gone too. A cluster manager such as YARN or Kubernetes usually starts a replacement executor.

One failure is not retried: an exception thrown by your own code that is deterministic (a division by zero on a certain row) fails the same way four times, then kills the job. Retries fix bad luck, not bad code.

The driver is a single point of failure. If the driver process dies, the application ends. Cluster managers can restart it (--supervise in standalone mode, spark.yarn.maxAppAttempts on YARN) and Structured Streaming can resume from a checkpoint, but a batch job normally starts over. Keep the driver on a reliable machine and avoid collecting huge results into it.

Speculative execution

Sometimes a task is not failing, just slow: a machine with a bad disk, a noisy neighbor, a flaky network card. Because a stage finishes only when its last task finishes, one such straggler can double the stage time.

With spark.speculation=true, Spark watches running tasks. Once a configurable share of tasks has finished (spark.speculation.quantile, default 0.75), it compares each remaining task with the median finished duration. A task that has run longer than spark.speculation.multiplier times the median (default 1.5) gets a speculative copy launched on another executor. Whichever copy finishes first wins; the other is killed.

Speculation is off by default because it can waste resources and, worse, it is unsafe for tasks with side effects. Two copies of the same task may both write output, so use it only with idempotent writes, such as writing files through a committer that keeps only one attempt's output. It does not help with data skew, where a task is slow because it has more data: the copy would be just as slow. Skew is covered in Performance Tuning.

The next stand-in simulates both rules. Attempts fail on a fixed schedule so the output is the same every run: a task that fails twice and then succeeds, a task that always fails and aborts the job, and a straggler that a speculative copy beats.

Python Task retries, a doomed task and speculative execution

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Checkpoints: cutting long lineages

Lineage recovery has a weak spot: long lineages. An iterative algorithm that applies one more transformation per loop (graph algorithms, machine-learning loops, a streaming query with state) builds a plan with hundreds of steps. Recomputing from the very start after a failure may take hours, and the plan itself can overflow the driver's stack when it is serialized.

A checkpoint writes the data to reliable storage (HDFS, S3 and the like) and cuts the lineage: the new RDD's parent is the file, not the old chain. After a failure Spark reloads from storage instead of recomputing. Checkpointing costs a write, so use it deliberately:

Python
sc.setCheckpointDir("hdfs:///checkpoints/job-42")

ranks = initial_ranks
for i in range(100):
    ranks = step(ranks)
    if i % 10 == 9:
        ranks.checkpoint()   # mark; the file is written when the next action runs
        ranks.count()        # an action triggers the write

Two points to remember:

  • checkpoint() is lazy. Spark computes the RDD once for the action and then a second time to write it, which is why the usual advice is to cache() the RDD before checkpointing it.
  • persist is not the same thing. Cached data is kept in executor memory or local disk and lives only as long as the executors do. A checkpoint is durable and survives the application. For DataFrames, df.checkpoint() is eager by default and returns a new DataFrame; localCheckpoint() is cheaper but stores data on executors, so it gives up fault tolerance.

The stand-in counts how many steps must be re-executed to rebuild a lost result at step 30, with and without a checkpoint every 10 steps.

Python How a checkpoint shortens recovery

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real versus stand-in. Real Spark tracks lineage as RDD dependency objects, retries tasks through its TaskSetManager, launches speculative copies from a periodic check on the driver, and writes checkpoint files through the Hadoop file system API. The stand-ins use lists, loops and arithmetic with a fixed failure schedule so the logic can be seen in a few lines. No executors, files or network are involved.

What Spark does not protect you from

  • Non-deterministic code. If a transformation uses random() or the current time and is recomputed, the new result differs from the lost one. Seed random generators from the partition index, or checkpoint the data.
  • Side effects. A recomputed task repeats writes to a database or an API. Make output idempotent (upserts, unique keys, overwrite).
  • Driver loss and cluster-wide failures. Use cluster-manager restarts and, for streaming, checkpoint directories.
  • Lost input. Lineage ends at the source. If the source files are deleted, there is nothing to recompute from.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. An executor dies and takes two cached partitions with it. What does Spark do?

    Lineage tells Spark how each partition was built, so it reruns just the chain needed for the missing partitions. Other partitions keep their results.
  2. 2. A task throws the same exception on every attempt because of a bug in your function. What happens?

    Retries help with transient problems. A deterministic bug fails the same way each time, and after the limit the job is aborted.
  3. 3. Which problem does speculative execution address?

    A speculative copy runs on another executor and the first to finish wins. If the task is slow because of its data, the copy is just as slow.
  4. 4. Why checkpoint an RDD in an iterative algorithm?

    A checkpoint stores the data durably and replaces the parent chain with the file, so recovery and plan serialization stay cheap.

Summary

  • Spark recovers by recomputing lost partitions from their lineage, not by replicating data.
  • Narrow dependencies are cheap to rebuild; wide dependencies rely on shuffle files, and a lost map output means re-running the earlier stage.
  • Failed tasks are retried (spark.task.maxFailures, default 4 on a cluster); deterministic bugs still kill the job.
  • Speculative execution launches a copy of a slow task; enable it only with idempotent output.
  • Checkpoints write to reliable storage and cut long lineages; cache before you checkpoint.
  • Keep code deterministic and writes idempotent so recomputation is safe.

Next: leave batch behind and process live data in Structured Streaming Basics.