Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 47 ยท Data Ingestion and Processing Models

Spark, Flink and What Came After MapReduce

See how Spark's RDDs, DataFrames and lazy DAGs replaced MapReduce, how Flink does true streaming, and what Apache Beam adds, with a lazy-evaluation demo.

MapReduce made large-scale processing possible, but each job writes its intermediate results to disk and every multi-step pipeline is a chain of separate jobs. Iterative algorithms and interactive analysis suffered. Apache Spark, started at UC Berkeley, kept data in memory between steps, offered a much friendlier API, and became the default engine of the Hadoop world. Apache Flink approached the same problem from the streaming side.

This lesson explains the ideas behind Spark (RDDs, DataFrames, lazy evaluation and the DAG), what Flink and Beam add, and why so much has moved beyond MapReduce. It follows Batch vs Stream Processing.

Spark and Flink do not run on the site. The PySpark snippets are read-only. The runnable example is a small Python stand-in that imitates lazy evaluation and the DAG, so you can see what Spark does when you call a transformation or an action.

What Spark changed

Why Spark won

In memory Keep data between steps, not on disk.
One API Batch, SQL, streaming and ML together.
Whole-plan view Optimizes the full DAG before running.
Friendly Python, Scala, Java, SQL and R.

Spark can run on YARN, Kubernetes or standalone.

1 / 2

In MapReduce, a pipeline of five steps means five jobs, and each reads from and writes to HDFS (see Fault Tolerance, Tuning and Limits of MapReduce). In Spark, the five steps are five transformations in one job, and data flows between them in memory, spilling to disk only when needed. For iterative workloads, such as machine learning that loops over the same data dozens of times, this can be many times faster.

RDDs: the original abstraction

The first core abstraction is the RDD (Resilient Distributed Dataset): an immutable collection of records split into partitions spread across the cluster. You create one from a file or from another RDD and apply operations in parallel.

Python
# read-only PySpark (needs a Spark cluster)
lines = sc.textFile("hdfs:///data/raw/weblogs/2026/01/01/*")
errors = lines.filter(lambda l: " 500 " in l)
pages = errors.map(lambda l: (l.split()[6], 1))
counts = pages.reduceByKey(lambda a, b: a + b)
top = counts.takeOrdered(5, key=lambda kv: -kv[1])

There are two kinds of operations:

  • Transformations (map, filter, flatMap, reduceByKey, join) build a new RDD from an old one. They are lazy: nothing is computed yet.
  • Actions (count, collect, take, saveAsTextFile) ask for a result. Only now does Spark run the job.

Transformations come in two types. Narrow ones (map, filter) work on each partition independently, so Spark can pipeline many of them in a single pass over the data. Wide ones (reduceByKey, groupByKey, join) need a shuffle, the same all-to-all data exchange you saw in Inside the Shuffle and Sort. A shuffle ends one stage and begins the next.

Lineage gives fault tolerance

Spark does not replicate in-memory data. Each RDD remembers its lineage, the recipe of transformations that built it. If a node dies and a partition is lost, Spark simply recomputes that partition from its parents (all the way back to HDFS if needed). You can cache (persist) an RDD that you will reuse, so it is not recomputed.

DataFrames and Spark SQL

Most new Spark code uses DataFrames: distributed tables with named, typed columns, like a pandas DataFrame or a SQL table. Because Spark knows the schema and the operations, its Catalyst optimizer can rewrite your query (push filters down, prune unused columns, choose join strategies) and its Tungsten engine generates compact binary code. This is the same idea as the Hive optimizer, applied to an in-memory engine, and it works very well with Parquet.

Python
# read-only PySpark
df = spark.read.parquet("hdfs:///lake/curated/orders")
result = (df.filter(df.status == "paid")
            .groupBy("country")
            .agg({"amount": "sum"})
            .orderBy("sum(amount)", ascending=False))
result.show(10)
result.explain()     # prints the physical plan with the optimizations applied

spark.sql("SELECT country, SUM(amount) FROM orders WHERE status = 'paid' GROUP BY country")

The DataFrame and the SQL query compile to the same plan. In practice you write DataFrames or SQL and touch RDDs only for unusual low-level work.

The DAG and lazy evaluation

When you call an action, Spark looks at the whole chain of transformations as a directed acyclic graph (DAG) and splits it into stages at the shuffle boundaries. Each stage becomes a set of tasks, one per partition, scheduled on executors. Laziness is what makes this possible: since Spark sees the whole plan before running anything, it can combine steps, skip work that no action needs, and avoid reading data it will filter away.

flowchart LR
    R["read files"] --> F["filter"]
    F --> M["map"]
    M -->|shuffle| G["reduceByKey"]
    G --> A["action: take(5)"]

Stage 1 pipelines read, filter and map; the shuffle starts stage 2.

The stand-in below builds such a lazy plan. Calling .map() and .filter() only records steps. The work happens when .collect() or .count() is called, and narrow steps are fused so each record flows through the whole chain in one pass.

Python A lazy pipeline: nothing runs until an action

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

You will see "transformations defined; nothing was read yet" first, and the reading record lines only after ACTION. Records are read once and flow through square and keep even together, which is exactly what narrow-transformation pipelining means.

A second stand-in shows lineage recovery: when a partition is "lost", it is rebuilt from its recipe.

Python Lineage: rebuild a lost partition

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Only partition 1 is recomputed, not the whole data set. The cost is proportional to the lost data.

Spark Structured Streaming

Spark also handles streams. Structured Streaming treats a stream as an unbounded table that keeps growing and runs your DataFrame query on new data in small micro-batches (with a continuous mode for lower latency). It supports event-time windows and watermarks with the same DataFrame API as batch, which is the main attraction: one set of code for both.

Python
# read-only PySpark Structured Streaming
clicks = (spark.readStream.format("kafka")
          .option("kafka.bootstrap.servers", "broker1:9092")
          .option("subscribe", "clicks").load())
counts = (clicks.withWatermark("timestamp", "5 minutes")
                .groupBy(window("timestamp", "1 minute")).count())
counts.writeStream.outputMode("update").format("console").start()

Apache Flink was designed around streams. Where Spark's streaming is traditionally micro-batch, Flink processes each event as it arrives, in a long-running dataflow of operators with managed state (counts, sessions, joins kept in an embedded, checkpointed store). Flink takes consistent checkpoints of all operator state plus input positions, so after a failure it restarts from the last checkpoint and, with a transactional sink, produces exactly-once results. It has first-class support for event time, watermarks, session windows and very low latency. A batch job in Flink is just a stream that ends.

Rule of thumb: Spark is the general workhorse for batch, SQL, machine learning and moderate-latency streaming; Flink is the stronger choice for stateful, low-latency, always-on streaming applications.

Beam: write once, run on many engines

Apache Beam is not an engine but a programming model: you describe a pipeline with PCollections (bounded or unbounded), transforms, windows and triggers, and choose a runner (Spark, Flink, Google Dataflow) to execute it. It aims to unify batch and streaming and to avoid lock-in, at the cost of an extra abstraction layer.

So what happened to MapReduce?

Most new work uses Spark, Flink, or SQL engines such as Trino, Hive on Tez or cloud warehouses. MapReduce is still found in older pipelines, and its ideas live everywhere: partitioned data, shuffles, combiners and moving code to data. Understanding it, as in The MapReduce Programming Model, makes the newer engines much easier to understand.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In Spark, which of these is an action?

    count returns a result to the driver, so it triggers execution. filter, map and reduceByKey are lazy transformations.
  2. 2. How does Spark recover a lost partition of an in-memory RDD?

    Each RDD records the transformations that produced it, so only the missing partition has to be rebuilt from its parents.
  3. 3. What ends a stage in a Spark job?

    Narrow transformations are pipelined inside one stage. A wide one such as reduceByKey needs a shuffle and begins a new stage.
  4. 4. Which statement about Flink is most accurate?

    Flink processes events continuously and keeps checkpointed state. The model that runs on several engines is Beam.

Summary

  • Spark keeps intermediate data in memory and runs a whole multi-step pipeline as one job, which is far faster than chained MapReduce jobs for iterative work.
  • Transformations are lazy; actions trigger the run. Narrow steps are pipelined, and wide steps shuffle and split stages in the DAG.
  • Lineage, not replication, gives RDD fault tolerance; DataFrames and Spark SQL add schemas and the Catalyst optimizer.
  • Flink is streaming first with managed state and checkpoints; Beam is a portable model that runs on several engines.
  • MapReduce's concepts live on in every newer engine.

Next: Data Lakes, Warehouses and Lakehouses.