Skip to content
Tutorials Center
My learning

Topics

Lesson 1 of 48 ยท Getting Started

What Is Apache Spark?

Learn what Apache Spark is, why it exists, how it differs from MapReduce, Hadoop and pandas, what its libraries do and when you should (and should not) use it.

Imagine a log file of 40 GB, or a table with ten billion rows. Your laptop cannot hold it in memory, and a single for loop would run for hours. The usual answer is to split the data into pieces, process the pieces on many machines at the same time, and combine the partial answers. Doing that by hand is painful: you must move data around, handle machines that crash, and merge results correctly.

Apache Spark is an open-source engine that does this work for you. You describe what to compute with a high-level API in Python, Scala, Java, R or SQL, and Spark decides how to split the job, run it in parallel on a cluster (or on the cores of one machine), and recover when something fails. This tutorial teaches Spark from the first idea to production tuning, using Python (PySpark) for most examples.

The problem Spark solves

Data grows faster than single machines do. When a dataset no longer fits in one computer's memory or disk, or when one computer is simply too slow, you have two options:

  • Scale up: buy a bigger machine. This works until it gets absurdly expensive or hits a hardware ceiling.
  • Scale out: use many ordinary machines together. This is cheaper and practically unlimited, but now you need software to coordinate them.

Spark is that coordination software. It treats a collection of machines as one big computer: you write a few lines, and Spark turns them into hundreds of small tasks that run side by side.

flowchart LR
    D["Big dataset"] --> S["Split into partitions"]
    S --> W1["Worker 1<br/>process part"]
    S --> W2["Worker 2<br/>process part"]
    S --> W3["Worker 3<br/>process part"]
    W1 --> M["Combine partial results"]
    W2 --> M
    W3 --> M
    M --> R["Final answer"]

Split, process in parallel, combine: the pattern behind almost every Spark job.

Spark, MapReduce and Hadoop

Before Spark, the standard tool was Hadoop MapReduce. Hadoop is really a family of pieces: HDFS (a distributed file system), YARN (a resource manager) and MapReduce (a processing model). MapReduce writes the result of every step to disk before the next step starts. That is safe but slow, especially for jobs with many steps, such as machine learning that loops over the same data again and again.

Spark was started at UC Berkeley's AMPLab around 2009 and became a top-level Apache project in 2014. Its key idea is to keep intermediate data in memory between steps when possible and to describe the whole job as a graph of operations that it can optimize before running anything. For many workloads this is much faster than MapReduce, and the API is far more pleasant than writing mapper and reducer classes.

Spark does not replace everything in the Hadoop world. It is a processing engine. It can read from HDFS and run on YARN, but it also works with Amazon S3, Azure storage, Google Cloud Storage, local files, JDBC databases, Kafka and more.

Spark compared with pandas

If you know pandas, Spark's DataFrame API will feel familiar, but the two tools solve different problems.

pandas Spark
Where it runs One machine, one process A cluster, or many cores
Data size Must fit in RAM Can be far larger than RAM
Execution Eager: each line runs now Lazy: a plan is built and runs on demand
Startup Instant Seconds (it starts a JVM and a scheduler)
Best for Up to a few GB, exploration Many GB to petabytes, pipelines

The last row matters. For small data, pandas is faster and simpler, because Spark pays a fixed cost to start and coordinate. Spark wins only when the data or the computation is big enough that parallelism outweighs that cost.

The Spark ecosystem

Spark has one core engine and several libraries that sit on top of it and share the same cluster, scheduler and memory management.

flowchart TB
    SQL["Spark SQL<br/>DataFrames"] --> Core
    ML["MLlib<br/>machine learning"] --> Core
    SS["Structured Streaming<br/>continuous data"] --> Core
    GX["GraphX<br/>graphs (Scala/Java)"] --> Core
    Core["Spark Core<br/>RDDs, scheduling, memory, fault tolerance"]

Every library is built on Spark Core, so you can mix them in one program.

  • Spark Core provides the basic data abstraction (the RDD, see RDD Basics), task scheduling and fault tolerance.
  • Spark SQL and DataFrames let you query structured data with SQL or a table-like API. This is what most people use day to day, and it has its own optimizer, Catalyst.
  • Structured Streaming processes data that keeps arriving, such as events from Kafka, using the same DataFrame API as batch jobs.
  • MLlib offers scalable machine-learning algorithms and pipelines.
  • GraphX handles graph computations. It is only available from Scala and Java; Python users often reach for the separate GraphFrames package.

Where Spark fits, and where it does not

Spark is a good choice when:

  • The data is large (tens of GB or more) or grows quickly.
  • You run ETL pipelines that clean, join and aggregate large tables.
  • You need to process streams and batches with the same code.
  • You train models on data too big for one machine.

Spark is probably the wrong choice when:

  • The data fits comfortably in memory on one machine. Use pandas, Polars or DuckDB.
  • You need single-row lookups in milliseconds. That is a job for a database.
  • You need very low latency per event (microseconds to a few milliseconds). Spark's streaming works in small batches and has more overhead.

A good rule: do not reach for a cluster until a single machine has clearly failed you.

A first stand-in: split, apply, combine

The real Spark cannot run inside this page: it needs a JVM and a cluster scheduler. So every lesson here shows the real PySpark code in read-only blocks and pairs it with a small runnable stand-in written in plain Python that mimics the idea. The stand-in is a teaching model, not Spark.

Here is the classic word count in real PySpark. Read it; it will make full sense after the RDD lessons.

Python
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("wordcount").getOrCreate()
lines = spark.sparkContext.textFile("logs/*.txt")

counts = (lines.flatMap(lambda line: line.lower().split())
               .map(lambda word: (word, 1))
               .reduceByKey(lambda a, b: a + b))

print(counts.take(5))

Spark splits the input files into partitions, each worker counts words in its own partitions, then the partial counts for the same word are merged. The stand-in below does the same in three explicit steps, with a plain list standing in for each partition.

Python Word count by split, apply, combine

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that each worker only needed its own partition to produce a useful partial answer. That independence is what lets Spark run the work on hundreds of machines at once. Only the small partial results travel to be combined.

The second stand-in measures why the pattern matters: it runs the same job over more and more "partitions" and confirms the answer never changes.

Python The answer does not depend on how you split

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

This property, that a combine step gives the same result however the data was split, is why operations such as sums and counts parallelize so well. Operations that cannot be combined this way (for example, an exact median) need more care, and you will meet them later.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the main reason Spark can be faster than Hadoop MapReduce for multi-step jobs?

    MapReduce writes the output of each step to disk. Spark can keep intermediate results in memory and optimizes the whole job before running it.
  2. 2. For a 200 MB CSV file that fits easily in memory, which tool is usually the better choice?

    Spark pays a fixed startup and coordination cost. For data that fits on one machine, single-machine tools are simpler and often faster.
  3. 3. Which Spark library lets you process continuously arriving data with the DataFrame API?

    Structured Streaming applies the DataFrame and SQL API to unbounded data such as events from Kafka.
  4. 4. Why does the split-apply-combine pattern scale to many machines?

    Because partitions are processed independently, adding machines adds capacity. Only the compact partial results are merged at the end.

Summary

  • Spark is an open-source engine that splits big data into partitions, processes them in parallel and combines the results.
  • It started at UC Berkeley, became an Apache top-level project in 2014, and keeps intermediate data in memory where it can.
  • Hadoop is a family of tools (HDFS, YARN, MapReduce); Spark is a processing engine that can use parts of it.
  • Spark SQL and DataFrames, Structured Streaming, MLlib and GraphX all build on Spark Core.
  • Choose Spark for large data and pipelines, not for data that fits on one machine or for millisecond lookups.
  • On this site, real Spark code is read-only and the runnable examples are plain-Python stand-ins.

Next: see how a Spark program is actually executed in Spark Architecture.