Skip to content
Tutorials Center
My learning

Topics

Apache Spark Tutorial

Learn Apache Spark step by step: 48 lessons in 10 chapters with examples, quizzes, diagrams and projects.

Start learning →

of lessons completed

Getting Started

  1. 1 What Is Apache Spark? Learn what Apache Spark is, why it exists, how it differs from MapReduce, Hadoop and pandas, what its libraries do and when you should (and should not) use it.
  2. 2 Spark Architecture Understand how Spark runs a program: the driver, executors, cluster manager, jobs, stages, tasks and the DAG, with a runnable driver and executor simulator.
  3. 3 Installing Spark and Your First PySpark Program Install PySpark with pip, run Spark in local mode, create a SparkSession, use spark-shell, notebooks and Docker, and write your first DataFrame program.
  4. 4 Lazy Evaluation, Transformations and Actions Learn why Spark is lazy, the difference between transformations and actions, narrow and wide transformations, and lineage, with a runnable lazy pipeline.

RDDs

  1. 5 RDD Basics Learn what a Resilient Distributed Dataset is, how to create RDDs with parallelize and textFile, why they are immutable, and how partitions and lineage work.
  2. 6 RDD Transformations Learn the core RDD transformations map, filter, flatMap, distinct, union, sample and sortBy, which ones shuffle, and try them in a runnable MiniRDD.
  3. 7 RDD Actions Learn the Spark RDD actions collect, count, take, first, reduce, fold, aggregate, foreach and saveAsTextFile, plus the collect pitfalls, with runnable stand-ins.
  4. 8 Key-Value (Pair) RDDs Learn Spark key-value RDDs: reduceByKey, groupByKey, aggregateByKey, combineByKey, join, cogroup and countByKey, and why reduceByKey beats groupByKey.
  5. 9 Broadcast Variables and Accumulators Learn how Spark ships closures to executors, why driver variables are copied, and how broadcast variables and accumulators share data safely, with runnable stand-ins.

DataFrames and Spark SQL

  1. 10 DataFrames: Creating and Inspecting Learn how to create Spark DataFrames, read the schema, inspect data with show and printSchema, use Row objects, load files and decide between RDD and DataFrame.
  2. 11 Sampling and Getting Rows Back Take samples of a Spark DataFrame and bring rows back safely: sample, sampleBy, takeSample, first, head, tail, take, collect and toLocalIterator.
  3. 12 Columns and Expressions Learn how Spark DataFrame columns and expressions work: select, withColumn, col, expr, alias, cast, when/otherwise, drop and rename, with pandas stand-ins.
  4. 13 Filtering, Sorting and Deduplicating Learn to filter rows, sort with orderBy, remove duplicates with distinct and dropDuplicates, use limit and handle nulls correctly in Spark DataFrames.
  5. 14 Cleaning Data: Nulls, Fill, Replace and Outliers Clean Spark DataFrames: find nulls, drop rows or columns, fill per column, replace values and remove outliers with the IQR rule, with runnable stand-ins.
  6. 15 Grouping and Aggregations Summarize Spark DataFrames with groupBy and agg, count and sum nulls correctly, build pivot tables, subtotals with rollup and cube, and approximate distinct counts.
  7. 16 Pivot, Unpivot and Union Reshape Spark DataFrames: pivot rows into columns, unpivot wide tables back, and stack tables with union or unionByName and allowMissingColumns.
  8. 17 Reading and Writing Data Read and write CSV, JSON, Parquet, ORC, Avro and JDBC data in Spark, choose write modes and partitionBy, and know when to use an explicit schema.
  9. 18 Spark SQL and Temp Views Query Spark DataFrames with plain SQL: register temp views, call spark.sql, use the catalog and CTEs, compare SQL with the DataFrame API and learn ANSI mode.
  10. 19 Joins Combine Spark DataFrames with inner, outer, semi, anti and cross joins, write safe join conditions, avoid duplicate columns and handle null keys.
  11. 20 Window Functions Use Spark window functions: partitionBy and orderBy, row_number, rank, dense_rank, lag and lead, running totals, and rows versus range frames with examples.
  12. 21 Arrays, Maps, Structs and JSON Work with nested data in Spark DataFrames: arrays, maps and structs, explode and posexplode, array functions, struct access, from_json and to_json.
  13. 22 User-Defined Functions and Pandas UDFs Learn when to write a Spark UDF, why Python UDFs are slow, how pandas UDFs use Arrow to work on batches, and why built-in functions should come first.
  14. 23 Datasets, Schemas and Data Types Understand Spark schemas: StructType, nullability, common data types, typed Datasets in Scala and Java, DataFrame versus Dataset, and schema evolution.
  15. 24 pandas API on Spark Use the pandas API on Spark (pyspark.pandas) to scale pandas code, convert with toPandas and Arrow safely, and know which pandas habits break at scale.

Under the Hood

  1. 25 Catalyst Optimizer and Query Plans See how Spark turns DataFrame code into logical and physical plans, how Catalyst pushes filters and prunes columns, and how to read explain() output.
  2. 26 Partitions and Partitioning Learn how Spark splits data into partitions, how to choose a partition count, repartition versus coalesce, hash and range partitioners, and bucketing.
  3. 27 Shuffles and Stages Understand what triggers a Spark shuffle, how map and reduce sides exchange data, how stages are cut, what spill means, and how to read it in the Spark UI.
  4. 28 Caching and Persistence Learn when to cache a Spark DataFrame or RDD, how cache differs from persist, what storage levels mean, how to unpersist, and when checkpointing is better.
  5. 29 Fault Tolerance and Lineage See how Spark survives lost tasks and executors: lineage recomputation, task retries, speculative execution and checkpoints, with runnable stand-ins.

Streaming

  1. 30 Structured Streaming Basics Learn the Structured Streaming model: an unbounded table, sources, sinks, triggers, output modes and checkpoints, with a runnable micro-batch simulator.
  2. 31 Event Time, Windows and Watermarks Group streaming events by when they happened: tumbling and sliding windows, late data, watermarks, state cleanup and exactly-once output, with a simulator.
  3. 32 Streaming with Kafka and Delta Read from and write to Kafka with Structured Streaming, manage offsets, use foreachBatch with idempotent writes, and learn how Delta Lake logs commits.
  4. 33 Legacy Spark Streaming: DStreams Understand the legacy DStream API: micro-batches, socketTextStream, windowed counts, updateStateByKey, checkpoints and why Structured Streaming replaced it.

MLlib

  1. 34 MLlib Overview and Pipelines Learn Spark MLlib: Transformer, Estimator and Pipeline, VectorAssembler, train/test split, cross-validation and saving models, with runnable stand-ins.
  2. 35 MLlib Classification: Logistic Regression and ROC-AUC Train a binary classifier with Spark MLlib: StringIndexer, VectorAssembler, LogisticRegression, thresholds, BinaryClassificationEvaluator and ROC-AUC.
  3. 36 MLlib Regression: Linear Regression Walkthrough Predict numbers with Spark MLlib LinearRegression: feature vectors, train/test split, coefficients, RMSE, R2, residuals and ridge, lasso and elastic net.
  4. 37 MLlib Clustering: K-Means and Silhouette Group unlabeled data with Spark MLlib KMeans: scale features, choose k with the silhouette score, read cluster centers and assign clusters to rows.

Performance and Production

  1. 38 Performance Tuning Make Spark jobs faster: find data skew and fix it with salting, use broadcast joins and AQE, size shuffle partitions, tune memory and Kryo, avoid small files.
  2. 39 Deploying Spark: YARN, Kubernetes, Standalone and Cloud Learn how to deploy Spark applications: spark-submit, client and cluster deploy modes, executor sizing, dynamic allocation, packaging and managed cloud services.
  3. 40 Testing, Debugging and the Spark UI Learn to unit test Spark transformations, run tests in local mode, read the Spark UI, find skew and fix common errors such as OOM and serialization failures.

Advanced Topics

  1. 41 Graph Processing with GraphX and GraphFrames Process graphs on Spark: property graphs, vertices and edges, GraphX in Scala, GraphFrames in PySpark, PageRank and connected components, with stand-ins.

Projects

  1. 42 Project: Web Server Log Analysis Build a log analysis job step by step: parse access logs, count bad lines, rank top URLs, find error spikes and write results, with PySpark code and runnable stand-ins.
  2. 43 Project: A Bronze-Silver-Gold ETL Pipeline Build a medallion ETL pipeline in PySpark: raw bronze data, cleaned and deduplicated silver tables, gold aggregates, quality checks and partitioned output.
  3. 44 Project: Real-Time Event Counter Build a real-time page view counter with Structured Streaming: running counts, one-minute windows, late events, watermarks and output modes, with runnable stand-ins.
  4. 45 Project: Movie Recommendations with ALS Build a movie recommender with Spark MLlib ALS: ratings matrix, latent factors, train and test split, RMSE, top-N recommendations and cold start, with a NumPy stand-in.

Practice and Review

  1. 46 Practice: Exercises with Solutions 12 Apache Spark practice exercises with solutions covering RDDs, DataFrames, aggregations, joins and window functions, with runnable Python stand-ins.
  2. 47 Interview Questions Common Apache Spark interview questions with short answers on architecture, lazy evaluation, shuffles, joins, partitions, skew, tuning and streaming.
  3. 48 Spark Cheat Sheet A one-page Apache Spark and PySpark cheat sheet: RDD, DataFrame, SQL, joins, window functions, key configs and spark-submit flags, with runnable stand-ins.