Skip to content
Tutorials Center
My learning

Topics

Lesson 3 of 48 ยท Getting Started

Installing Spark and Your First PySpark Program

Install PySpark with pip, run Spark in local mode, create a SparkSession, use spark-shell, notebooks and Docker, and write your first DataFrame program.

The fastest way to learn Spark is to run it. You do not need a cluster: Spark has a local mode that runs everything inside one process on your laptop. This lesson shows the common ways to install it, explains the single object every program starts with (the SparkSession), and walks through a first complete program.

Installing and running Spark needs a JVM and a local process, so it cannot happen inside this page. The commands and programs below are read-only. Each section is followed by a runnable stand-in (a mock SparkSession written in plain Python) so you can practice the shape of the code here.

What you need

Spark runs on the Java Virtual Machine, so you need a Java runtime. Recent Spark releases support Java 17 and 21; check the version table of the release you install, because the supported range changes over time. Python is only needed for PySpark, and it should be a supported 3.x version.

Check Java first:

Terminal
java -version

If it prints a version that your Spark release supports, you are ready. Otherwise install a JDK (for example Temurin) and make sure JAVA_HOME points at it.

Option 1: pip install pyspark

The simplest route is a Python virtual environment:

Terminal
python -m venv .venv
source .venv/bin/activate        # on Windows: .venv\Scripts\activate
pip install pyspark
python -c "import pyspark; print(pyspark.__version__)"

The pyspark package on PyPI bundles the Spark jars, so no separate Spark download is needed. It is enough for learning, testing and local development. You can also start an interactive shell with the pyspark command that the package installs.

Option 2: download a Spark distribution

For spark-shell, spark-submit and cluster work, download a prebuilt Spark archive from the official Apache Spark download page, unpack it, and set two environment variables:

Terminal
tar -xzf spark-<version>-bin-hadoop3.tgz
export SPARK_HOME=$PWD/spark-<version>-bin-hadoop3
export PATH="$SPARK_HOME/bin:$PATH"

spark-shell      # Scala shell
pyspark          # Python shell
spark-submit --version

spark-shell gives you a Scala prompt with a ready-made spark variable. pyspark gives you the same in Python. Both start in local mode unless you pass --master.

Option 3: notebooks and Docker

Spark works well in notebooks. After pip install pyspark jupyterlab, create a SparkSession in the first cell and run everything else in later cells. Hosted notebooks (Databricks, Google Colab, cloud notebooks) usually give you a session already created and called spark.

If you prefer not to install Java and Python yourself, run a ready-made container image that bundles Spark and Jupyter:

Terminal
docker run -it --rm -p 8888:8888 -p 4040:4040 quay.io/jupyter/pyspark-notebook

Port 8888 serves Jupyter and port 4040 serves the Spark UI for the running session. Open the UI whenever a job feels slow.

The SparkSession

SparkSession is the entry point of every modern Spark program. It owns the connection to the cluster (or local engine), your configuration, and the methods that create DataFrames and run SQL. You build it with a builder:

Python
from pyspark.sql import SparkSession

spark = (SparkSession.builder
         .appName("first-program")
         .master("local[*]")
         .config("spark.sql.shuffle.partitions", "8")
         .getOrCreate())

print(spark.version)
  • appName names the application in the Spark UI.
  • master("local[*]") runs locally with one thread per core. On a real cluster you normally leave master out of the code and pass it with spark-submit, so the same script runs in both places.
  • config(...) sets options. Here we lower shuffle.partitions because the default (200) is far too many tasks for a tiny local dataset.
  • getOrCreate() returns the existing session if there is one, which makes it safe to call twice.

Behind a session there is also a lower-level SparkContext, reachable as spark.sparkContext, used for RDDs.

flowchart LR
    A["Your script"] --> B["SparkSession.builder"]
    B --> C["getOrCreate()"]
    C --> D["spark.read / spark.sql<br/>DataFrames"]
    C --> E["spark.sparkContext<br/>RDDs"]
    C --> F["spark.stop()"]

One session gives you DataFrames, SQL and the RDD context; stop it when you are done.

Your first program

Here is a complete first program: create a small DataFrame, add a column, filter, aggregate and show the results. Read it now and look back after the DataFrame lessons.

Python
from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.appName("first").master("local[*]").getOrCreate()

orders = spark.createDataFrame(
    [("alice", "books", 12.5), ("bob", "games", 40.0),
     ("alice", "games", 25.0), ("carol", "books", 8.0)],
    ["customer", "category", "amount"],
)

result = (orders
          .withColumn("amount_with_tax", F.round(F.col("amount") * 1.2, 2))
          .filter(F.col("amount") > 10)
          .groupBy("category")
          .agg(F.sum("amount").alias("revenue")))

result.show()
spark.stop()

You would run it with python first.py (if pyspark is pip-installed) or spark-submit first.py. Nothing happens until show(), which is the action; see Lazy Evaluation, Transformations and Actions.

Stand-in: a mock SparkSession

The class below imitates only the surface of the code above: a builder, a createDataFrame that returns a table, and filter, groupBy().sum() and show() methods. It is built on plain lists and has no cluster behind it. Use it to get a feel for the chained style.

Python A mock SparkSession and DataFrame

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The output table has the same look as result.show() in real Spark. What is real in the stand-in is the chained, builder-style API and the table layout. What is not is everything underneath: the real DataFrame is distributed, lazy and optimized, while the mock runs eagerly on one Python list.

Configuration and the Spark UI

While a real session is running, open http://localhost:4040. It lists jobs, stages, tasks, storage and the SQL plans of your queries. If port 4040 is busy because another session is running, Spark tries 4041, 4042 and so on. The UI disappears when the session ends; to keep history, enable the event log and run the history server (see Testing, Debugging and the Spark UI).

A few options you will meet often:

Setting What it controls
spark.master Where to run (local[*], yarn, k8s://...)
spark.driver.memory Memory for the driver
spark.executor.memory Memory per executor
spark.sql.shuffle.partitions Partitions after a shuffle (default 200)

You can set them in the builder, with --conf key=value on spark-submit, or in conf/spark-defaults.conf.

Notebooks, Colab and version checks

Notebooks are the friendliest way to explore Spark, and the setup differs a little by platform.

Jupyter on your machine. Install Java, then pip install pyspark jupyterlab and start a notebook. The first cell creates the session:

Python
from pyspark.sql import SparkSession

spark = (SparkSession.builder
         .master("local[*]")
         .appName("notebook")
         .config("spark.sql.shuffle.partitions", "8")
         .getOrCreate())

If you downloaded a Spark distribution instead of using pip, Python cannot find it by itself. The small findspark package fixes that by adding Spark's Python folders to sys.path:

Python
import findspark
findspark.init("/opt/spark")           # or findspark.init() if SPARK_HOME is set
import pyspark

Skip findspark when you installed through pip; it is only needed for a manually unpacked Spark.

Google Colab and Kaggle. Both already ship Python and (on Colab) a Java runtime, so pip install pyspark followed by the same SparkSession code is usually enough. If Java is missing, install it first with the platform's package manager (on Colab, !apt-get install -y openjdk-17-jdk-headless -qq). These sessions are temporary and have limited memory, so use local[*], small data and expect to reinstall each session. Pin the version (pip install pyspark==4.0.0, for example) so notebooks stay reproducible.

Hosted platforms (Databricks, EMR notebooks, Dataproc) create spark for you; calling getOrCreate() returns that session, so the same code works everywhere.

Checking versions

When something fails, the first question is always "which versions?":

Python
import sys, pyspark
print("Python :", sys.version.split()[0])
print("PySpark:", pyspark.__version__)
print("Spark  :", spark.version)                              # the Spark that actually runs
print("Scala  :", spark.sparkContext._jvm.scala.util.Properties.versionString())
print("Java   :", spark.sparkContext._jvm.System.getProperty("java.version"))
Terminal
spark-submit --version                  # Spark and Scala versions from the command line
java -version
python --version

pyspark.__version__ is the Python package; spark.version is the Spark engine. They normally match, but on a cluster the engine may be a different build, which is a common cause of confusing errors. Each Spark release supports specific Java and Python versions, so check the release notes before upgrading one of them.

Python A version compatibility check

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Common first-run problems

  • JAVA_HOME is not set or "Unsupported class file major version": the Java version is missing or not supported by your Spark release. Install a supported JDK.
  • Very slow on a tiny dataset: lower spark.sql.shuffle.partitions for local work.
  • Address already in use warnings for port 4040: harmless; Spark picks the next port.
  • Python worker errors: make sure the driver and workers use the same Python (PYSPARK_PYTHON).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does master("local[*]") do?

    local[*] keeps everything in one process and uses as many threads as there are cores.
  2. 2. Why does getOrCreate() exist?

    getOrCreate reuses the active session instead of building another one.
  3. 3. Which pip package gives you PySpark for local development?

    The pyspark package on PyPI bundles the Spark jars. You still need a supported Java runtime.
  4. 4. Where do you usually look to see jobs, stages and SQL plans of a running local session?

    The Spark UI listens on port 4040 by default (4041 and up if that port is taken).

Summary

  • Spark needs a supported Java runtime; PySpark also needs Python.
  • pip install pyspark is the quickest way to start; a downloaded distribution adds spark-shell and spark-submit.
  • Notebooks and the Jupyter PySpark Docker image avoid manual setup.
  • SparkSession.builder...getOrCreate() is the entry point; spark.sparkContext is the RDD context.
  • Leave master out of the code for clusters and pass it to spark-submit.
  • The Spark UI on port 4040 shows what your job is doing.
  • Here, real Spark code is read-only; the mock session is a stand-in for the code shape only.

Next: Lazy Evaluation, Transformations and Actions.