Skip to content
Tutorials Center
My learning

Topics

Lesson 39 of 48 ยท Performance and Production

Deploying Spark: YARN, Kubernetes, Standalone and Cloud

Learn how to deploy Spark applications: spark-submit, client and cluster deploy modes, executor sizing, dynamic allocation, packaging and managed cloud services.

A Spark program that works in a notebook is only the first step. In production the same code has to run on a cluster you do not sit in front of, start on a schedule, get the right amount of CPU and memory, and ship its dependencies with it. Deploying Spark means choosing a cluster manager, packaging your application, and handing it over with spark-submit and a set of resource settings.

The real thing cannot run on this site: there is no cluster, no YARN, no Kubernetes. Every command and configuration below is read-only. What does run here are small stand-ins in plain Python that calculate executor sizes, check a spark-submit command for common mistakes and simulate dynamic allocation, so you can see the reasoning behind the settings. If you need a refresher on the pieces involved, read Spark Architecture first.

The four ways to run Spark

Spark itself does not own machines. It asks a cluster manager for containers, starts executors in them and then schedules tasks. You pick the manager with the --master flag.

Manager --master value Typical use
Local local[4], local[*] Development and tests: driver and executor share one JVM
Standalone spark://host:7077 Spark's own simple manager, small dedicated clusters
YARN yarn Hadoop clusters, on-premises and EMR
Kubernetes k8s://https://api-server:443 Container platforms, cloud-native teams

Mesos support was deprecated and removed in recent releases, so you will only meet it in old systems.

Standalone is the easiest to understand: you start a master process (sbin/start-master.sh) and one worker per machine (sbin/start-worker.sh spark://host:7077), and applications connect to the master. It has no multi-tenant queues or fine-grained security, which is why larger organisations choose YARN or Kubernetes.

YARN (Yet Another Resource Negotiator) is the resource manager from Hadoop. Spark talks to the ResourceManager, which starts an ApplicationMaster for your job and containers for each executor. Queues, quotas and Kerberos security come from YARN.

Kubernetes runs the driver and every executor as a pod. You give Spark a container image that contains Spark and your code, and it asks the Kubernetes API server to create the pods. When the job ends, the executor pods are deleted and the driver pod stays in Completed state so you can read its logs.

spark-submit and deploy modes

spark-submit is the one launcher for every manager. The general shape is the same everywhere:

Terminal
spark-submit \
  --master yarn \
  --deploy-mode cluster \
  --name daily-sales \
  --num-executors 29 \
  --executor-cores 5 \
  --executor-memory 19g \
  --driver-memory 4g \
  --conf spark.sql.shuffle.partitions=400 \
  --py-files deps.zip \
  sales_job.py --date 2026-10-31

Everything after the application file (sales_job.py) is passed to your program as its own arguments; everything before it configures Spark.

The deploy mode decides where the driver runs:

  • client: the driver runs inside the spark-submit process, on the machine where you typed the command. Output appears in your terminal and interactive shells work, but if that machine loses its network connection the job dies, and the driver must be able to talk to every executor.
  • cluster: the driver is started on a machine inside the cluster. You can disconnect, and the driver sits near the executors. This is the normal choice for scheduled production jobs.
flowchart LR
    subgraph C["client mode"]
        direction LR
        L1["Your machine:<br/>spark-submit + driver"] --> E1["Executors<br/>in the cluster"]
    end
    subgraph K["cluster mode"]
        direction LR
        L2["Your machine:<br/>spark-submit exits"] --> D2["Driver<br/>in the cluster"]
        D2 --> E2["Executors<br/>in the cluster"]
    end

In client mode the driver lives where you launch the job; in cluster mode the cluster hosts it.

Spark shells and notebooks always use client mode because you need the driver next to you. On a standalone cluster, cluster mode is not supported for Python applications, so PySpark jobs there run in client mode.

The same program on Kubernetes needs an image and a namespace instead of a queue:

Terminal
spark-submit \
  --master k8s://https://k8s.example.com:443 \
  --deploy-mode cluster \
  --name daily-sales \
  --conf spark.executor.instances=10 \
  --conf spark.kubernetes.container.image=registry.example.com/data/spark-sales:1.4.0 \
  --conf spark.kubernetes.namespace=data-jobs \
  --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
  local:///opt/app/sales_job.py

The local:// scheme means "this file is already inside the container image", so nothing has to be uploaded.

Sizing executors

The three resource flags you will tune most are --executor-cores (task slots per executor), --executor-memory (JVM heap per executor) and --num-executors (how many). Some facts shape every decision:

  • Each executor container needs memory overhead on top of the heap for off-heap buffers, Python workers and the JVM itself. The default is 10% of the heap with a minimum of 384 MiB (spark.executor.memoryOverhead). For PySpark jobs with heavy Python code, raise it.
  • The operating system and the cluster manager's agents need a core and about a gigabyte per node, so you cannot give them all to executors.
  • Executors with too many cores (more than about 5) tend to suffer from slow disk and HDFS client throughput; executors with only one core cannot share broadcast variables or cached data between tasks and waste overhead.
  • On YARN cluster mode, one container is taken by the ApplicationMaster that hosts the driver.

The stand-in below turns those rules into numbers. It is a rule of thumb calculator, not something Spark runs.

Python Executor sizing calculator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The result is the classic recipe: 5 cores per executor, 3 executors per 16-core node. Notice how one-core executors give many tiny containers with a small heap each, while 15-core executors give one huge container per node: both ends are worse than the middle for typical jobs. Treat the numbers as a starting point and then look at the Spark UI (see Testing, Debugging and the Spark UI) to adjust.

Dynamic allocation

Fixed executor counts waste money when a job has phases: a heavy join needs 100 executors, the next stage needs five. Dynamic allocation lets Spark grow and shrink the pool by itself:

Terminal
--conf spark.dynamicAllocation.enabled=true \
--conf spark.dynamicAllocation.minExecutors=2 \
--conf spark.dynamicAllocation.maxExecutors=50 \
--conf spark.dynamicAllocation.executorIdleTimeout=60s \
--conf spark.dynamicAllocation.shuffleTracking.enabled=true

Spark requests executors when tasks wait in the queue (after schedulerBacklogTimeout, one second by default), asking for 1, then 2, then 4, then 8 more, doubling while the backlog lasts. It removes an executor that has been idle for executorIdleTimeout.

The catch is shuffle data. Executors hold the map output of shuffles (see Lazy Evaluation, Transformations and Actions for what triggers one), so removing one could delete data a later stage needs. You must therefore enable either the external shuffle service (the usual setup on YARN: spark.shuffle.service.enabled=true) or shuffle tracking (the usual setup on Kubernetes, where there is no such service), so Spark only releases executors that no longer hold needed shuffle files.

Python Simulating dynamic allocation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The stand-in releases all spare executors at once; real Spark checks each executor's own idle time and also keeps executors that hold cached data for longer (cachedExecutorIdleTimeout, infinite by default). The shape is the same: a ramp up while work waits, a drop after quiet.

Packaging code and dependencies

The driver and every executor need your code and libraries. How you provide them depends on the language:

Terminal
# Java and Scala: one "fat" JAR built with sbt-assembly or Maven Shade
spark-submit --class com.example.SalesJob --master yarn app-assembly-1.4.0.jar

# Libraries from Maven coordinates, downloaded at startup
spark-submit --packages org.apache.spark:spark-avro_2.13:4.0.0 sales_job.py

# Extra Python files or a zipped package of your own code
spark-submit --py-files deps.zip,helpers.py sales_job.py

# A whole Python environment: build it once, ship it as an archive
python -m venv pyspark_venv && . pyspark_venv/bin/activate
pip install pandas pyarrow venv-pack && venv-pack -o pyspark_venv.tar.gz
export PYSPARK_PYTHON=./environment/bin/python
spark-submit --archives pyspark_venv.tar.gz#environment sales_job.py

# Data and config files copied into each executor's working directory
spark-submit --files lookup.csv,settings.yaml sales_job.py

Do not bundle Spark itself into your fat JAR: mark it provided, because the cluster already has it and two copies of the classes can conflict. On Kubernetes, the cleaner way is to bake dependencies into the container image, because every pod then starts with exactly the same environment.

The checker below mirrors a few of the rules a launch can trip over. It is a stand-in that validates a plain Python description of the command.

Python A spark-submit pre-flight checker

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real versus stand-in. The calculator, the allocation simulation and the checker are tiny models of ideas. Real spark-submit validates far more, negotiates containers with YARN or Kubernetes, uploads files, and the cluster manager enforces its own limits. Always confirm settings on a real cluster with a small test run.

Managed cloud platforms

You do not have to run a cluster yourself. The big clouds sell Spark as a service, and all of them accept the same spark-submit style of job:

  • Amazon EMR runs Spark on YARN (or on EKS, the Kubernetes service, or as EMR Serverless). A job is a "step" that calls spark-submit, or a serverless job run with an IAM role and S3 paths.
  • Databricks is a managed Spark platform: clusters and notebooks live in a workspace, jobs are defined in a UI or with the Databricks CLI and APIs, and it adds Delta Lake and its own optimised runtime.
  • Google Cloud Dataproc creates YARN clusters in about a minute (gcloud dataproc jobs submit pyspark), and has a serverless batch option too.
  • Azure offers Synapse Spark pools, HDInsight and Databricks.

Cloud setups share a few habits worth copying: keep data in object storage (S3, GCS, ADLS) so clusters are disposable, create a cluster per job or use autoscaling pools, store application code in a versioned bucket or image registry, and never hard-code credentials: use the platform's IAM role.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A nightly job is started by a scheduler on a small gateway machine that may restart. Which deploy mode keeps the driver safe from that restart?

    In cluster mode the driver runs inside the cluster, so the submitting machine can exit or restart without killing the job.
  2. 2. Why must dynamic allocation be paired with an external shuffle service or shuffle tracking?

    Executors hold map output of shuffles. The shuffle service or shuffle tracking keeps Spark from releasing an executor whose files are still required.
  3. 3. What does the local:// prefix in a Kubernetes spark-submit path mean?

    local:// refers to a path inside the image, so nothing needs to be uploaded when the pods start.
  4. 4. Which is the usual advice when packaging a Scala application as a fat JAR?

    The cluster already provides Spark. Bundling a second copy can cause class conflicts.

Summary

  • --master selects the cluster manager: local, standalone, YARN or Kubernetes; spark-submit is the single launcher for all of them.
  • Client mode keeps the driver on the submitting machine; cluster mode runs it in the cluster, which is the normal choice for production jobs.
  • Size executors around 5 cores each, leave a core and a gigabyte per node for the system, and include memory overhead in the container budget.
  • Dynamic allocation grows and shrinks executors with the workload but needs the external shuffle service or shuffle tracking.
  • Ship dependencies with --py-files, --packages, --archives, fat JARs or container images, and keep credentials out of code.
  • EMR, Databricks and Dataproc run Spark for you; keep data in object storage so clusters stay disposable.

Next: find out why a deployed job is slow or failing in Testing, Debugging and the Spark UI.