Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 48 ยท DataFrames and Spark SQL

pandas API on Spark

Use the pandas API on Spark (pyspark.pandas) to scale pandas code, convert with toPandas and Arrow safely, and know which pandas habits break at scale.

If you know pandas, you already know a lot about working with tables in Python. The catch: pandas keeps everything in memory on one machine. When a file is 200 GB, pandas stops. Spark can handle that size, but its DataFrame API looks different. There are two bridges between the worlds, and this lesson covers both: the pandas API on Spark (pyspark.pandas, which lets you write pandas-style code that runs on Spark), and conversion between Spark and pandas DataFrames with toPandas() and createDataFrame(pandas_df), speeded up by Apache Arrow.

It builds on DataFrames: Creating and Inspecting and User-Defined Functions and Pandas UDFs. Spark cannot run on this site, so the PySpark code is read-only. The runnable examples are plain-Python stand-ins that show why some pandas habits (such as row order) do not carry over to a distributed engine.

Two different things named "pandas on Spark"

It is easy to mix these up:

Tool What it is Data lives
pyspark.pandas (pandas API on Spark, formerly Koalas) A pandas-like API implemented on top of Spark DataFrames Distributed, on the cluster
df.toPandas() Collect a Spark DataFrame into a real pandas DataFrame Driver memory, one machine
Pandas UDFs / applyInPandas Run real pandas functions on batches of a Spark DataFrame Distributed; each batch is a small pandas object

pyspark.pandas has been part of PySpark since version 3.2. Nothing extra is needed beyond pip install pyspark (and pandas and PyArrow, which it depends on).

Using pyspark.pandas

Python
import pyspark.pandas as ps

psdf = ps.read_parquet("s3://data/trips/")            # a pandas-on-Spark DataFrame
psdf["fare_per_km"] = psdf["fare"] / psdf["distance"]
top = (psdf[psdf["distance"] > 1]
            .groupby("city")["fare_per_km"]
            .mean()
            .sort_values(ascending=False)
            .head(5))
print(top)                                             # looks and prints like pandas

The syntax is deliberately the pandas one: psdf["col"], boolean masks, groupby, merge, value_counts, describe, plot. Under the hood each call builds a Spark query plan, so the work runs in parallel and is lazy. Seeing a result (printing, head, to_pandas) triggers a job.

You can move between the three representations:

Python
sdf = psdf.to_spark()                  # to a regular Spark DataFrame (index dropped unless index_col given)
psdf2 = sdf.pandas_api()               # back to pandas on Spark (Spark 3.3+; older: sdf.to_pandas_on_spark())
pdf = psdf.to_pandas()                 # to real pandas, collects everything to the driver

That makes it a gentle migration path: prototype with pandas on a sample, switch the import to pyspark.pandas, and fix the few places that differ.

What changes at scale

Distributed engines cannot offer some pandas guarantees cheaply, so behavior differs in important ways:

  • No global row order by default. pandas has a positional index and keeps order. A Spark DataFrame has no inherent order. pandas on Spark creates a default index for you, and the cost depends on the type: distributed-sequence (the default) gives 0..n-1 but needs an extra pass, distributed is cheap but only monotonically increasing, and sequence pulls all data onto one node and should be avoided on big data. Prefer to set a real index column: ps.read_parquet(path, index_col="trip_id").
  • Operations that need all data in order are slow or unavailable. iloc on big data, cumsum across the whole table, rolling windows and shift need ordering, so they trigger sorts or single-partition work. Spark may even warn that it is "moving all data to a single partition".
  • No in-place row mutation. Loops that set one cell at a time (df.loc[i, "x"] = ...) do not belong on a distributed engine; express changes as column operations.
  • Some APIs are missing (a few indexing forms, some apply patterns) and some semantics differ slightly, such as null handling versus NaN. Check the documentation's supported-API list.
  • Results are not always in the same order as in pandas, so sort explicitly when it matters.

A good rule: think in columns. Operations that apply to whole columns or groups scale; operations that depend on "the row before this one" or on a Python loop do not.

Python Why a global running total needs ordering

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Converting with toPandas and Arrow

toPandas() is the way to hand a small result to pandas, matplotlib or scikit-learn:

Python
summary = (spark.table("sales").groupBy("region").agg(F.sum("amount").alias("total")))
pdf = summary.toPandas()          # tiny: one row per region
pdf.plot.bar(x="region", y="total")

The danger is the same as with collect() (see Sampling and Getting Rows Back): every row moves to the driver. On a table with a billion rows it fails with an out-of-memory error. Aggregate, filter or sample first so the result fits in the driver, and check size with summary.count() if unsure.

Moving data the other way, spark.createDataFrame(pdf), ships the pandas data from the driver to the cluster, so it is also only for small or medium data; for big data read files with Spark directly.

By default conversion goes row by row through Python objects, which is slow. Apache Arrow is a columnar in-memory format that Spark and pandas can both read without converting each value; enabling it makes toPandas() and createDataFrame(pdf) much faster (often by an order of magnitude), and it is also what powers Pandas UDFs.

Python
spark.conf.set("spark.sql.execution.arrow.pyspark.enabled", "true")
spark.conf.set("spark.sql.execution.arrow.pyspark.fallback.enabled", "true")   # fall back if a type is unsupported
spark.conf.set("spark.sql.execution.arrow.maxRecordsPerBatch", "10000")        # batch size

Arrow does not support every type (some nested or map types raise errors), and Spark falls back to the slow path when fallback is enabled. Recent Spark versions enable Arrow in more places by default; check your version's configuration page rather than assuming.

Python Value-by-value versus bulk columnar encoding

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real versus stand-in. Real pandas on Spark builds Spark query plans and shuffles data across executors, and real Arrow transfers typed columnar buffers between the JVM and Python. The stand-ins only illustrate the ideas (partition-wise work, ordering cost, row versus column handling) with ordinary Python lists; timings will vary by machine.

When to use which

Use plain Spark DataFrames for new pipelines, where the optimizer and the SQL-like API are strongest. Use pandas API on Spark to port existing pandas code or when the team knows pandas better than Spark. Use toPandas() only for small results that you then analyze or plot locally. Use Pandas UDFs when you need a pandas or NumPy function (a scikit-learn model, for example) applied on the cluster.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Where does the data of a pyspark.pandas DataFrame live?

    pandas on Spark is an API layer over Spark DataFrames, so the data stays distributed until you call to_pandas().
  2. 2. What is the main risk of calling toPandas() on a huge DataFrame?

    toPandas() is an action that moves every row to one machine. Aggregate, filter or sample first.
  3. 3. Which pandas habit is expensive on a distributed engine?

    Anything that depends on a global row order needs partition totals and an ordering, so it is slow or may move data to one partition.
  4. 4. What does enabling spark.sql.execution.arrow.pyspark.enabled do?

    Arrow transfers whole columns between the JVM and Python instead of converting value by value.

Summary

  • pyspark.pandas gives a pandas-style API that runs distributed on Spark; calls build query plans and are lazy.
  • to_spark(), pandas_api() and to_pandas() move between pandas on Spark, Spark DataFrames and real pandas.
  • Avoid pandas habits that need global order or Python loops: iloc, whole-table cumsum, cell-by-cell updates; set a real index_col.
  • toPandas() and createDataFrame(pdf) move data through the driver: use them for small data, after aggregating or sampling.
  • Enable Arrow (spark.sql.execution.arrow.pyspark.enabled) for fast conversion and Pandas UDFs.

Next: learn the string, date and regex helpers in Columns and Expressions.