Skip to content
Tutorials Center
My learning

Topics

Lesson 22 of 48 ยท DataFrames and Spark SQL

User-Defined Functions and Pandas UDFs

Learn when to write a Spark UDF, why Python UDFs are slow, how pandas UDFs use Arrow to work on batches, and why built-in functions should come first.

Spark ships with hundreds of built-in functions: string tools, date math, math, array and map helpers, and more. Sooner or later you meet a rule that none of them covers, such as "normalize this product code with our company's own format". For that, Spark lets you register your own function and call it on a column. It is called a user-defined function, or UDF.

UDFs are easy to write and easy to overuse. A UDF is a black box to Spark's optimizer, and in PySpark every row may have to travel between the JVM and a Python process. This lesson shows what a UDF costs, how pandas UDFs reduce that cost by working on whole batches, and a decision order that keeps your jobs fast. You should already know Columns and Expressions.

A plain Python UDF

In PySpark you wrap a Python function with udf and declare the return type. Spark cannot infer it, because it never looks inside your code.

Python
from pyspark.sql import functions as F
from pyspark.sql.types import StringType

def normalize_code(code):
    if code is None:
        return None
    return code.strip().upper().replace(" ", "-")

normalize_udf = F.udf(normalize_code, StringType())

df = spark.createDataFrame([("ab 12",), (" cd 7 ",), (None,)], ["code"])
df.select(normalize_udf("code").alias("clean")).show()
# +-----+
# |clean|
# +-----+
# |AB-12|
# | CD-7|
# | NULL|
# +-----+

Two details matter. First, your function must handle None, because Spark passes SQL NULL as None and a crash on one bad row fails the whole task. Second, you can also register the function for SQL with spark.udf.register("normalize_code", normalize_code, StringType()) and then write SELECT normalize_code(code) FROM products.

Why Python UDFs are slow

A DataFrame normally runs entirely inside the JVM, in Spark's compact binary row format, with generated code (see Catalyst Optimizer and Query Plans). A Python UDF breaks that:

  1. Executor JVM serializes the rows that the UDF needs.
  2. The rows are sent to a separate Python worker process.
  3. Python calls your function once per row.
  4. The results are serialized and sent back to the JVM.

Each step costs time, and the Python interpreter is slower than generated JVM code at plain arithmetic. There is a second, quieter cost: the optimizer cannot see inside the UDF. It cannot push a filter through it, cannot prune work, and cannot fuse it with neighbouring operators the way it can with built-in functions.

flowchart LR
    subgraph JVM["Executor JVM"]
        A["Rows in JVM format"]
        D["Results back in JVM format"]
    end
    subgraph PY["Python worker process"]
        B["Deserialize each row"]
        C["Call your function per row"]
    end
    A -->|"serialize"| B
    B --> C
    C -->|"serialize"| D

A row-wise Python UDF crosses the JVM and Python boundary twice for every row.

In Scala and Java the story is milder: a UDF runs inside the JVM, so there is no process boundary. The optimizer still treats it as opaque, but it avoids serialization to another process.

Built-in functions first

Before writing a UDF, check whether a built-in does the job. Most "custom" logic is a combination of built-ins:

Python
# The UDF from above, rewritten with built-in functions
df.select(F.upper(F.regexp_replace(F.trim("code"), " ", "-")).alias("clean")).show()

This version stays in the JVM, handles NULL automatically (any function of NULL is NULL), and can be optimized. Conditional rules usually map to when/otherwise, text parsing to regexp_extract or split, and list logic to the array functions you saw in Arrays, Maps, Structs and JSON. Spark's higher-order functions, such as transform and filter on arrays, take lambda expressions and run natively, so they often replace a UDF that loops over a list.

A good rule: write a UDF only when you have tried the built-ins and the logic really needs arbitrary code, such as calling a library for phone number parsing.

Pandas UDFs: batches with Arrow

A pandas UDF, also called a vectorized UDF, receives a whole batch of values as a pandas.Series instead of one value at a time. Spark moves data between the JVM and Python using Apache Arrow, a columnar in-memory format that avoids per-row serialization. Your code then uses vectorized pandas or NumPy operations that run in compiled C.

Python
import pandas as pd
from pyspark.sql.functions import pandas_udf

@pandas_udf("double")
def celsius_to_fahrenheit(temps: pd.Series) -> pd.Series:
    return temps * 9 / 5 + 32

readings = spark.createDataFrame([(0.0,), (21.5,), (100.0,)], ["celsius"])
readings.select(celsius_to_fahrenheit("celsius").alias("fahrenheit")).show()
# 32.0, 70.7, 212.0

The function must return a Series with the same length as the input. Type hints tell Spark which flavour you wrote: pd.Series -> pd.Series is a scalar pandas UDF. Other flavours include an iterator of Series (useful when you need to load a model once per task rather than once per batch), and grouped operations such as groupBy("key").applyInPandas(func, schema), which hands each group to your function as a pandas DataFrame. Batch size is controlled by the setting spark.sql.execution.arrow.maxRecordsPerBatch (10,000 rows by default).

Pandas UDFs are usually several times faster than row-wise Python UDFs for numeric work, though the gain depends on the job. They still run in a Python process and still stay opaque to the optimizer, so a built-in function remains the best choice when one exists.

A stand-in that counts the crossings

The real boundary between the JVM and Python cannot run on this site. The stand-in below uses plain Python and pandas to show the structural difference: a row-wise UDF calls your function once per row, while a vectorized one calls it once per batch. We count calls instead of timing them, so the output is the same on every run.

Python Row-wise versus vectorized calls

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two approaches give identical results, but the row-wise version enters your function 10,000 times and the vectorized one only 4 times. In real Spark each row-wise call also pays the serialization costs from the diagram. Real versus stand-in: here nothing is serialized and nothing crosses a process boundary, so the stand-in demonstrates call counts, not real timings.

Writing UDFs safely

Whatever flavour you choose, a few habits prevent painful bugs. The next stand-in shows two of them: handling missing values, and the idea that an unhandled error kills the entire task.

Python Null-safe UDF logic

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

More habits:

  • Declare the return type exactly. If the function returns a different type than declared, Spark usually produces NULL instead of an error, which hides bugs.
  • Keep UDFs deterministic. Spark may run a UDF more than once for the same row, for example after a retry. If you need to mark a non-deterministic one, call .asNondeterministic().
  • Avoid side effects. Do not write to a database or a counter inside a UDF; retries would repeat the effect. Use accumulators only for rough diagnostics.
  • Test the function alone. A plain Python function is easy to unit test before you wrap it with udf.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is a row-wise Python UDF usually slower than a built-in function?

    Executors must move rows to a Python worker and back, and the function is a black box to Catalyst. Built-ins stay in the JVM and can be optimized.
  2. 2. What does a scalar pandas UDF receive and return?

    A scalar pandas UDF works on a batch as a Series, and it must return a Series with the same number of elements. Arrow carries the data between the JVM and Python.
  3. 3. Which should you try first when a column needs custom logic?

    Built-ins run natively and stay visible to the optimizer. Reach for a UDF only when they truly cannot express the logic.
  4. 4. What happens if a UDF does not handle None and meets a NULL value?

    Spark passes NULL to Python as None. Code such as code.strip() raises AttributeError, and a task that keeps failing ends the job.

Summary

  • A UDF lets you apply your own function to a column, but Spark cannot optimize inside it.
  • A row-wise Python UDF moves every row between the JVM and a Python process; a pandas UDF moves batches through Arrow and uses vectorized code.
  • Always try built-in functions first, including when/otherwise, regex functions and higher-order array functions.
  • Make UDFs null-safe, deterministic, side-effect free and declare their return types exactly.
  • The stand-in counted calls to show the batching idea; real timings depend on the cluster and data.

Next: look at how types and schemas are defined in Datasets, Schemas and Data Types.