Skip to content
Tutorials Center
My learning

Topics

Lesson 21 of 48 ยท DataFrames and Spark SQL

Arrays, Maps, Structs and JSON

Work with nested data in Spark DataFrames: arrays, maps and structs, explode and posexplode, array functions, struct access, from_json and to_json.

Real data is rarely flat. An order has a list of items, a user has an address with several fields, an event carries a bag of properties, and an API returns a JSON string. Spark handles this with three complex types: arrays (an ordered list of values of one type), maps (key-value pairs) and structs (a fixed group of named fields, like a row inside a row).

Because these types live inside a single column, you can keep nested data nested, query it directly and flatten it only when needed. This lesson shows how to read each type, how explode turns an array into rows, the most useful array functions, and how to convert between JSON text and structs with from_json and to_json. It follows Window Functions.

The real functions need Spark, so the Spark code is read-only. The runnable examples use plain Python lists, dictionaries, the json module and pandas to show the same transformations.

The three types

Python
from pyspark.sql import functions as F

df = spark.createDataFrame([
    (1, "Ann", ["python", "sql"],  {"city": "Berlin", "zip": "10115"}, {"plan": "pro", "seats": "5"}),
    (2, "Bo",  ["spark"],          {"city": "Paris",  "zip": "75001"}, {"plan": "free"}),
    (3, "Cy",  [],                 {"city": "Rome",   "zip": "00100"}, None),
], "id INT, name STRING, skills ARRAY<STRING>, address STRUCT<city: STRING, zip: STRING>, "
   "settings MAP<STRING, STRING>")
Text
root
 |-- id: integer
 |-- name: string
 |-- skills: array<string>
 |-- address: struct<city: string, zip: string>
 |-- settings: map<string,string>

Accessing elements:

Python
df.select(
    F.col("skills")[0],                    # first element (0-based), null if missing
    F.col("address.city"),                 # struct field with dot notation
    F.col("address")["zip"],               # also works
    F.col("settings")["plan"],             # map value by key, null if the key is absent
    F.size("skills"),                      # array length (-1 for a null array in older ANSI-off mode)
    F.element_at("skills", 1),             # element_at is 1-BASED
)

Watch the indexing: col[0] and getItem(0) are zero-based, while element_at is one-based. element_at("skills", -1) gives the last element. Indexing past the end gives null when ANSI mode is off and raises an error when it is on (see Spark SQL and Temp Views).

Create nested values with F.array(...), F.struct(...) and F.create_map(...):

Python
df.select(F.struct("name", "id").alias("who"), F.array("id", "id").alias("twice"))

Structs are the right choice for a fixed set of related fields (an address). Maps fit open-ended keys (tags, properties) but cannot be typed per key, and maps cannot be used as grouping or join keys. Arrays fit lists of the same kind of thing.

Python Arrays, structs and maps as Python values

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

explode: arrays become rows

explode produces one output row per array element, repeating the other columns. It is the main way to flatten arrays.

Python
df.select("name", F.explode("skills").alias("skill"))
Text
+----+------+
|name| skill|
+----+------+
| Ann|python|
| Ann|   sql|
|  Bo| spark|
+----+------+

Notice that Cy disappeared: an empty or null array produces no rows. Use explode_outer to keep such rows with a null value. posexplode adds the position of each element as a second column (pos, col), and exploding a map gives key and value columns. In SQL you will see the same thing as LATERAL VIEW explode(skills) t AS skill, or SELECT name, explode(skills) FROM people.

Exploding multiplies rows, so a one-million-row table with arrays of 100 elements becomes 100 million rows. Filter before you explode when you can, and aggregate back (groupBy(...).agg(F.collect_list(...))) when you need arrays again.

Python explode, explode_outer and posexplode with pandas

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

explode variants and map columns

Besides plain explode there are three variants, and they differ in how they treat empty arrays and positions:

Function Empty or null array Output columns
explode row disappears col
explode_outer row kept, value is null col
posexplode row disappears pos, col
posexplode_outer row kept, pos and col null pos, col
Python
df.select("name", F.explode_outer("skills").alias("skill"))
df.select("name", F.posexplode("skills").alias("pos", "skill"))

select allows only one generator (such as explode) per call. To explode several arrays that belong together (skill and level in parallel arrays), zip them into one array of structs first, then explode once:

Python
paired = df.withColumn("pair", F.explode(F.arrays_zip("skills", "levels")))
paired.select("name", F.col("pair.skills").alias("skill"), F.col("pair.levels").alias("level"))

Exploding the two arrays one after the other would produce every combination (a cross product) instead of matched pairs.

A map column turns into rows with explode (columns key and value). To turn it into several ordinary columns, pick the keys you know with bracket access, or pivot the exploded keys:

Python
df.select("id", F.col("attrs")["color"].alias("color"), F.col("attrs")["size"].alias("size"))

df.select("id", F.explode("attrs")).groupBy("id").pivot("key").agg(F.first("value"))

For a struct or parsed JSON column, col("s.*") expands every field into its own column.

Python explode_outer, posexplode and zipped arrays

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Array functions

Spark has a large set of built-in array functions, so you rarely need a UDF (see User-Defined Functions and Pandas UDFs):

Function What it does
size(arr) Number of elements
array_contains(arr, v) True if the value is present
array_distinct(arr) Removes duplicates
sort_array(arr, asc) Sorts; nulls first when ascending
array_union, array_intersect, array_except Set operations on two arrays
array_join(arr, sep) Joins strings into one string
slice(arr, start, length) Sub-array (1-based start)
flatten(arr_of_arrays) One level of nesting removed
arrays_zip(a, b) Array of structs pairing elements
collect_list, collect_set Aggregate rows into an array

Higher-order functions take a lambda and process each element inside the column, without exploding:

Python
df.select(
    F.transform("prices", lambda p: p * 1.2).alias("with_tax"),
    F.filter("prices", lambda p: p > 10).alias("expensive"),
    F.aggregate("prices", F.lit(0.0), lambda acc, p: acc + p).alias("total"),
    F.exists("prices", lambda p: p > 100).alias("has_big"),
)

The SQL form is transform(prices, p -> p * 1.2). These run in Spark's engine without moving rows, which is cheaper than explode, process, regroup.

Structs and flattening nested data

A struct column is addressed with dots, and "address.*" expands all its fields into top-level columns:

Python
df.select("name", "address.*")                       # name, city, zip
df.withColumn("city", F.col("address.city"))
df.withColumn("address", F.col("address").withField("zip", F.lit("00000")))   # update one field
df.select(F.struct("city", "zip").alias("address"))  # build one back

Deeply nested data (struct of arrays of structs) is flattened step by step: select("order_id", F.explode("items").alias("item")), then select("order_id", "item.sku", "item.qty"). Parquet and JSON keep this nested layout, and Parquet can read a single nested field without reading the others, so keeping data nested is not a performance problem. Flatten only for the output you need.

JSON in columns: from_json and to_json

JSON text often arrives as a string column (a Kafka message, an API payload, a log line). from_json parses it into a struct using a schema you provide:

Python
from pyspark.sql.types import StructType, StructField, StringType, IntegerType, ArrayType

schema = StructType([
    StructField("user", StringType()),
    StructField("age", IntegerType()),
    StructField("tags", ArrayType(StringType())),
])

parsed = raw.withColumn("data", F.from_json("payload", schema))
parsed.select("data.user", "data.age", F.explode("data.tags").alias("tag"))

# a DDL string works too
F.from_json("payload", "user STRING, age INT, tags ARRAY<STRING>")

Behavior to remember: a string that is not valid JSON gives a null struct (in the default permissive mode), and a field that is missing or has the wrong type becomes null for that field. Nothing fails, so count nulls after parsing to detect bad rows. to_json goes the other way and turns a struct, array or map into a JSON string, which is how you prepare messages for Kafka. For one-off extractions, get_json_object(col, "$.user.name") and json_tuple pull values by path without a schema, but they re-parse the string for every call, so from_json is better when you need several fields. schema_of_json can propose a schema from a sample string, which is handy in exploration but, as with inferSchema in Reading and Writing Data, you should write the schema down for production.

flowchart LR
    S["JSON string column"] -->|"from_json(schema)"| T["Struct column"]
    T -->|"data.field"| F["Flat columns"]
    T -->|"explode on array fields"| R["More rows"]
    T -->|"to_json"| S

from_json and to_json convert between text and structs; dot access and explode flatten them.

The last stand-in does the same round trip with Python's json module, including the "bad JSON becomes null" behavior.

Python from_json and to_json round trip with null handling

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real versus stand-in. Spark stores arrays, maps and structs in typed columnar form, runs explode and the higher-order functions in its engine across partitions, and parses JSON with a declared schema. The Python versions imitate the results and rules: zero-based versus one-based access, rows vanishing from explode for empty arrays, and bad JSON becoming null. They are not the same implementation, and they are not meant for large data.

Common mistakes

  • Expecting explode to keep rows with empty or null arrays (use explode_outer).
  • Mixing up zero-based col[i] with one-based element_at.
  • Using a map where a struct fits, losing types and discoverability.
  • Exploding several arrays in the same select: you get a row for every combination (use arrays_zip first, or explode one at a time).
  • Calling get_json_object repeatedly on the same string instead of parsing once with from_json.
  • Never checking for nulls after from_json and silently losing bad records.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A row has skills = [] (an empty array). What does select(name, explode(skills)) do with it?

    explode drops rows whose array is empty or null. explode_outer keeps them with a null element.
  2. 2. Which access is one-based?

    element_at counts from 1 (and from the end with negative numbers), while [] and getItem are zero-based.
  3. 3. What does from_json return for a string that is not valid JSON in the default mode?

    Permissive parsing yields null. Check for nulls after parsing to find malformed records.
  4. 4. When is a struct a better choice than a map?

    A struct has named, individually typed fields. A map has open-ended keys with one value type.
  5. 5. Why prefer F.transform(array, lambda) over exploding, changing and regrouping?

    Higher-order functions run per row in the engine; explode multiplies rows and regrouping requires a shuffle.

Summary

  • Spark has three complex types: arrays (ordered lists), maps (key-value pairs) and structs (named fields); nested data can stay nested.
  • Access elements with col[i] (zero-based), element_at (one-based), dot notation for structs and col[key] for maps.
  • explode turns array elements into rows and drops empty arrays; explode_outer keeps them, posexplode adds positions.
  • Built-in array functions and higher-order functions (transform, filter, aggregate, exists) avoid both UDFs and exploding.
  • from_json parses a string column into a struct using a schema (bad input becomes null) and to_json converts back.

Next: when built-in functions are not enough, see User-Defined Functions and Pandas UDFs.