Skip to content
Tutorials Center
My learning

Topics

Lesson 23 of 48 ยท DataFrames and Spark SQL

Datasets, Schemas and Data Types

Understand Spark schemas: StructType, nullability, common data types, typed Datasets in Scala and Java, DataFrame versus Dataset, and schema evolution.

Every Spark DataFrame has a schema: a list of column names, data types and nullability flags. The schema is how Spark knows how to store your data compactly, which operations are legal, and how to read and write files. Getting types right early prevents bugs such as numbers sorted as text or dates that silently turn into NULL.

This lesson covers the type system, how to define schemas by hand, what a Dataset is and how it differs from a DataFrame, and how schemas change over time. It builds on DataFrames: Creating and Inspecting.

The main data types

Spark SQL has its own type system that maps to each language's types.

Spark type Holds Python example
ByteType, ShortType, IntegerType, LongType whole numbers of 8, 16, 32, 64 bits int
FloatType, DoubleType binary floating point float
DecimalType(p, s) exact decimals, precision and scale decimal.Decimal
StringType text str
BooleanType true or false bool
DateType, TimestampType calendar date, instant in time date, datetime
BinaryType raw bytes bytes
ArrayType, MapType, StructType nested collections and records list, dict, Row

Use DecimalType for money. DoubleType cannot store 0.1 exactly, and sums of doubles drift. Python integers are mapped to LongType when Spark infers a schema.

Schemas with StructType

A schema is a StructType, which is a list of StructField entries: name, type and whether the column may be NULL.

Python
from pyspark.sql.types import (
    StructType, StructField, StringType, IntegerType, DoubleType, DateType, ArrayType,
)

schema = StructType([
    StructField("order_id", IntegerType(), nullable=False),
    StructField("customer", StringType(), nullable=True),
    StructField("amount", DoubleType(), nullable=True),
    StructField("order_date", DateType(), nullable=True),
    StructField("tags", ArrayType(StringType()), nullable=True),
])

df = spark.read.schema(schema).json("orders/")
df.printSchema()
# root
#  |-- order_id: integer (nullable = false)
#  |-- customer: string (nullable = true)
#  |-- amount: double (nullable = true)
#  |-- order_date: date (nullable = true)
#  |-- tags: array (nullable = true)
#  |    |-- element: string (containsNull = true)

PySpark also accepts a DDL string, which is shorter: "order_id INT NOT NULL, customer STRING, amount DOUBLE, order_date DATE, tags ARRAY<STRING>". An explicit schema saves Spark from scanning the files to guess types, and it protects you from surprises when one file contains an odd value. We return to this in Reading and Writing Data.

flowchart TD
    S["StructType: order"]
    S --> F1["order_id: integer, not null"]
    S --> F2["customer: string"]
    S --> F3["amount: double"]
    S --> F4["address: StructType"]
    F4 --> G1["city: string"]
    F4 --> G2["zip: string"]

A schema is a tree: a StructType contains fields, and a field can itself be a StructType.

Nullability is a hint, not a guarantee

nullable = false tells the optimizer it can skip null checks. It is not enforced when reading most file sources: Spark commonly marks every column as nullable on read, since files cannot promise anything. If your data does contain a NULL in a column declared non-nullable, the result can be wrong or fail later in an obscure way. Declare nullable = false only when you control the data, and validate it with explicit checks (see the cleaning steps in Filtering, Sorting and Deduplicating).

Datasets: typed rows on the JVM

A Dataset is a distributed collection of typed objects. In Scala you describe the record with a case class, and the compiler checks your lambdas:

SCALA
case class Order(orderId: Int, customer: String, amount: Double)

import spark.implicits._
val orders = spark.read.parquet("orders/").as[Order]   // Dataset[Order]

val big = orders.filter(o => o.amount > 100.0)         // compile-time checked
val names = orders.map(o => o.customer)                // Dataset[String]

Datasets keep data in Spark's internal binary format and use encoders to convert to and from your objects only when needed. In Java, Dataset<Order> works similarly with bean classes and Encoders.bean(Order.class).

A DataFrame is simply an alias for Dataset[Row]: a Dataset whose elements are generic untyped rows. That is why the two share almost all of their API.

DataFrame (Dataset[Row]) Dataset[T]
Languages Python, R, Scala, Java Scala and Java only
Column mistakes found at run time field names checked at compile time
Lambda functions not typed, run as UDFs (Python) typed, but opaque to the optimizer
Optimization full Catalyst optimization column expressions optimized; lambdas are black boxes

PySpark has no typed Dataset, because Python is dynamically typed. You work with DataFrames, and the schema is checked when the query is analyzed, which happens before any data is processed, but after your code starts running. When you need static typing, use Scala or Java, or add your own validation in Python.

Prefer column expressions (filter($"amount" > 100)) over typed lambdas (filter(o => o.amount > 100)) in Scala when speed matters: the optimizer can see through the first and not the second, similar to the UDF problem in User-Defined Functions and Pandas UDFs.

Seeing a schema as data

A schema can be exported to JSON and read back. This is how Spark stores schemas in files and catalogs, and it is a handy way to share a contract between jobs.

The stand-in below builds a tiny schema description in plain Python, renders a printSchema-style tree, and validates rows against it, including the nullability rule.

Python A toy schema and validator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real Spark does the equivalent work in the analyzer and in file readers, though with richer rules, such as casting a string to a number when asked and producing NULL for unparsable values in permissive reading mode. The validator above is a stand-in for the idea, not Spark's code.

Schema evolution

Data changes: a new field appears, a column is renamed, a type widens from int to long. How Spark copes depends on the file format.

  • Parquet can merge schemas across files: spark.read.option("mergeSchema", "true").parquet("events/") produces the union of all columns, filling missing values with NULL. It is off by default because merging means reading every file's footer.
  • JSON and CSV carry no schema, so inference or your explicit schema decides. Missing fields become NULL.
  • Avro and table formats like Delta Lake and Apache Iceberg support explicit evolution rules and track schema versions.

Safe changes are adding a nullable column or widening a numeric type. Risky changes are renaming or dropping columns and changing a type incompatibly, because old files and new readers disagree. Whatever the format, version your schema, test with old files, and document each change.

The stand-in below merges two schema versions the way a union read does.

Python Merging two schema versions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Columns missing from one version come out as None, which stands for NULL. Incompatible changes, for example string to int, would be flagged as a conflict, and a real read would fail or produce nulls depending on its settings.

When a new field is expected downstream, define the schema explicitly with that field as nullable. The old files then read cleanly, and you avoid the cost and risk of schema inference on every run.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is a DataFrame in terms of Datasets?

    DataFrame is an alias for Dataset[Row]. That is why they share most of their API and the same optimizer.
  2. 2. Why is DecimalType preferred for money?

    Binary floating point drifts on decimal fractions. DecimalType(p, s) keeps exact precision and scale.
  3. 3. What does nullable = false guarantee when reading a CSV or JSON file?

    File sources cannot promise non-null data, so the flag is not enforced on read. Wrong declarations can cause wrong results later.
  4. 4. Which language offers typed Datasets of your own classes?

    Typed Datasets need compile-time types and encoders, which exist in Scala and Java. PySpark works with DataFrames.
  5. 5. How can you read Parquet files whose columns differ between versions?

    With mergeSchema enabled, Spark reads each file's schema and merges them, filling missing columns with NULL.

Summary

  • A schema is a StructType of fields with a name, a type and a nullable flag, and it can be nested.
  • Use explicit schemas for files you do not control, and DecimalType for exact values such as money.
  • nullable is an optimizer hint that most file sources do not enforce, so validate your data.
  • A DataFrame is a Dataset[Row]; typed Datasets exist only in Scala and Java, and typed lambdas are opaque to the optimizer.
  • Schemas evolve: adding nullable columns and widening numbers is safe, while renames and incompatible type changes need care.

Next: see how Spark turns your query into an efficient plan in Catalyst Optimizer and Query Plans.