Skip to content
Tutorials Center
My learning

Topics

Lesson 10 of 48 ยท DataFrames and Spark SQL

DataFrames: Creating and Inspecting

Learn how to create Spark DataFrames, read the schema, inspect data with show and printSchema, use Row objects, load files and decide between RDD and DataFrame.

An RDD is a bag of arbitrary Python objects: Spark knows nothing about what is inside. A DataFrame is different: it is a distributed table with named columns and types, like a spreadsheet or a SQL table that happens to be split into partitions across a cluster. Because Spark knows the column names and types, it can store data compactly, check your code before running it and apply the optimizer you will meet in Catalyst Optimizer and Query Plans.

DataFrames are the main way people use Spark today. Like RDDs they are immutable and lazy: every operation returns a new DataFrame and nothing runs until an action such as show() or count() (see Lazy Evaluation, Transformations and Actions). This lesson covers creating one, looking at it, and understanding its schema.

The entry point: SparkSession

Everything starts from a SparkSession, the object created in Installing Spark and Your First PySpark Program. It is conventionally called spark.

Python
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("demo").getOrCreate()

Creating a DataFrame

The simplest way is from a Python list. Give it rows and the column names:

Python
data = [("Alice", 34, "Berlin"), ("Bob", 29, "Lisbon"), ("Chen", 41, "Berlin")]

df = spark.createDataFrame(data, ["name", "age", "city"])
df.show()
Text
+-----+---+------+
| name|age|  city|
+-----+---+------+
|Alice| 34|Berlin|
|  Bob| 29|Lisbon|
| Chen| 41|Berlin|
+-----+---+------+

Notice that show() right-aligns values, prints up to 20 rows by default, and cuts any cell longer than 20 characters with .... You can change both: df.show(5, truncate=False).

Other ways to create a DataFrame:

  • from a list of Row objects or dictionaries,
  • from a pandas DataFrame: spark.createDataFrame(pandas_df),
  • from an RDD: rdd.toDF(["name", "age"]),
  • from files and tables: spark.read.csv(...), spark.read.parquet(...), spark.table("sales") (see Reading and Writing Data).

A Row is a lightweight record. You can read its fields by name, by position or as a dictionary:

Python
from pyspark.sql import Row

r = Row(name="Alice", age=34)
r.name          # 'Alice'
r["age"]        # 34
r.asDict()      # {'name': 'Alice', 'age': 34}

Schema: names, types and nullability

The schema is the list of columns with their types and whether they may contain nulls. Spark can infer it by looking at your data, or you can declare it.

Python
df.printSchema()
Text
root
 |-- name: string (nullable = true)
 |-- age: long (nullable = true)
 |-- city: string (nullable = true)

Python integers become long (64-bit), floats become double, strings stay string, and Python bool becomes boolean. Dates, timestamps, decimals, arrays, maps and structs have their own types, covered in Datasets, Schemas and Data Types.

An explicit schema is built from StructType and StructField:

Python
from pyspark.sql.types import StructType, StructField, StringType, IntegerType

schema = StructType([
    StructField("name", StringType(), nullable=False),
    StructField("age", IntegerType(), nullable=True),
])

df = spark.createDataFrame([("Alice", 34), ("Bob", 29)], schema)

Or as a compact DDL string: "name STRING, age INT". Declaring the schema is faster (no sampling pass) and safer (a surprising value fails loudly instead of changing a column's type).

flowchart LR
    D["Python rows<br/>or files"] --> I{"Schema given?"}
    I -->|"yes"| S["Use the declared<br/>StructType"]
    I -->|"no"| F["Infer types<br/>from the data"]
    S --> DF["DataFrame<br/>(names + types + data)"]
    F --> DF

A DataFrame always has a schema; you either declare it or let Spark infer it.

Inspecting a DataFrame

These are the calls you use constantly while exploring:

Call What it does Kind
df.show(n) prints the first n rows as a table action
df.printSchema() prints the column tree no job
df.columns list of column names no job
df.dtypes list of (name, type) pairs no job
df.count() number of rows action
df.take(n) / df.head(n) first n rows as Row objects action
df.first() the first row action
df.describe().show() count, mean, stddev, min, max per column action
df.collect() all rows to the driver (careful, like for RDDs) action

printSchema, columns and dtypes only read metadata, so they are instant even for huge data. show needs a job but usually reads only a few partitions.

A stand-in with pandas

Real PySpark cannot run on this site. To let you experiment with the same ideas, the code below uses pandas DataFrames and two helper functions that print their output in Spark's format: show and print_schema. A pandas DataFrame lives in one process and is not lazy, but its table model is the same, and mapping pandas types to Spark names (int64 to long) makes the output look like what you would see on a cluster.

Python Create, show and print the schema

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The output of show and print_schema above has the same shape as the Spark output in the lesson, so reading real Spark results will feel familiar.

Inferred versus explicit schema, and files

When Spark infers a schema from a CSV file it must read the data an extra time. And by default, a CSV reader does not infer at all: every column is read as string unless you pass inferSchema=True or give a schema. This is a classic surprise: a column of numbers sorts as text, so "9" comes after "10".

Python
spark.read.csv("people.csv", header=True).printSchema()                       # all strings
spark.read.csv("people.csv", header=True, inferSchema=True).printSchema()     # guesses types
spark.read.csv("people.csv", header=True, schema="name STRING, age INT")      # declared

The stand-in reads a small CSV text three ways. The io.StringIO object plays the role of the file.

Python CSV as strings, inferred, and with a declared schema

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Sorting the string column puts "9" after "41", which is the bug a missing schema produces.

DataFrame creation recipes

createDataFrame accepts several kinds of input. Pick the one that matches the data you have:

Python
from pyspark.sql import Row

# 1. A list of tuples plus column names (types are inferred)
spark.createDataFrame([("Alice", 34), ("Bob", 29)], ["name", "age"])

# 2. A list of dictionaries (the keys become the column names)
spark.createDataFrame([{"name": "Alice", "age": 34}, {"name": "Bob", "age": 29}])

# 3. A list of Row objects
spark.createDataFrame([Row(name="Alice", age=34), Row(name="Bob", age=29)])

# 4. A pandas DataFrame (small data only: it passes through the driver)
spark.createDataFrame(pandas_df)

# 5. An RDD, with toDF or createDataFrame
rdd = sc.parallelize([("Alice", 34), ("Bob", 29)])
rdd.toDF(["name", "age"])
spark.createDataFrame(rdd, ["name", "age"])

Inference from Python values samples the data and can guess wrong: a column whose values are all None has no type to infer, and mixed ints and strings can fail. Passing an explicit schema removes the guessing.

Empty DataFrames with a schema

An empty DataFrame is useful as the starting point of a loop that unions results, or as a typed placeholder:

Python
from pyspark.sql.types import StructType, StructField, StringType, IntegerType

schema = StructType([
    StructField("name", StringType(), nullable=False),
    StructField("age", IntegerType(), nullable=True),
])
empty = spark.createDataFrame([], schema)
empty.printSchema()

Spark cannot infer types from zero rows, so a schema is required. A shorter DDL string works too: spark.createDataFrame([], "name string, age int").

Renaming, retyping and metadata

After the data exists you can fix the schema in two ways: toDF("a", "b") renames by position, and cast retypes a column. To attach metadata to a column (a description, units), alias it with a metadata dictionary:

Python
df = (df.withColumn("age", F.col("age").cast("int"))
        .withColumn("score", F.col("score").alias("score", metadata={"unit": "points"})))
df.schema["score"].metadata        # {'unit': 'points'}

Metadata travels with the schema and is saved by some formats, but ordinary transformations on the column may drop it, so treat it as a hint rather than a guarantee.

Python Inferring versus declaring a schema

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

DataFrame or RDD?

RDD DataFrame
Elements any Python or Java objects rows with a schema
Spark can look inside no, functions are opaque yes, columns are known
Optimizer none Catalyst plans, pushes filters, prunes columns
Storage objects, often large compact binary format
Code style functional (map, reduceByKey) SQL-like (select, groupBy, join)
Best for unstructured data, low-level control almost everything else

You can move between them: df.rdd gives an RDD of Row objects, and rdd.toDF(["a", "b"]) goes back. Prefer the DataFrame API whenever your data has columns. The same question answered both ways shows the difference in style.

Python Average age per city: RDD style and DataFrame style

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real versus stand-in. A real Spark DataFrame is partitioned across executors, lazy, and optimized before it runs; show triggers a job. The pandas stand-ins hold everything in memory in one process and run each line immediately. They imitate the table model and Spark's output format, not Spark's execution.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a DataFrame have that a plain RDD does not?

    RDDs are also partitioned, immutable and lazy. The schema is what lets Spark understand the data and apply the Catalyst optimizer.
  2. 2. You read a CSV with spark.read.csv('f.csv', header=True) and sort by an age column, but 9 sorts after 41. Why?

    Without inferSchema or an explicit schema all CSV columns are strings, and strings sort character by character: '41' comes before '9'.
  3. 3. Which call does NOT start a Spark job?

    printSchema only reads metadata. count, show and collect are actions that compute data.
  4. 4. What type does Spark infer for Python integers?

    Python integers become LongType. Use an explicit schema if you need IntegerType.

Summary

  • A DataFrame is an immutable, lazy, distributed table with named, typed columns, created from a SparkSession.
  • Create one from Python data with createDataFrame, from an RDD with toDF, or from files with spark.read.
  • The schema lists columns, types and nullability; it is inferred or declared with StructType or a DDL string. Declaring is faster and safer.
  • Use show, printSchema, columns, dtypes, count, take and describe to inspect; avoid collect on big data.
  • CSV files are read as strings unless you infer or declare types.
  • Prefer DataFrames over RDDs for structured data; df.rdd and rdd.toDF convert between them.

Next: transform columns in Columns and Expressions.