Skip to content
Tutorials Center
My learning

Topics

Lesson 11 of 48 ยท DataFrames and Spark SQL

Sampling and Getting Rows Back

Take samples of a Spark DataFrame and bring rows back safely: sample, sampleBy, takeSample, first, head, tail, take, collect and toLocalIterator.

A Spark DataFrame can hold billions of rows spread over many machines, but your screen and your Python process can only hold a few thousand. Two skills bridge that gap. Sampling lets you work with a small, representative piece of a big table. Getting rows back (to the driver) lets you inspect, print or hand results to other code, and doing it carelessly is one of the most common ways to crash a Spark job.

This lesson covers sample, sampleBy, RDD takeSample, the row-retrieval methods (first, head, take, tail, collect) and toLocalIterator. It builds on DataFrames: Creating and Inspecting and the idea of actions from Lazy Evaluation, Transformations and Actions. Spark cannot run on this site, so the PySpark code is read-only and the runnable examples are plain-Python stand-ins.

Random samples with sample

DataFrame.sample returns a random subset. It is a transformation: nothing runs until an action asks for rows.

Python
df = spark.range(1000).withColumnRenamed("id", "n")

small = df.sample(fraction=0.1, seed=42)
small.count()          # about 100, but not exactly 100

The important points:

  • fraction is a probability per row, not a target size. Each row is kept independently with that probability, so sample(0.1) of 1,000 rows gives roughly 100, sometimes 93, sometimes 108. If you need exactly N rows, see takeSample and limit below.
  • seed makes the sample repeatable on the same data and partitioning. Without it every action may give a different sample, which is confusing when you cache nothing and call two actions.
  • withReplacement=True allows the same row to appear more than once (a bootstrap sample). The default is False.

Sampling still has to read the data to flip a coin for every row, so it saves downstream work, not the scan itself. For a quick look at a huge Parquet table, limit is cheaper.

Stratified samples with sampleBy

A plain sample may under-represent a rare class. If 1% of customers churn, a 1% sample might contain none of them. sampleBy takes a different fraction per value of a column:

Python
fractions = {"churn": 1.0, "stay": 0.05}          # keep every churner, 5% of the rest
balanced = customers.sampleBy("status", fractions, seed=7)
balanced.groupBy("status").count().show()

Values missing from the dictionary get fraction 0, so list every class. This is the standard way to build a smaller, balanced training set for MLlib.

Python Sampling and stratified sampling by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Exact-size samples with takeSample

When you need exactly N random items, use the RDD action takeSample. It returns a Python list on the driver, not a DataFrame:

Python
picked = df.rdd.takeSample(withReplacement=False, num=5, seed=1)
# [Row(n=812), Row(n=7), ...]

Because it returns a list, num must be small enough to fit in driver memory. Another way to get an exact count is df.orderBy(F.rand(seed)).limit(n), which stays a DataFrame but sorts everything first, so it is expensive.

Getting rows back to the driver

Every method below is an action: it runs the job and ships rows to the driver. The difference is how many.

Method Returns Rows moved
df.first() one Row (or None if empty) 1
df.head() one Row; head(n) returns a list 1 or n
df.take(n) list of the first n Row objects n
df.tail(n) list of the last n rows n (reads the whole data)
df.limit(n) a new DataFrame (a transformation) none yet
df.collect() list of all rows everything
df.toLocalIterator() iterator over all rows, one partition at a time everything, in pieces
Python
row = df.first()
row["n"]                 # access by name
row.n                    # or as an attribute
row.asDict()             # {'n': 0}

df.take(3)               # [Row(n=0), Row(n=1), Row(n=2)]
df.tail(3)               # last three rows
df.limit(3).show()       # a DataFrame, printed

take and head are cheap: Spark scans partitions one by one until it has enough rows, so on a large table take(5) often reads only the first partition. tail is the opposite: it must read the whole dataset because the end of a distributed table has no shortcut. Remember that "first" only means something after an orderBy; an unsorted DataFrame gives no promise about which rows come first.

Why collect() is risky

collect() pulls every row into the driver's memory. On a table with 500 million rows the driver runs out of memory (OutOfMemoryError or "Total size of serialized results ... is bigger than spark.driver.maxResultSize") and the whole application dies. The cluster was never the problem; one machine receiving everything was.

Safer habits:

  1. Reduce first. Aggregate, filter or limit before you collect, so the result is small.
  2. Use show(n), take(n) or limit(n) while exploring.
  3. Write big results to storage with df.write instead of returning them.
  4. If you really must walk through all rows in Python, use toLocalIterator().
Python
for row in df.toLocalIterator():           # one partition in driver memory at a time
    handle(row)

toLocalIterator() keeps only one partition's rows in memory at once, so the driver can stream through data larger than its memory. The price is speed: it processes partitions sequentially. Check the prefetchPartitions option if you want the next partition fetched while you work. Even so, ask whether a Spark transformation or a UDF could do the job on the cluster instead of looping in Python.

Python take, tail, collect and toLocalIterator on partitions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real versus stand-in. Real take launches small jobs on increasing numbers of partitions and ships Row objects through the network to the driver. The stand-in only shows the access pattern: take can stop early, tail and collect cannot.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. df.sample(0.1) on a 1,000-row DataFrame returns exactly 100 rows.

    fraction is a per-row probability, so the result is roughly, not exactly, 10% of the rows.
  2. 2. Which method keeps all rare-class rows while sampling the common class lightly?

    sampleBy takes a separate fraction for each value of a column, which gives a stratified sample.
  3. 3. Which is the safest way to loop over every row of a very large DataFrame in Python?

    toLocalIterator brings one partition at a time to the driver, so memory use stays bounded.
  4. 4. Why is df.tail(5) slower than df.take(5) on a large table?

    take can stop after the first partitions, but the end of a distributed table can only be found by reading everything.

Summary

  • sample(fraction, seed) keeps each row with a probability, so its size is approximate; set a seed for repeatability.
  • sampleBy gives stratified samples by a per-value fraction; list every class.
  • takeSample returns an exact number of random items as a Python list.
  • first, head, take and limit are cheap ways to look at data; tail reads everything.
  • collect() moves every row to the driver and can crash it; reduce first, write to storage, or use toLocalIterator().

Next: add, rename and drop columns in Columns and Expressions.