Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 37 ยท IO and Formats

Reading and Writing CSV

Read and write CSV files with Polars. Control separators, headers, types, null markers, dates and bad rows, then learn the lazy scan_csv for large files.

CSV is the world's most common data exchange format, and also one of the messiest: different separators, missing headers, odd null markers, dates in a dozen layouts. pl.read_csv handles all of it with a handful of well-named options, and it is one of the fastest CSV readers available because it splits the file into chunks and parses them in parallel.

About this site's runner. The examples here read CSV text from a string with io.StringIO, because the browser sandbox has no data files on disk. The functions and options are exactly the ones you use with a real path such as pl.read_csv("data/sales.csv"). The browser build of Polars cannot run lazy file scans (scan_csv) or the Parquet and JSON readers, so those are shown as read-only snippets and labelled as such.

The basic read

Python read_csv from text

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Polars looks at the first 100 rows (infer_schema_length=100) to guess each column's type. Integers become Int64, decimals Float64, everything else String.

Separators, headers and names

Python Semicolons, no header, custom names

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Without a header Polars names the columns column_1, column_2, and so on. Tab-separated files use separator="\t". For files with a European decimal comma (3,14), add decimal_comma=True together with a different separator.

Controlling types

Automatic inference is convenient but can be wrong: a zip code like 02134 must stay text, an ID column with 100,000 digits might overflow the first-100-rows guess. Take control with schema_overrides (change some columns), schema (describe all columns), or infer_schema=False (read everything as text, then cast).

Python schema_overrides keeps zip codes as text

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Reading as Int32 instead of Int64 halves the memory of that column, a cheap saving covered in data types and memory.

If a column starts with integers and a decimal or text appears after row 100, inference fails with a "could not parse" error. Increase infer_schema_length (or use None to scan the whole file), or give that column an explicit type.

Missing values and dirty cells

Polars treats an empty field as null. Other markers such as NA, - or ? stay as text unless you list them in null_values.

Python null_values and ignore_errors

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

ignore_errors=True is a blunt tool: it silently nulls everything it cannot parse. Prefer null_values when you know the markers, and check null_count() afterwards.

Dates

Add try_parse_dates=True and Polars converts columns that look like ISO dates or datetimes. For other layouts, read as text and parse with an explicit format, which is stricter and faster.

Python Parsing dates

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Choosing rows and columns

To save time and memory, read only what you need: columns=[...] picks columns, n_rows= reads the first rows (perfect for peeking at a huge file), skip_rows= jumps over junk at the top, and comment_prefix="#" ignores comment lines.

Python columns, n_rows, skip_rows

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Quoting and ragged lines

Fields containing the separator are wrapped in quotes, and Polars handles "New York, NY" correctly by default (quote_char='"'). If some rows have fewer fields than the header, add truncate_ragged_lines=True rather than failing.

Writing CSV

df.write_csv() with no argument returns the text; with a path it writes a file. Useful options: separator, include_header, null_value (what to print for nulls, empty by default), float_precision, quote_style, and date_format / datetime_format.

Python write_csv options

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice the comma inside "Ana, the first": Polars quotes it automatically so the file reads back correctly.

Large files: scan_csv (read-only)

read_csv loads the whole file. For big files, pl.scan_csv builds a lazy query, and Polars reads only the columns and rows the query needs. This snippet cannot run in the browser build, but it is how you work on a laptop with files bigger than comfortable:

Python
import polars as pl

result = (
    pl.scan_csv("data/trips-*.csv")            # a glob reads many files
    .filter(pl.col("distance") > 10)           # pushed into the reader
    .group_by("city")
    .agg(pl.col("fare").mean())
    .collect()
)

You will meet the idea behind this in lazy evaluation and see its plan in query plans and optimizations. To write a huge result without building it in memory, a lazy frame has sink_csv("out.csv").

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A CSV column of zip codes loses its leading zeros after read_csv. What is the cleanest fix?

    Polars inferred an integer type. Declaring the column as String keeps the text exactly as written.
  2. 2. How are empty fields in a CSV treated by default?

    Empty fields become null. Other markers such as NA must be listed in null_values.
  3. 3. What does ignore_errors=True do when a cell cannot be parsed as the column type?

    It silently replaces unparsable cells with null, so check null_count() afterwards or prefer null_values.
  4. 4. Which function reads CSV lazily so that only the needed columns and rows are loaded?

    scan_csv returns a LazyFrame; filters and column selections are pushed down into the reader.

Summary

  • pl.read_csv is fast and has options for separators, headers, names, rows, columns and comments.
  • Control types with schema_overrides and infer_schema_length; keep codes and IDs as String.
  • Empty fields are null; list other markers in null_values.
  • Use try_parse_dates=True for ISO dates and str.to_date(format) for other layouts.
  • write_csv supports separator, null_value, float_precision and date formats.
  • scan_csv + sink_csv handle big files lazily (not runnable in the browser build).

Next: Parquet, JSON and Arrow IPC