Skip to content
Tutorials Center
My learning

Topics

Lesson 54 of 64 ยท Quality and Production

Building Data Pipelines

Build clean pandas pipelines: small functions that take and return DataFrames, pipe and method chaining, row-count logging, idempotent steps and configuration.

A notebook that works once is an experiment. A script you can rerun next month on new data, hand to a colleague and trust is a pipeline. The difference is not fancy tooling. It is structure: each step does one job, takes a DataFrame, returns a DataFrame, and can be tested and reordered on its own.

This lesson shows how to turn a pile of exploratory lines into a readable pipeline with plain pandas: small functions, pipe, method chaining, row-count logging, idempotent steps and one place for configuration. No framework needed; the same shape scales up to schedulers like Airflow or Dagster later.

The shape of a pipeline

Most data jobs have four stages, and keeping them apart is the biggest improvement you can make:

flowchart LR
    A["load<br/>read raw data"] --> B["clean<br/>fix types, names,<br/>drop junk"]
    B --> C["transform<br/>derive columns,<br/>join, aggregate"]
    C --> D["report<br/>write tables,<br/>charts, files"]
    X["config<br/>paths, thresholds,<br/>column names"] -.-> A
    X -.-> B
    X -.-> C
    X -.-> D

Four stages with one shared configuration; each stage only talks to its neighbours through DataFrames.

  • Load only reads. It does not fix anything, so you can look at the raw data later.
  • Clean makes the data trustworthy: names, dtypes, whitespace, impossible values. See Data Cleaning in Practice.
  • Transform is your business logic: new columns, joins, groupings.
  • Report is the only stage with side effects such as writing files or drawing charts.

When something is wrong, the stage boundaries tell you where to look.

Step functions: DataFrame in, DataFrame out

The building block is a function with a simple contract: it takes a DataFrame (plus options) and returns a new DataFrame, leaving its input untouched. With Copy-on-Write, assign, rename, query and friends already behave this way, so your functions can be chains of them.

Python
def drop_cancelled(df: pd.DataFrame) -> pd.DataFrame:
    return df.loc[df["status"] != "cancelled"]

No globals, no print inside, no reading files. That makes every step trivial to test with a three-row fixture.

pipe and method chaining

DataFrame.pipe(func, *args, **kwargs) calls func(df, *args, **kwargs) and returns the result. It lets your own functions sit in a method chain next to built-in methods, so the pipeline reads top to bottom like a recipe instead of inside-out like c(b(a(df))):

Python pipe turns nested calls into a chain

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Method chaining has one rule of style: put each step on its own line inside parentheses. Keep chains short enough to read, and give a name to an intermediate result when you want to inspect it. A long chain of fifteen steps is a sign that it should be split into two or three named functions.

Make every step visible: log the row count

The most common silent failure in a pipeline is a step that removes (or multiplies) more rows than you intended. A tiny decorator can report the shape before and after each step, with no change to the step itself:

Python A decorator that logs rows in and out

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In a real job you would use the logging module instead of print, so messages carry timestamps and levels and can go to a file. The idea is identical. A line like drop_negative 4 -> 3 rows makes a surprising number jump out immediately, and it gives you the numbers for a "data quality" note in your report.

Idempotent steps

A step is idempotent if running it twice gives the same result as running it once. Pipelines get rerun: after a crash, with overlapping date ranges, or because someone clicked the button twice. Idempotent steps make reruns safe.

Examples of the difference:

  • Good: df["city"].str.strip().str.lower(), drop_duplicates, fillna("unknown"), converting a column to a dtype.
  • Bad: df["price"] * 100 (every rerun multiplies again), appending today's rows to a file without checking whether they are already there.

Two habits help. First, never overwrite the raw input; read from it and write results to a different place. Second, when output must be stored, replace the whole output for a given day or key instead of appending to it. You can check idempotence mechanically: run the step on its own output and compare.

One place for configuration

Magic strings and thresholds scattered through the code are the second great source of pipeline bugs. Gather them into one config object. A plain dictionary or a frozen dataclass is enough, and it can be loaded from a JSON or YAML file in a real job.

Python A configured, idempotent pipeline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Because run is just function composition, it is also easy to test end to end: feed it a ten-row string, compare with the expected text.

Common mistakes

  • Mutating inputs. A step that edits its argument in place surprises its caller. Return new frames.
  • Hidden state. Reading a global DataFrame or a file inside a transform step makes it untestable. Pass data in as arguments.
  • One giant function. If you cannot name a step in three words, it is doing too much.
  • Reporting inside cleaning. Printing or writing in the middle of a transform stage hides side effects; keep them in the report stage.
  • Silent row loss. Log the row counts, and consider a final check that totals still add up (see Common pandas Pitfalls).

Where to go next

When one script is not enough, the same functions slot into a workflow tool that adds scheduling, retries and dependency graphs. You can also save intermediate results (for example as Parquet files) so a failed late step does not force a re-run of the early ones. Those tools are outside the browser, but they all want what you just built: small, pure, well-named steps. Before you build one on real data, try the sales analysis project.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does df.pipe(f, x=1) do?

    pipe passes the DataFrame as the first argument to your function and returns what the function returns.
  2. 2. Which pipeline stage should be the only one to write files or draw charts?

    Keeping side effects in the report stage leaves load, clean and transform as pure functions that are easy to test and rerun.
  3. 3. Which step is NOT idempotent?

    Multiplying by 100 compounds with every rerun. The others give the same result when applied to their own output.
  4. 4. Why log the number of rows before and after each step?

    Unexpected row-count changes are the most common silent failure; a logged line makes them visible straight away.

Summary

  • Split a job into load, clean, transform and report, and keep side effects in the last stage.
  • Write steps as functions that take a DataFrame and return a new one without changing the input.
  • Use pipe and parenthesised method chains so the code reads top to bottom.
  • Log row counts per step (a decorator makes it one line) and keep a check that totals still add up.
  • Make steps idempotent: do not overwrite raw data, replace outputs instead of appending.
  • Put thresholds, names and paths in one config object instead of scattering literals.

Next: Project: Sales Analysis