Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 40 ยท Projects

Project: A Data Analysis Pipeline

Build a small Julia data pipeline: read a CSV, clean missing values, compute revenue, group and summarize with DataFrames.jl, and plot, with a pandas port.

Most data work follows the same four steps: load a file, clean it, summarize it, and show the result. In this project you will build that pipeline in Julia with CSV.jl, DataFrames.jl and Plots.jl, using a small table of sales records. Each step is its own function, so the pipeline is easy to test and to extend with new questions.

Julia is not installed on this site, so the Julia code is read-only but complete. A pandas port of every step runs here, and its output is exactly what you should expect from the Julia version (apart from number formatting). If you have not met the library yet, DataFrames.jl introduces the tools used here, and Plotting covers the last step.

The pipeline

Load Read the CSV into a DataFrame.
Clean Drop rows with missing values.
Summarize Group and aggregate.
Show Plot the result.
1 / 2

The data

The table has ten sales: a date, a region, a product, the number of units and the price. One row (East, January 12) has no units. Save this as sales.csv, or in the examples below read it from a string:

Text
date,region,product,units,price
2026-01-05,North,Widget,10,2.50
2026-01-06,South,Widget,4,2.50
2026-01-09,North,Gadget,3,9.99
2026-01-12,East,Widget,,2.50
2026-02-02,South,Gadget,6,9.99
2026-02-03,East,Gadget,2,9.99
2026-02-10,North,Widget,8,2.60
2026-02-14,South,Widget,12,2.60
2026-03-01,East,Gadget,5,10.49
2026-03-09,North,Gadget,7,10.49

Step 1: load

CSV.read takes a source and a sink type. It guesses the column types, parses ISO dates into Date values, and turns an empty field into missing:

JULIA
using CSV, DataFrames, Statistics, Dates

function load_sales(source)
    return CSV.read(source, DataFrame)
end

df = load_sales("sales.csv")      # or: load_sales(IOBuffer(raw_text))
size(df)                          # (10, 5)
count(ismissing, df.units)        # 1

A DataFrame stores each column as a typed vector, so df.units is a Vector{Union{Missing, Int64}}: integers where present, missing where not. Julia's missing is a real value that propagates: missing * 2.5 is missing. That is useful, because a forgotten gap cannot silently turn into a zero, but it means a calculation over the raw column would give missing for that row.

Step 2: clean

We drop the rows whose units or price is missing, and then add a revenue column. dropmissing takes the columns to look at and returns a new frame; transform! adds the column in place:

JULIA
function clean_sales(df)
    clean = dropmissing(df, [:units, :price])
    transform!(clean, [:units, :price] => ByRow(*) => :revenue)
    return clean
end

clean = clean_sales(df)
nrow(clean)                       # 9

The pair syntax inputs => function => output is the heart of DataFrames.jl. Here ByRow(*) applies multiplication to each row's units and price. Without ByRow, the function would receive two whole column vectors, and * is not defined for a vector times a vector, so it would fail.

Step 3: summarize

The "split-apply-combine" pattern is groupby followed by combine. We want total, average and number of orders per region, sorted by total:

JULIA
function summarize_by_region(clean)
    by_region = combine(groupby(clean, :region),
        :revenue => sum  => :total,
        :revenue => mean => :average,
        nrow             => :orders)
    return sort(by_region, :total, rev = true)
end

summary = summarize_by_region(clean)
# 3ร—4 DataFrame
#  Row โ”‚ region  total    average  orders
#      โ”‚ String  Float64  Float64  Int64
# โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
#    1 โ”‚ North    149.2    37.3         4
#    2 โ”‚ South    101.14   33.71        3
#    3 โ”‚ East      72.43   36.22        2

(The exact display is formatted by DataFrames.jl; the values are what matter.) Each argument of combine has the form column => function => new_name. The special nrow counts the rows in each group and needs no input column.

Monthly totals use the same pattern. First derive a month from the date, then group by it:

JULIA
function monthly_revenue(clean)
    withmonth = transform(clean, :date => ByRow(d -> Dates.format(d, "yyyy-mm")) => :month)
    return combine(groupby(withmonth, :month), :revenue => sum => :total)
end

Overall statistics come straight from the Statistics standard library applied to a column:

JULIA
mean(clean.revenue), median(clean.revenue), std(clean.revenue)

Step 4: show

A bar chart of the regional totals with Plots.jl takes a few lines. String.(...) makes sure the labels are plain strings:

JULIA
using Plots

function plot_regions(summary)
    p = bar(String.(summary.region), summary.total,
            legend = false,
            ylabel = "Revenue",
            title = "Revenue by region")
    savefig(p, "revenue_by_region.png")
    return p
end

Then the whole program is a few lines that read like the plan:

JULIA
df      = load_sales("sales.csv")
clean   = clean_sales(df)
summary = summarize_by_region(clean)
plot_regions(summary)
println(summary)

Because every step is a function that takes a DataFrame and returns one, you can test each in isolation with @test (Testing with Test.jl) and reuse them on next month's file.

The pandas port

This port follows the same four steps. In pandas, missing numbers become NaN, groupby(...).agg(...) plays the role of combine, and a text bar chart stands in for the saved plot, since the pipeline's last step is a picture:

Python The pipeline in pandas

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

North leads with 149.20, South follows with 101.14, and East is last with 72.43. The standard deviation uses the sample formula, like std in Julia's Statistics library.

A second port computes the totals with a plain dictionary, which shows what groupby and combine do underneath: collect the rows of each key, then reduce each group.

Python What groupby does underneath

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Going further

  • Add a product breakdown with groupby(clean, [:region, :product]), which groups by two keys at once.
  • Join the table with a second one (a list of regional managers) using leftjoin(summary, managers, on = :region).
  • Handle messy input: CSV.read(file, DataFrame; missingstring = ["", "NA", "n/a"]) tells the reader which strings mean missing, and coalesce.(df.units, 0) replaces missing with a default when that is the right choice.
  • Wrap the steps in a package and run them as a script from the command line (see Packages and Environments).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does missing do when you multiply it by a number in Julia?

    missing propagates through arithmetic, so a gap cannot silently become a number.
  2. 2. In [:units, :price] => ByRow(*) => :revenue, what does ByRow do?

    Without ByRow, the function receives entire column vectors. ByRow(*) multiplies the two values in every row.
  3. 3. Which pair of functions implements split-apply-combine in DataFrames.jl?

    groupby splits the table by key and combine applies aggregations to each group and joins the results.
  4. 4. Why write each pipeline step as a function that takes and returns a DataFrame?

    Small functions with clear inputs and outputs are easy to test and compose.

Summary

  • A data pipeline is load, clean, summarize and show; write each step as a function from DataFrame to DataFrame.
  • CSV.read(source, DataFrame) infers types and turns empty fields into missing, which propagates through arithmetic.
  • dropmissing, transform! with ByRow, and groupby plus combine cover most cleaning and summarizing work.
  • The column => function => name pair syntax is shared by transform, select and combine.
  • pandas ports translate step by step: combine becomes groupby().agg(), missing becomes NaN.

Next: build and organize reusable code in Project: A Small Generic Package.