Skip to content
Tutorials Center
My learning

Topics

Lesson 37 of 44 ยท Real-world Charts and Integration

Loading and Cleaning Data

Load CSV and JSON with d3-fetch, convert types with autoType and row functions, parse dates, clean missing values, and summarize with group, rollup, sum and cumsum.

Real data is never ready to draw. Numbers arrive as strings, dates have odd formats, some rows are empty, and what you need is rarely the table you were given. D3 includes a complete toolkit for this: d3-fetch to load files, d3-dsv to parse them, d3-time-format for dates and d3-array for grouping and summarizing. A large part of D3 skill is data preparation, so this lesson is worth reading slowly.

Loading files

JavaScript
const rows = await d3.csv("sales.csv");
const obj  = await d3.json("config.json");
const text = await d3.text("notes.txt");

Each function is a thin wrapper around fetch that returns a promise: d3.csv, d3.tsv, d3.json, d3.text, d3.xml, d3.html, d3.image and d3.blob. They throw if the HTTP status is not OK, so wrap them in try/catch and show the reader a message when loading fails. A CSV from another domain needs CORS headers on the server; a local file opened from the disk will not load, so use a small static server during development.

d3.csv returns an array of objects whose values are all strings. That is the first thing to fix. We will use d3.csvParse(text), the same parser working on a string, so the examples below run without any network.

Converting types

The easiest fix is d3.autoType: it guesses numbers, booleans, null for empty fields, and ISO dates. Pass it as the row function (second argument of d3.csv) or call it after parsing:

JavaScript
const rows = await d3.csv("sales.csv", d3.autoType);

autoType is convenient but guesses. A column of postal codes ("02134") would turn into the number 2134, and "1e3" into 1000. For production data write your own row function, which also lets you rename fields and drop columns:

JavaScript
const parseDate = d3.utcParse("%d/%m/%Y");
const rows = await d3.csv("sales.csv", d => ({
  date: parseDate(d.Date),
  region: d.Region.trim(),
  units: +d.Units,
  price: d.Price === "" ? null : +d.Price
}));

The unary plus converts strings to numbers (+"" is 0, and +"abc" is NaN, so check for empty strings first). Returning undefined from a row function removes that row, which is a convenient filter.

Parse and convert a CSV string

Printing the shape of the data at each step is the best debugging habit. Note raw.columns: d3.csvParse attaches the header list to the array, useful for building tables.

Dates

D3 has a formatter and a parser that share the same format codes (like strftime):

JavaScript
const fmt = d3.timeFormat("%b %d, %Y");        // Date -> string  "Jan 03, 2025"
const parse = d3.timeParse("%Y-%m-%d");        // string -> Date  (local time)
const parseUtc = d3.utcParse("%Y-%m-%d");      // string -> Date  (UTC)

Common codes: %Y year, %m month number, %B month name, %b abbreviated month, %d day, %H:%M:%S time, %A weekday, %Z time zone. If a string cannot be parsed the result is null: filter for that. Prefer UTC variants for date-only data so charts do not shift by a day for readers in other time zones. Format numbers the same way with d3.format(",.2f") (thousands separator, two decimals), d3.format(".0%") or d3.format("~s") for SI units such as 1.2M.

Cleaning checklist

  1. Trim whitespace and fix case in categories ("North " vs "North").
  2. Treat empty strings, "N/A", "-" as null, not 0.
  3. Remove duplicates (d3.rollup or a Set of keys).
  4. Check ranges: d3.extent, d3.min, d3.max, plus d3.median and d3.quantile for outliers.
  5. Sort the data (d3.ascending, d3.descending, or (a, b) => a.date - b.date) before building lines.
  6. Keep an eye on NaN: it breaks scale domains silently (d3.max ignores it, but + does not).

Group, rollup and summarize

d3-array replaces the hand-written loops you would otherwise write:

JavaScript
d3.group(rows, d => d.region);                 // Map: region -> array of rows
d3.rollup(rows, v => d3.sum(v, d => d.units), d => d.region);  // Map: region -> total
d3.rollups(rows, v => v.length, d => d.region);                // array of [key, value]
d3.index(rows, d => d.id);                     // Map: id -> single row
d3.flatRollup(rows, v => d3.mean(v, d => d.price), d => d.region, d => d.month); // flat tuples

Grouping and rollup accept several key functions to nest, and d3.sort, d3.groupSort and d3.cumsum are available too. d3.bin groups numbers into intervals (see Chord Diagrams and Binning).

Summaries with rollup and group

Array.from(map, ([key, value]) => ({...})) is the standard way to turn a Map result into the array-of-objects form that data joins expect.

Wide to long (and back)

Spreadsheet-style data has one column per series (Jan, Feb, Mar). Charts that draw one mark per value need the long form: one row per (series, value). Reshape with flatMap:

JavaScript
const long = wide.flatMap(r => ["Jan", "Feb", "Mar"].map(m => ({ name: r.name, month: m, value: +r[m] })));

Group by name then, and you have one array per line to draw.

Loading more than one file

Promise.all fetches files in parallel: const [a, b] = await Promise.all([d3.csv("a.csv"), d3.json("b.json")]);. Handle failures once with try/catch and draw an error message in the chart area. A chart that stays blank with the error hidden in the console is a poor experience.

flowchart LR
  F["d3.csv / d3.json"] --> T["Row function: types, dates, trim"]
  T --> C["Clean: nulls, duplicates, sort"]
  C --> S["Summarize: group, rollup"]
  S --> D["Array of plain objects"]
  D --> J["Scales and data join"]

Prepare the data first; drawing is the short last step.

Common mistakes

  • Forgetting that d3.csv values are strings, so d3.max compares text and "9" > "10".
  • Trusting autoType for identifiers, phone numbers or leading-zero codes.
  • Parsing dates in local time and getting off-by-one days in other time zones.
  • Calling d3.csv without await and drawing before the data exists.
  • Mutating arrays that other charts share (sort works in place): copy with [...rows].sort(...) or d3.sort(rows, ...).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What type are the values returned by d3.csv('f.csv') without a row function?

    CSV is text, so every field is a string until you convert it, with d3.autoType or your own row function.
  2. 2. What does returning undefined from a row function do?

    Rows for which the function returns undefined are filtered out, which is handy for cleaning.
  3. 3. Which call computes the total of units for each region?

    rollup groups by the key and reduces each group with the function, here a sum.
  4. 4. Why prefer d3.utcParse for date-only strings?

    Local-time parsing can display the previous or next day in some time zones; UTC keeps the calendar date stable.

Summary

  • d3.csv, d3.json and friends load files with promises; always handle errors.
  • Convert types with d3.autoType or, better, your own row function that also renames, trims and filters.
  • Parse and format dates with timeParse/utcParse and timeFormat; numbers with d3.format.
  • Use group, rollup, groupSort, cumsum and extent from d3-array to reshape data before drawing.

Next: D3 with React, Vue and Other Frameworks