Skip to content
Tutorials Center
My learning

Topics

Lesson 3 of 76 ยท What Data Science Is

The Data Science Workflow

Walk through the data science workflow stage by stage, see what each stage produces, where the time really goes, and build a script-friendly pipeline of functions.

A data science project is easier to manage when you can name its stages. Without a map, people jump straight to "let's train a model", discover three weeks later that the data was wrong, and start over. A workflow gives you checkpoints: at each one you know what you should have produced and what question to ask before moving on.

The best-known description is CRISP-DM (Cross-Industry Standard Process for Data Mining), which dates from the 1990s and is still widely used. This lesson uses a modern, slightly simplified version of its stages and shows how to turn them into code you can rerun.

The stages and their deliverables

Every stage should end with something you can point to, not just a feeling that you "looked at the data".

flowchart TD
    A["1. Understand the problem<br/>deliverable: written question and success criteria"] --> B["2. Get the data<br/>deliverable: raw data snapshot and notes"]
    B --> C["3. Clean and prepare<br/>deliverable: tidy dataset plus a cleaning script"]
    C --> D["4. Explore<br/>deliverable: findings and charts"]
    D --> E["5. Model or test<br/>deliverable: baseline and candidate models"]
    E --> F["6. Evaluate<br/>deliverable: honest metrics and error analysis"]
    F --> G["7. Communicate and deploy<br/>deliverable: report, decision, running system"]
    F -.->|"iterate"| A
    D -.->|"iterate"| B

Seven stages, each with a concrete deliverable; dotted arrows show the common reasons to go back.

Stage You ask You produce
Understand the problem What decision will this change? A one-paragraph question and a success measure
Get the data Where is it, who owns it, can I use it? A raw snapshot, a data dictionary
Clean and prepare Is it correct and consistent? A tidy table and the code that built it
Explore What does it look like? Charts, summary tables, a list of surprises
Model or test What is the simplest thing that works? A baseline and one or two better candidates
Evaluate Would it work on new data? Metrics on held-out data, a list of failure cases
Communicate and deploy Who needs to know and what will they do? A short report, a decision, maybe a service

Where the time really goes

Surveys of working data scientists keep finding the same thing: preparing data takes the largest share of the project, often more than half. Modelling, which gets most of the attention in courses, is often a small slice. The exact split varies, so treat the numbers below as a rough picture rather than a measurement.

Python A typical split of project time

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

This is a good reason to learn cleaning and exploration well: the reliability of every later step depends on them.

Iteration is normal

The dotted arrows in the diagram are not failures. You will go back, for example:

  • Exploration reveals that a column means something different from what the documentation says, so you return to the data owner.
  • A model performs suspiciously well, so you search for leakage in the preparation step (Train/Test Splits and Cross-Validation explains the idea).
  • Stakeholders see an early chart and realize they asked the wrong question.

Plan for it. Short loops with a rough end-to-end version (a "thin slice") beat one long pass where each stage is polished before the next begins. A crude baseline built on day one tells you if the project is feasible.

Notebook-free and script-friendly

Many people explore in notebooks, which is excellent for looking around. For anything you will rerun, though, move the logic into plain functions in a script or module. Reasons:

  • Functions can be tested (Testing Data Code and Data Validation covers this) and reused.
  • A script runs top to bottom every time, so there is no hidden state from cells run out of order.
  • Version control shows clean diffs of .py files, while notebooks produce noisy ones.
  • A scheduled job or a colleague can run the whole thing with one command.

A useful pattern is one small function per stage, with a main function that calls them in order. Each function takes data in and returns data out, with no global variables.

A skeleton pipeline

Here is a complete miniature pipeline built from functions. The data is generated inside load_data so the example is self-contained; in a real project that function would read a file or query a database.

Python A pipeline of small functions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each function can be tried alone and replaced without breaking the others. If a colleague changes how the data is loaded, only load_data changes. This is the shape you will see in the projects at the end of the course.

Recording decisions

At each stage, write down what you decided and why: which rows you dropped, which threshold you chose, what you assumed. A short NOTES.md or comments at the top of each function is enough. Six months later, "why did we exclude 2020?" will be a question someone asks, and the answer should not live only in your memory. Reproducible Analysis turns this into a full habit.

Tip: Build the ugliest possible end-to-end version first, even if the "model" is just the overall average. You then improve whichever stage hurts most, instead of perfecting one stage in the dark.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which stage usually takes the largest share of project time?

    Cleaning and preparation tends to dominate, because every later step depends on correct, consistent data.
  2. 2. A model scores almost perfectly on its first try. What is the best next move?

    Results that look too good usually mean the model saw information it should not have had, or the evaluation is flawed.
  3. 3. Why prefer functions in a script to a long chain of notebook cells for a repeatable analysis?

    Hidden state from cells run out of order is a classic source of irreproducible results. Functions make dependencies explicit.
  4. 4. What is a good deliverable for the stage 'Understand the problem'?

    Before touching data you should have the question and the criteria for success in writing, so the team agrees on what done means.

Summary

  • The workflow has seven stages: understand, get data, clean, explore, model or test, evaluate, communicate or deploy.
  • Each stage should end with a concrete deliverable.
  • Cleaning and exploring take far more time than modelling in most real projects.
  • Iteration is normal; build a thin end-to-end slice early.
  • Put reusable logic in small functions and keep a short log of decisions.

Next: Framing a Good Question, the first stage and the one that decides whether everything else matters.