Skip to content
Tutorials Center
My learning

Topics

Lesson 2 of 39 ยท Getting Started

The KDD Process and CRISP-DM

Follow the steps of knowledge discovery (KDD) and the CRISP-DM cycle, from business understanding to deployment, and run a miniature pipeline end to end in Python.

Running one algorithm is the small part of a mining project. Most of the work happens before and after it: finding the right question, collecting and cleaning data, checking the result and putting it to use. Two well-known recipes describe this work: the KDD process and CRISP-DM. They say nearly the same thing in different words, and both are loops, not straight lines.

The KDD process

KDD stands for knowledge discovery in databases. Its classic steps are:

  1. Selection: choose the data that is relevant to the goal.
  2. Preprocessing (cleaning): remove noise and duplicates, handle missing values.
  3. Transformation: scale, encode, aggregate or reduce the data into a form the algorithm accepts.
  4. Data mining: apply an algorithm to find patterns.
  5. Interpretation and evaluation: judge which patterns are real and useful.
  6. Knowledge: act on the patterns, document them, and share them.

Data mining itself is only step 4. That is why this course gives whole chapters to preprocessing and exploration before touching any algorithm.

flowchart LR
    A["Raw data"] --> B["Selection"]
    B --> C["Preprocessing"]
    C --> D["Transformation"]
    D --> E["Data mining"]
    E --> F["Evaluation"]
    F --> G["Knowledge"]
    F -. "not good enough" .-> B

The dashed arrow is the point: results send you back to earlier steps.

CRISP-DM

CRISP-DM (Cross-Industry Standard Process for Data Mining) is the most widely used project template. It has six phases that you cycle through:

CRISP-DM, phases 1 to 3

Business understanding Goal and success measure.
Data understanding Collect, explore, check quality.
Data preparation Clean, join, transform.

Usually 60 to 80 percent of the effort.

1 / 2

The difference from KDD is the strong start and end. CRISP-DM begins with business understanding: what decision will this support, and how will we know it worked? "Reduce churn by 5 percent" is a goal; "build a classifier" is not. It also ends with deployment and monitoring, because a model that nobody uses, or that quietly goes stale, is wasted work.

A miniature pipeline

Here is the whole loop on a tiny table, each step labeled. The business question: which customers will respond to an offer?

Python KDD in thirty lines

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice the last line: if only 20 percent respond, a model that always answers "no" is 80 percent accurate. Evaluation is not just reading one number. Better measures such as precision and recall come in the classification chapter; for now, remember to compare against a trivial baseline.

Iteration in practice

Real projects loop. A poor result may mean the question was wrong (back to business understanding), a feature is missing (back to preparation), or the algorithm is a poor fit (back to modeling). Keep each step reproducible by putting it in a function, so you can rerun it after a change.

Python Steps as functions, so the loop is cheap

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Common reasons projects fail

  • No clear goal: the team mines first and looks for a question afterward.
  • Bad data: wrong units, duplicates, leakage of the answer into the inputs.
  • Evaluating on training data: the model looks great and fails in production.
  • No deployment plan: insights stay in a slide deck.
  • Ignoring ethics: using sensitive attributes or building something people would not accept.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In the KDD process, which step is the data mining algorithm itself?

    Mining is one step; selection, preprocessing and transformation come before it and evaluation after it.
  2. 2. What does CRISP-DM add at the start compared with KDD?

    CRISP-DM starts by defining the business goal and success measure.
  3. 3. Which phase usually takes the largest share of project time?

    Cleaning, joining and transforming data is typically the biggest part of the effort.
  4. 4. A model scores 80 percent accuracy and 80 percent of the rows are class 0. What does this suggest?

    Compare with a trivial baseline before celebrating a number.

Summary

  • KDD: selection, preprocessing, transformation, mining, evaluation, knowledge.
  • CRISP-DM: business understanding, data understanding, data preparation, modeling, evaluation, deployment.
  • Both are loops; evaluation can send you back to any earlier step.
  • Most effort goes into understanding and preparing the data.
  • Define the goal first, compare against a baseline, and plan deployment.

Next: Data Mining Tasks and Applications