Skip to content
Tutorials Center
My learning

Topics

Lesson 1 of 39 ยท Getting Started

What Is Data Mining?

Learn what data mining is, how it differs from statistics and machine learning, what patterns it finds, and where it is used, with runnable Python examples.

Every shop, hospital, bank and website records data about what happens. A supermarket stores each receipt, a phone company stores each call, a web server stores each click. After a few years the records fill terabytes, and nobody can read them line by line. Data mining is the set of techniques that digs through such piles and pulls out patterns that are valid, new, useful and understandable.

The name is a little misleading. Mining gold is about getting gold, not rock. Mining data is about getting knowledge from data, and that is why you will also meet the longer phrase knowledge discovery in databases (KDD). In this course you learn the ideas, the algorithms behind them, and how to run each one in Python in your browser.

A first taste

Imagine a small store with a few receipts. A person can spot a pattern in five receipts. A mining algorithm can spot one in fifty million. Run this to see the smallest version of the idea: counting which items occur together.

Python Which items are bought together?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The pair bread, butter occurs in half of the receipts. That is a pattern: it was not written anywhere in advance, it came out of the data. A store could put the two items on neighboring shelves. The same idea, scaled up and made efficient, is the Apriori algorithm you will build later.

What counts as data mining

A good working definition has four parts. Mining finds patterns that are:

  1. Valid: they hold on new data, not only on the rows you looked at.
  2. Novel: they tell you something you did not already know.
  3. Useful: someone can act on them.
  4. Understandable: a person can explain them, at least roughly.

Looking up one customer's address in a database is not mining; it is a query. Asking "which customers are likely to leave next month?" is mining, because the answer is not stored anywhere and must be learned from many records.

Data mining, statistics and machine learning

The three fields overlap heavily, and people use the words loosely. The differences are mostly in focus.

Field Main question Typical size of data
Statistics Is this effect real, and how sure are we? Small samples, designed experiments
Machine learning How do we build a model that predicts well? Any size, model-centered
Data mining What useful patterns are hiding in this large, messy store? Large, found data, often from databases

Data mining borrows from both: statistics for testing and summarizing, machine learning for the algorithms, and database systems for handling volume. Its signature is the whole pipeline, from raw tables to a finished insight. If you want the model-building side in depth, see Machine Learning and scikit-learn; this course uses them as tools.

What kinds of patterns exist

Mining tasks fall into a few families. You will study each in its own chapter.

Four families of patterns

Associations Items that occur together.
Classification Predict a category.
Clusters Natural groups.
Anomalies Rare odd records.

Text and web mining apply the same ideas to documents and links.

1 / 2
  • Association rules find "if A then often B" relations. Chapter 4.
  • Classification and prediction learn from labeled examples to label new ones. Chapter 5.
  • Clustering groups similar records with no labels. Chapter 6.
  • Anomaly detection finds records that do not fit. Chapter 7.
  • Text and web mining extract patterns from documents and links. Chapter 8.

Tasks that describe the existing data are called descriptive; tasks that estimate unknown values are predictive. The next lessons expand this view in Data Mining Tasks and Applications.

The same table, mined two ways

A table can be mined in two ways, depending on whether it has a target column.

Python Predictive versus descriptive

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The tree needed the buys column; the clustering did not. Both are data mining.

Where data mining is used

  • Retail: basket analysis, promotions, demand forecasting.
  • Banking: credit scoring, fraud detection.
  • Healthcare: disease risk, patient groups, drug-interaction signals.
  • Telecom and subscriptions: churn prediction, plan recommendations.
  • Web and media: recommendations, spam filtering, trending topics.
  • Science and industry: sensor monitoring, genomics, predictive maintenance.

Limits and responsibility

Mining finds correlations, and a correlation is not always a cause. Patterns can also be accidents of a small sample, or reflect bias in the data (for example, past hiring decisions). Always check that a pattern holds on data the algorithm has not seen, ask whether it makes sense, and think about privacy. We return to these points in The KDD Process and CRISP-DM.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which task is data mining rather than a simple database query?

    The prediction is not stored anywhere; it must be learned from many records. The others just retrieve or count stored facts.
  2. 2. Which of these is NOT one of the four qualities of a good mined pattern?

    Patterns should be valid, novel, useful and understandable. Encryption has nothing to do with it.
  3. 3. Clustering is usually described as which kind of task?

    Clustering groups similar records without needing a target, so it describes the data.
  4. 4. Why must a mined pattern be checked on new data?

    A pattern is only useful if it is valid beyond the sample it was found in.

Summary

  • Data mining extracts valid, novel, useful and understandable patterns from large data.
  • It overlaps with statistics and machine learning but focuses on the whole pipeline over big, found data.
  • Main task families: associations, classification and prediction, clustering, anomalies, text and web mining.
  • Descriptive tasks summarize; predictive tasks estimate unknown values.
  • Always validate patterns, and remember correlation is not causation.

Next: The KDD Process and CRISP-DM