Skip to content
Tutorials Center
My learning

Topics

Lesson 4 of 39 ยท Getting Started

Data Types and Data Quality

Learn attribute types (nominal, ordinal, interval, ratio), kinds of data sets, metadata, and how to measure data quality: accuracy, completeness, consistency and more.

An algorithm only sees numbers and labels, but it is your job to know what they mean. Is 3 a count of visits, a rating, or a category code? Can you average it? A wrong guess leads to confident nonsense. This lesson covers the types of attributes, the shapes of data sets you will meet, the idea of metadata, and the dimensions of data quality. The next chapter then shows how to repair quality problems.

Records and attributes

Data mining usually works on a table. Each row is an object (also called record, instance, sample, tuple). Each column is an attribute (also called feature, variable, field). The values in one column share a type, and the type decides which operations make sense.

Attribute types

There are four classic levels, from weakest to strongest:

Type What it tells you Example Allowed operations
Nominal Names only color, city, product code equal or not equal, mode
Ordinal Order, but gaps unknown small/medium/large, 1-5 stars order, median
Interval Order and equal gaps, no true zero temperature in Celsius, year differences, mean
Ratio Order, gaps and a true zero age, price, weight, count ratios, all statistics

Two other splits matter in practice. Discrete attributes take countable values (number of children); continuous ones take any value in a range (height). A binary attribute has two values; if only one of them (say "purchased") is interesting, it is called asymmetric binary, a point that matters for similarity in a later lesson.

Python Checking types with pandas

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A common mistake is to store a nominal code as a number (city_id = 3) and then average it. The mean of city ids means nothing. Always ask: what does an arithmetic operation on this column mean?

Kinds of data sets

Besides plain tables, mining meets other shapes:

  • Record data: tables, transaction baskets (each row has a variable-size set of items), data matrices.
  • Ordered data: time series, sequences of events, genome strings, spatial data.
  • Graph data: social networks, the web, molecules.
  • Text and multimedia: documents, images, audio, video, usually converted to numbers first.

Properties that influence the method: dimensionality (number of attributes, the "curse of dimensionality" makes distances less meaningful), sparsity (mostly zeros, as in baskets and text), resolution (the scale at which data was collected) and size.

Python Sparsity of a basket matrix

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Metadata: data about data

Metadata describes a data set: column names, types, units, allowed ranges, meaning, source, time of collection, owner, and how fields were derived. Without it, rev could mean revenue or revision, in dollars or thousands. Good mining projects keep a data dictionary, and warehouses keep metadata centrally (see Data Warehouses and OLAP).

Python A tiny data dictionary used as a validator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Data quality

Mining cannot repair data that is wrong in ways nobody noticed. Quality is usually judged along several dimensions:

  • Accuracy: values are correct (no typos, wrong units, sensor errors).
  • Completeness: the needed values and records are present.
  • Consistency: the same fact agrees everywhere (a customer's country in two tables).
  • Timeliness: the data is recent enough for the decision.
  • Validity: values follow the allowed format and range.
  • Uniqueness: no unwanted duplicates.
  • Believability and interpretability: users trust the source and understand the fields.

Typical defects are noise (random error), outliers, missing values, duplicates, inconsistent coding (M, Male, male) and bias (the data does not represent the population you care about).

A quick quality report

Python Measure quality in a few lines

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Turning such a report into repairs is the subject of Data Cleaning. Run a report like this on every new data set before modeling: it costs minutes and often saves days.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which attribute type has both order and a true zero, so ratios make sense?

    Ratio attributes like age, weight and price have a meaningful zero, so '2 times as much' is valid.
  2. 2. Shirt sizes S, M, L are best described as:

    They have a natural order but the gaps between sizes are not defined.
  3. 3. 'M', 'male' and 'Male' in one column are a problem of:

    The same fact is coded in several ways, which breaks grouping and counting.
  4. 4. What is metadata?

    Metadata documents names, types, units, source and meaning of the data.
  5. 5. Why is averaging a numeric city code wrong?

    Nominal values are only names; arithmetic on the codes produces meaningless numbers.

Summary

  • Rows are objects, columns are attributes; types are nominal, ordinal, interval and ratio.
  • The type decides which statistics and distance measures are valid.
  • Data sets can be tables, baskets, time series, graphs, text or media; watch dimensionality and sparsity.
  • Metadata and a data dictionary say what each column means and which values are valid.
  • Quality has many dimensions: accuracy, completeness, consistency, timeliness, validity, uniqueness.
  • Measure quality before you model.

Next: Data Warehouses and OLAP