Skip to content
Tutorials Center
My learning

Topics

Lesson 6 of 76 ยท What Data Science Is

Types of Data and Measurement

Learn the levels of measurement, numeric versus categorical data, time series, text and nested data, units, granularity and tidy tables, with pandas dtypes in action.

Before you can summarize, plot or model a column, you need to know what kind of thing it is. The average of a column of heights makes sense. The average of a column of zip codes does not, even though both are made of digits. Picking the wrong operation for a data type is one of the most common beginner errors, and it often produces results that look fine but mean nothing.

This lesson gives you a vocabulary for the kinds of data you will meet, shows how they map to pandas dtypes, and introduces two ideas that make later work easier: granularity and tidy data.

Levels of measurement

A classic scheme, due to the psychologist Stanley Smith Stevens, sorts variables by what you can meaningfully do with their values.

Level What it tells you Example Meaningful operations
Nominal Name or label only Country, payment method Count, mode, equality
Ordinal Order, but gaps unknown Satisfaction (low, medium, high) Rank, median, comparison
Interval Order and equal gaps, no true zero Temperature in Celsius, calendar year Differences, mean
Ratio Order, equal gaps and a true zero Price, age, number of visits Differences, ratios, mean

The test for "true zero" is whether "twice as much" makes sense. 20 dollars is twice 10 dollars, but 20 degrees Celsius is not "twice as hot" as 10 degrees, because zero degrees is not an absence of heat.

flowchart TD
    A["A variable"] --> B{"Are values just labels?"}
    B -->|"yes"| N["Nominal"]
    B -->|"no"| C{"Is there a natural order?"}
    C -->|"order only"| O["Ordinal"]
    C -->|"equal steps"| D{"Is zero a true absence?"}
    D -->|"no"| I["Interval"]
    D -->|"yes"| R["Ratio"]

A quick decision path for the level of measurement.

Why bother? Because the level tells you which summaries are honest. You can report the most common payment method (nominal) but not its average. You can report the median satisfaction (ordinal) but be careful with the mean, because the distance between "low" and "medium" may not equal the distance between "medium" and "high".

Numeric versus categorical

In practice most people sort columns into two big groups:

  • Numeric: counts and measurements. It splits into discrete (whole counts like the number of orders) and continuous (can take any value in a range, like weight).
  • Categorical: a limited set of labels, either unordered (nominal) or ordered (ordinal).

Two traps deserve a warning:

  1. Numbers that are labels. Zip codes, phone numbers, product IDs and ratings coded 1 to 5 are stored as numbers but usually should not be averaged. A 1-to-5 rating is ordinal; whether you may average it is a judgment call you should state.
  2. Booleans and flags. A yes/no column is categorical, but a 0/1 version has a useful property: its mean is the share of "yes" values.

Seeing types in pandas

pandas stores each column with a dtype, and you can use it as a first clue. Remember that the dtype describes the storage, not the meaning: an integer column may still be a nominal ID.

Python Dtypes tell you part of the story

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at what pandas decided and ask whether it matches the meaning. order_id is an integer, but it is a label. rating is an integer, but it is ordinal. size is text, but it has an order (S, M, L). The dtype is a starting point, not the answer.

Giving columns the right type

Tell pandas what you know. A category dtype can even record the order of an ordinal variable, which makes sorting and comparisons correct:

Python Classifying columns and fixing their types

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that comparison df["size"] > "S" only works because the category is ordered. With plain text, "L" would sort before "M" alphabetically.

Time series, text and nested data

Not all data is a table of simple values. Three other shapes appear constantly:

  • Time series: values recorded over time, such as daily sales. Order matters, neighbors are related, and you must be careful about gaps and time zones. See Trends, Seasonality and Time Patterns.
  • Text: reviews, support tickets, free-form survey answers. It must be converted to numbers (for example word counts) before most models can use it. See Working with Text Data.
  • Nested data: JSON from web services, where a record can contain lists and sub-records. It is usually flattened into columns first; Loading Data from CSV, JSON and SQL shows how, and the pandas lesson on JSON and nested data goes deeper.

Images, audio and graphs are other families. They are mostly beyond this course, but the same questions apply: what is a single observation, and what do we know about it?

Structured, semi-structured and unstructured data

Levels of measurement describe what a column means. A different question is how the whole dataset is organised, because that decides what you must do before you can analyse it. Data comes in three broad forms.

Form Shape Examples Typical storage
Structured Fixed rows and columns, every column has a type Orders table, sensor readings, spreadsheet SQL database, CSV, Parquet
Semi-structured Self-describing tags or keys, but records can differ and nest JSON from an API, XML, log lines with key=value pairs JSON files, document stores such as MongoDB
Unstructured No predefined fields Free text, images, audio, video, PDFs Object storage, file systems

Real projects mix all three. An online shop keeps orders in tables (structured), receives product details as JSON from suppliers (semi-structured) and collects customer reviews as text and photos (unstructured).

flowchart LR
    A["Unstructured<br/>text, images, audio"] -->|"extract features"| D["Structured table<br/>rows and typed columns"]
    B["Semi-structured<br/>JSON, XML, logs"] -->|"flatten and parse"| D
    C["Structured<br/>tables, CSV"] --> D
    D --> E["Analysis and models"]

Almost all analysis starts by turning the other forms into a table.

Handling structured data

This is the easy case: load it, check types, and go. The work is quality, not shape; see Loading Data and the Data Quality Audit.

Handling semi-structured data

The challenge is that records disagree. Some have a field, others do not, and values can be nested lists or objects. The standard approach is to flatten the nesting into columns and decide how to treat missing keys. In pandas, pd.json_normalize does the flattening and explode turns list fields into one row per item.

Python Flattening nested JSON into a table

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice the choices hiding in the code. json_normalize with record_path drops order 3 because it has no items, while explode keeps it with a missing value. Neither is wrong, but you must pick one on purpose and check the row counts afterwards.

Handling unstructured data

Text, images and audio have no columns, so you must create them. The step is called feature extraction. For text it can be as simple as counting words or as advanced as embeddings from a language model; for images it can be size and average colour or the output of a neural network. The result is a table with one row per document or image, which the rest of the toolkit understands. Text Data Basics goes deeper.

Python From free text to a feature table

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The hand-made features above are crude on purpose. The lesson is the conversion: once unstructured data becomes columns, every table technique applies. Later lessons show better text and image features.

Choosing a strategy

  • Is there a schema you can rely on? If yes, structured tools are enough. If not, write down the keys you expect and test for surprises (missing keys, new keys, changed types).
  • Does the structure repeat or vary? Repeating nesting flattens cleanly; highly variable records may stay in a document store and only the needed fields get extracted.
  • How much information do you lose? Every flattening or feature step discards detail. Keep the raw data, so you can extract again in a better way later.
  • Scale: big unstructured collections (millions of images or documents) need dedicated tools and storage, not a single DataFrame.

Units and granularity

Two details cause real trouble:

Units. A column named weight could be kilograms or pounds. A price could include or exclude tax and be in different currencies. Put the unit in the column name (weight_kg) or in a data dictionary, and never mix units in one column.

Granularity (also called grain) is what one row represents. Is a row an order, an item in an order, a customer, or a customer-day? Mixing grains causes double counting. If you join an orders table (one row per order) to an items table (one row per item), each order's total is repeated once per item, and summing it inflates revenue.

Python Mixed granularity double counts

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The true revenue is 70, but the joined table reports 170, because order 1 now appears three times. Before any aggregation, state the grain of each table in one sentence ("one row per order") and check that it still holds after a join.

Tidy data

A very practical convention is tidy data: each variable is a column, each observation is a row, and each type of observational unit gets its own table. Many tools, including pandas group-by and plotting libraries, work most naturally on tidy tables.

A "wide" table with one column per month (jan, feb, mar) stores the variable "month" in the headers. A tidy "long" version has month and sales columns. Reshaping between them is covered in the pandas lesson on reshaping.

Tidy does not mean "always long". Wide tables are fine for display and for models that expect one column per feature. The point is to know which form you have and to convert on purpose.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Temperature in degrees Celsius is which level of measurement?

    Celsius has equal steps but zero is not an absence of heat, so ratios like "twice as hot" are meaningless. That makes it interval.
  2. 2. Customer zip codes are stored as integers. What is the best treatment?

    A zip code is a label. Arithmetic on it is meaningless, and storing it as text also keeps leading zeros.
  3. 3. Why does joining an orders table to an items table and then summing order_total give a wrong revenue?

    The join changes the grain from one row per order to one row per item. Order-level values are repeated and double counted when summed.
  4. 4. Which is the correct pairing?

    Satisfaction has a natural order with unknown gaps, so it is ordinal. Sizes are also ordinal, payment method is nominal, and calendar year is interval.

Summary

  • Levels of measurement (nominal, ordinal, interval, ratio) decide which summaries are meaningful.
  • Numeric columns can be discrete or continuous; labels stored as numbers should be treated as categories.
  • A pandas dtype is only a clue; set the types that match the real meaning, for example ordered categoricals.
  • Record units, and know the granularity (what one row means) of every table to avoid double counting.
  • Tidy data puts each variable in a column and each observation in a row.

Next: Where Data Comes From, the places data is born and what to ask about each.