Skip to content
Tutorials Center
My learning

Topics

Lesson 11 of 39 ยท Exploration and Visualization

Descriptive Statistics and Distributions

Describe a data set with measures of center, spread and shape: mean, median, mode, variance, quantiles, skewness and kurtosis, plus the five-number summary in pandas.

Before you run any mining algorithm, look at your data. Exploratory data analysis (EDA) means summarizing and visualizing each attribute and the relations between them. It finds errors, suggests which transformations and algorithms to use, and often answers half your questions. This lesson covers numeric summaries; the next ones cover plots and relations.

Summaries describe three things: where the values are centered, how spread out they are, and what shape they have.

Measures of center

  • Mean: the sum divided by the count. Uses every value, but one outlier drags it.
  • Median: the middle value of the sorted data. Robust to outliers.
  • Mode: the most frequent value, the only center for nominal data.
  • Trimmed mean: the mean after dropping a fixed share of extremes (say 10 percent each side).
Python Mean versus median

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The mean of 135.7 describes nobody; the median of 51 describes the typical person. When a distribution is skewed, report the median.

Measures of spread

  • Range: max minus min; sensitive to outliers.
  • Quartiles and IQR: Q1 and Q3 are the 25th and 75th percentiles; IQR = Q3 - Q1 is the spread of the middle half.
  • Variance and standard deviation: average squared (then rooted) distance from the mean. Use ddof=1 for a sample.
  • MAD (median absolute deviation): a robust spread.
  • Coefficient of variation: std / mean, to compare spread of attributes with different units.

The five-number summary (min, Q1, median, Q3, max) is what a box plot draws.

Python Five-number summary and spread

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Shape: skewness and kurtosis

Skewness measures asymmetry. A positive value means a long right tail (income, prices), negative a long left tail, near zero a symmetric shape. In a right-skewed distribution, mean > median. Kurtosis measures tail heaviness: high kurtosis means more extreme values than a normal distribution would give.

Python Skewness and kurtosis of three shapes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Common rule of thumb: |skew| < 0.5 is roughly symmetric, 0.5 to 1 moderately skewed, above 1 strongly skewed. If skew is strong, consider a log transform (Data Transformation).

Describing categorical attributes

Use frequency tables, proportions and the mode. A column with one value in 99 percent of rows carries little information; a column with a unique value in every row (an id) is useless for modeling.

Python Frequency table and cardinality

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Summaries by group

Overall numbers hide differences between groups. Group first, then summarize.

Python Aggregates per group

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Percentiles and ranks

A percentile tells where a value stands in the data: the 90th percentile is above 90 percent of values. Percentiles are the basis of box plots, equal-frequency bins and the IQR outlier rule.

Python Percentile and percentile rank

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A checklist for every new data set

  1. df.shape, df.dtypes, df.head().
  2. Missing values per column.
  3. describe() for numbers; value_counts() for categories.
  4. Compare mean and median; check skewness.
  5. Look for impossible values (negative ages) and constant columns.
  6. Repeat per important group.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Salaries are 40, 42, 45, 47 and 400. Which center best describes a typical salary?

    The median (45) ignores the extreme 400; the mean is pulled far up.
  2. 2. In a right-skewed distribution:

    The long right tail pulls the mean above the median; skewness is positive.
  3. 3. The IQR is:

    IQR is the width of the middle 50 percent of the data.
  4. 4. Which statistic compares the spread of height (cm) and weight (kg)?

    std divided by mean is unit-free, so attributes with different units can be compared.

Summary

  • Center: mean, median, mode, trimmed mean. Skewed data prefers the median.
  • Spread: range, IQR, standard deviation, MAD, coefficient of variation.
  • Shape: skewness and kurtosis; log-transform strong right skew.
  • Use frequency tables for categories and always summarize per group too.
  • Start every project with a quick checklist of summaries.

Next: Visualizing Data