Skip to content
Tutorials Center
My learning

Topics

Lesson 21 of 38 ยท Statistics

Correlation, Regression and Bootstrap

Measure relationships with pearsonr and spearmanr, fit lines with linregress, build confidence intervals with the bootstrap, run Monte Carlo simulations and use gaussian_kde.

Two questions come up constantly once you have data: how are these variables related? and how sure am I about this number? The first is answered by correlation and regression. The second by a confidence interval, and when no neat formula exists, by the bootstrap, a technique that lets the data estimate its own uncertainty by resampling.

This lesson covers pearsonr, spearmanr, linregress, stats.bootstrap, Monte Carlo simulation, and smooth density estimates with gaussian_kde. It follows Hypothesis Testing and uses the distribution objects from Distributions and Descriptive Statistics.

flowchart LR
    D["original sample<br/>(n values)"] --> R1["resample with replacement"]
    D --> R2["resample with replacement"]
    D --> R3["... thousands of times"]
    R1 --> S1["statistic 1"]
    R2 --> S2["statistic 2"]
    R3 --> S3["statistic k"]
    S1 --> H["distribution of the statistic"]
    S2 --> H
    S3 --> H
    H --> CI["confidence interval<br/>= middle 95%"]

The bootstrap treats the sample as a stand-in for the population and measures how much the statistic wobbles between resamples.

Correlation: how strongly do two variables move together?

The Pearson correlation r measures the strength of a linear relationship, from -1 (perfect downhill line) through 0 (no linear relationship) to +1 (perfect uphill line). Spearman's coefficient applies the same idea to the ranks of the values, so it detects any monotonic relationship (always rising or always falling, even if curved) and is not thrown off by outliers. kendalltau is a third rank-based option that behaves well on small samples with ties.

Both functions return the coefficient and a p-value for the null hypothesis "no correlation".

Python Pearson versus Spearman

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Pearson stays high for the curved data but is lower than Spearman, and a single outlier wrecks it while Spearman barely moves. Always draw a scatter plot first: very different datasets can have the same r.

Warning: Correlation is not causation. Ice cream sales and sunburn rise together because both follow the weather. A correlation also says nothing about slope: r = 0.9 does not mean y changes by 0.9 per unit of x.

Linear regression with linregress

Correlation says how strong; regression says how much: the straight line y = intercept + slope * x that fits the data best in the least-squares sense. For one predictor, stats.linregress(x, y) returns everything you need in one object: slope, intercept, rvalue, pvalue (a test of "slope is zero"), and the standard errors stderr and intercept_stderr.

Python Fitting a line and drawing it

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

R squared is the share of the variance in y explained by the line. The right panel is the habit that saves you: plot the residuals (data minus fit). They should look like shapeless noise around zero. A curve, a funnel (the spread growing with x) or a cluster of outliers means a straight line is the wrong model. For several predictors or non-linear shapes, move on to Curve Fitting and Least Squares and Least Squares and Pseudo-inverses.

Confidence intervals

A sample mean is only an estimate. A 95% confidence interval is a range computed by a procedure that, repeated over many samples, would contain the true value in 95% of them. For a mean the classic recipe uses the t distribution: mean ยฑ t * standard error, which stats.t.interval computes directly.

Python A confidence interval for a mean

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The coverage experiment is the honest meaning of "95%": it describes the method, not any single interval.

The bootstrap

The t interval works for means of roughly normal data. What about the median, a correlation, the ratio of two means, or a trimmed mean? Formulas are complicated or do not exist. The bootstrap replaces the algebra with computation: draw many samples with replacement from your data, each the same size as the original, compute the statistic on each, and read the spread of those values.

stats.bootstrap does the whole job. You pass the data as a tuple of samples, a statistic that accepts an axis argument, and options.

Python Bootstrap interval for a median

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Three practical notes. method="BCa" is the default and usually the most accurate; "percentile" simply takes the middle 95% of the resampled statistics. With very small samples (under about 20) the bootstrap itself is unreliable. And for two-sample or paired statistics pass paired=True or a tuple of two samples, so that the resampling matches the study design.

Monte Carlo simulation

The bootstrap is one case of a bigger idea: when a probability or an expectation is hard to derive, simulate the process many times and count. Accuracy improves with the square root of the number of runs, so 100 times more samples buys one extra digit.

Python Monte Carlo: probabilities by simulation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Example 2 has a closed form because sums of independent normals are normal, which lets us check the simulation. Replace the normals with skewed distributions or add dependencies and the formula disappears, but the simulation barely changes.

Smooth densities with gaussian_kde

A histogram depends on where you put the bins. A kernel density estimate (KDE) places a small bell curve on every observation and adds them up, giving a smooth estimate of the underlying density. stats.gaussian_kde(data) returns a callable: call it with points to get density values, or use .resample() to draw new values and .integrate_box_1d(a, b) for probabilities. The key setting is the bandwidth (bw_method): too small gives a spiky curve, too large washes out real structure.

Python A density estimate of two overlapping groups

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The default "scott" rule is a sensible start. The tiny bandwidth produces a jagged curve that follows noise, the large one merges the two groups into one hump. Look at several before trusting a shape.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Your data rises steadily but along a curve, and one measurement is a wild outlier. Which coefficient is the better summary?

    Spearman is based on ranks, so it captures any monotonic relationship and one extreme value hardly changes it. Pearson only measures linear association and is sensitive to outliers.
  2. 2. What does linregress(x, y).pvalue test?

    The p-value is for the null hypothesis that the true slope is zero (no linear relationship). Intercept and slope have separate standard errors.
  3. 3. In a bootstrap, how is each resample drawn?

    A bootstrap resample draws n values from the n observations with replacement, so some points repeat and others are missing.
  4. 4. What is the meaning of a 95% confidence interval?

    The 95% describes the long-run behaviour of the method. Any one computed interval either contains the true value or does not.
  5. 5. What happens with a very small bw_method in gaussian_kde?

    Bandwidth controls the width of the bell placed on each point. A tiny bandwidth gives a jagged estimate, a large one oversmooths.

Summary

  • pearsonr measures linear association, spearmanr and kendalltau measure monotonic association and resist outliers; always plot the data and never read causation into a correlation.
  • linregress fits a straight line and returns slope, intercept, rvalue, p-value and standard errors; check the residual plot.
  • A confidence interval describes a method's long-run coverage; for means use stats.t.interval.
  • stats.bootstrap gives intervals for almost any statistic by resampling with replacement; BCa is a good default.
  • Monte Carlo simulation estimates probabilities by repeating a random process; error shrinks with the square root of the run count.
  • gaussian_kde produces smooth density estimates; the bandwidth controls the smoothness.

Next up: switch from data about the world to data over time with The Fast Fourier Transform.