Skip to content
Tutorials Center
My learning

Topics

Lesson 2 of 36 ยท Getting Started

Bayesian vs Frequentist Thinking

Compare Bayesian and frequentist statistics: what is random, how confidence intervals differ from credible intervals, and when each approach is the better tool.

Two schools of statistics answer the question "what does the data say?" in different ways. Frequentists treat the parameter as a fixed unknown number and ask how the data would vary over many imagined repetitions. Bayesians treat the data as fixed (it is what you actually saw) and describe uncertainty about the parameter with a probability distribution.

Neither is "wrong". They are tools with different promises, and a working analyst should be able to read both. This lesson compares them with one small example and a simulation.

What is random?

Question Frequentist Bayesian
The parameter ฮธ Fixed, unknown Uncertain, so described by a distribution
The data Random (could have come out differently) Fixed once observed
Probability means Long-run frequency Degree of belief
Prior knowledge Not part of the formal method Enters through the prior
Main output Estimate, p-value, confidence interval Posterior distribution

Same data, two answers

You test a new button color. Of 40 visitors who saw it, 9 clicked. Estimate the click rate.

The frequentist point estimate is 9/40 = 0.225 with a 95% confidence interval from a normal approximation. The Bayesian answer, with a flat prior (a Beta(1, 1) distribution, explained in the Beta-Binomial lesson), is a posterior Beta(10, 32) with a 95% credible interval.

Python Confidence interval vs credible interval

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two intervals overlap heavily (the Bayesian one is shifted up a little by the flat prior), but their meanings differ, and only the Bayesian result lets you print the last line: the probability that the click rate exceeds 15%.

Interpreting the intervals

A 95% confidence interval is a statement about the procedure: if you repeated the experiment many times and built an interval each time, about 95% of those intervals would contain the true value. For your one interval, the true value is either inside it or not; "95%" describes the method.

A 95% credible interval is a statement about the parameter: given the model, prior and data, there is a 95% probability that ฮธ lies in this interval. That is the sentence people usually wish confidence intervals said.

Checking the frequentist promise by simulation

Frequentist coverage can be verified directly. We repeat the click experiment many times with a known true rate and count how often the interval contains it.

Python Coverage of a confidence interval

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The coverage is about 91%, clearly below 95%, for a small sample and a small rate. The procedure's promise holds only approximately, which is why statisticians offer better intervals (Wilson, exact). Bayesian intervals have their own guarantees: they are calibrated against the prior you stated, a point we revisit in model checking (a later chapter).

P-values and the Bayesian alternative

A p-value is the probability of data at least as extreme as yours if the null hypothesis were true. It is not the probability that the null is true. To get that probability you need a prior, and then Bayes' theorem gives it directly. Bayesians often report the posterior probability of a hypothesis or of an effect being larger than a practical threshold instead of rejecting a null.

When each approach shines

  • Frequentist methods are fast, standard, and well suited to large samples, regulated settings with fixed protocols, and simple questions with no prior information.
  • Bayesian methods shine with small data, with real prior information, with hierarchical or multi-level structure, with decisions under uncertainty, and when you want probabilities of hypotheses.
  • With lots of data and a weak prior, both usually agree numerically. The disagreement matters most when data is scarce.

A common and healthy attitude is pragmatic: use whichever answers the actual question, and check results with both when stakes are high.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In the Bayesian view, which of these is treated as random (uncertain)?

    Bayesians describe uncertainty about the parameter with a probability distribution; the observed data is fixed.
  2. 2. What does a 95% credible interval mean?

    A credible interval is a direct probability statement about the parameter.
  3. 3. What does a p-value tell you?

    A p-value is computed assuming the null is true; it is not the probability that the null is true.
  4. 4. When will Bayesian and frequentist answers usually be very close?

    Large data overwhelms a weak prior, so the posterior is driven by the likelihood, as in frequentist estimation.

Summary

  • Frequentists: parameters fixed, data random, probability is long-run frequency.
  • Bayesians: data fixed, parameters uncertain, probability is belief.
  • Confidence intervals describe a procedure; credible intervals describe the parameter.
  • A p-value is not the probability that a hypothesis is true.
  • With ample data and weak priors the two usually agree; with scarce data the prior matters.

Next: A Probability Refresher