Skip to content
Tutorials Center
My learning

Topics

Lesson 44 of 47 ยท Projects and Practice

Project 4: appraise a nutrition study

Practice critical appraisal on a fictional cohort study: calculate relative and absolute risk, confidence intervals and an E-value, and judge how far to trust the claim.

The last project of the course puts the evidence chapter to work. You will take a made-up but realistic summary of a nutrition study, compute the numbers that the abstract leaves out and decide how much weight the result deserves. This is a skill you will use for the rest of your life, whenever a headline tells you that a food "cuts risk by a third". The study here is fictional: its numbers were invented for practice and do not describe real research.

Four steps: identify the question, check the design, compute the numbers, judge the claim
Caption: four questions to ask of any study, which this lesson walks through.

What you will learn

  • How to read a study summary: population, exposure, outcome and design.
  • How to compute relative risk, absolute risk difference, number needed to treat and a confidence interval from counts.
  • What an E-value says about unmeasured confounding.
  • How to write a balanced one-paragraph appraisal.

Key terms

  • Confidence interval (CI): a range of values compatible with the data at a given confidence level, usually 95 percent.
  • Absolute risk difference: risk in the exposed group minus risk in the comparison group.
  • Number needed to treat (NNT): 1 divided by the absolute risk difference: the number of people who need the exposure for one to benefit, assuming the effect is causal.
  • E-value: the minimum strength of association an unmeasured confounder would need with both exposure and outcome to explain away the observed result.
  • Selection bias: bias from how participants were chosen or retained.

The fictional study

Nut intake and cardiovascular events: a prospective cohort. (Fictional, for exercise.) We followed 12,000 adults aged 45-65, free of cardiovascular disease at baseline, for 10 years. Nut intake was measured once at baseline with a food frequency questionnaire. 4,000 participants ate nuts five or more times per week ("high"), and 8,000 ate them less than once a week ("low"). During follow-up there were 160 cardiovascular events in the high group and 480 in the low group. After adjustment for age, sex, smoking, body mass index and physical activity, the hazard ratio was 0.80 (95% CI 0.67 to 0.96). The authors conclude that eating nuts regularly protects against cardiovascular disease.

Step 1: question and population

Who was studied? Adults aged 45-65 without disease. What was the exposure? Nut intake at baseline. What was the outcome? Cardiovascular events over 10 years. What is the claim? That nuts protect. Already you can note that the claim is causal ("protects"), while the design, as you will see next, can strictly show only an association.

Step 2: design and biases

This is a prospective cohort study, positioned in the middle of the hierarchy of evidence. Check for the usual problems:

  • Confounding. People who eat nuts regularly may exercise more, smoke less, eat more vegetables and be wealthier. The authors adjusted for several factors, but residual confounding from unmeasured or poorly measured ones (diet quality, income, education) is likely.
  • Measurement error. Nut intake was recorded once. Diets change over ten years, and questionnaires are imperfect, which usually weakens apparent effects but may distort them unpredictably.
  • Reverse causation. People with undiagnosed illness may have changed their diet. Excluding early events would help.
  • Healthy-user effect. Regular nut eaters are health-conscious, which may be correlated with other protective behaviors.
  • Selection and loss to follow-up. Not reported.
  • Generalizability. Only adults aged 45-65 in one setting.

Step 3: compute the numbers

The abstract gives an adjusted hazard ratio but not the crude figures, the absolute difference or an NNT. Compute them from the counts. The program also finds a 95 percent CI for the crude risk ratio (using the standard log method) and an E-value, a tool introduced by VanderWeele and Ding (2017).

Python Risk ratio, confidence interval, NNT and E-value

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When you run it, you will see that the crude risk ratio is about 0.67, a one-third lower risk, and the absolute difference is 2 percentage points, giving an NNT of 50 over 10 years. That sounds impressive until you ask whether nuts caused it. After adjustment the hazard ratio is weaker (0.80), which shows that confounding by measured factors already explained a share of the difference. The E-value for 0.80 is about 1.8: an unmeasured factor associated with both nut-eating and cardiovascular events by a risk ratio near 1.8 could explain the result entirely. The E-value for the confidence limit nearest to no effect (0.96) is only about 1.25, so a modest hidden confounder could push the lower bound across 1. A diet-quality or income gradient can easily have an association of that size with both, so the result is suggestive, not conclusive.

Step 4: judge and write the appraisal

A good appraisal is balanced. It names the strengths, the limitations and a conclusion about how far to believe the claim and what would raise confidence. Here is one for the fictional study:

This large prospective cohort found that adults who ate nuts at least five times a week had a lower 10-year rate of cardiovascular events than those who ate them rarely (adjusted HR 0.80, 95% CI 0.67-0.96; crude absolute difference 2 percentage points). Strengths include the size, prospective design and adjustment for major risk factors. Limitations include a single baseline measure of diet, likely residual confounding by diet quality and socioeconomic status, no exclusion of early events and a causal conclusion that a cohort design cannot support. An E-value of about 1.8 suggests that a plausible hidden confounder could account for the finding. The result is consistent with other evidence (such as the PREDIMED trial, in which one arm was given nuts), which raises plausibility but does not make the effect size certain. Verdict: moderate confidence that high nut intake is associated with lower risk; limited confidence in the size of any causal effect.

Notice that it does not say the study is useless or the claim is false, and it does not treat the headline as settled. The checklist below helps you appraise any study this way. Tick the items you would require before treating a nutrition claim as strong evidence.

Appraisal checklist scorer

Practice

Apply the method to a headline you find in the news: find the original paper (not just the press release), identify the design, find the baseline risk and the absolute effect and write a five-sentence appraisal. Share it with a friend who has not studied nutrition and see whether they can follow it: communicating uncertainty clearly is the other half of the skill.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. In the fictional study, 160 of 4,000 had events in the high group and 480 of 8,000 in the low group. What is the crude risk ratio?

    4.0 percent divided by 6.0 percent is about 0.67.
  2. 2. What is the number needed to treat if the absolute risk difference is 2 percentage points?

    NNT = 1 / 0.02 = 50.
  3. 3. What does an E-value tell you?

    A larger E-value means the result is more robust to hidden confounding.
  4. 4. Why is 'nuts protect against heart disease' too strong a conclusion from this cohort?

    Causal claims need more than an association with adjustment.
  5. 5. What should a balanced appraisal include?

    Good appraisal is fair: it weighs strengths and weaknesses and states how far to believe.

Summary

  • Read a study by population, exposure, outcome and design.
  • Compute absolute risks, NNT, confidence intervals and an E-value to go beyond the abstract.
  • Write a balanced appraisal that states how much confidence the evidence deserves.

Next: Glossary of key terms