Skip to content
Tutorials Center
My learning

Topics

Lesson 39 of 46 ยท Learning Technology and Projects

Adaptive tutoring and spaced-repetition software

How software personalizes practice: Bloom's two sigma problem, intelligent tutoring systems, Bayesian knowledge tracing, the Leitner box and the SM-2 spacing algorithm.

A good human tutor does three things that a classroom teacher with thirty students cannot. The tutor watches what you know, picks the next task to suit you, and answers each step you take. Software cannot yet match a great tutor, but it can do parts of this for every learner at once. Two families of tools put the previous chapters into practice: adaptive tutoring systems, which track what you know and choose what to practice, and spaced-repetition software, which schedules reviews so that you see each item just before you would forget it.

This lesson explains how both work, runs the key algorithms in your browser and looks at what the evidence shows.

Five boxes labelled box 1 to box 5 with review intervals from every day to every 16 days, a green arrow for right answers and a pink arrow back to box 1 for wrong answers
The Leitner system: a right answer moves a card to a box that is reviewed less often. A wrong answer sends it back to box 1.

What you will learn

  • Bloom's two sigma problem and what the research on tutoring found
  • How an intelligent tutoring system models a learner
  • How Bayesian knowledge tracing updates the probability that you know a skill
  • How the Leitner box and the SM-2 algorithm schedule spaced reviews
  • The weaknesses of these tools

Key terms

Intelligent tutoring system (ITS): software that models a learner's knowledge and gives hints and feedback at the level of individual steps.

Knowledge tracing: estimating whether a learner knows a skill from their answers over time.

Mastery learning: moving on only when a skill has been mastered.

Spaced-repetition system (SRS): software that schedules reviews of items at growing intervals.

Ease factor: in SM-2, a number that controls how fast the review interval for an item grows.

The two sigma problem

In 1984 Benjamin Bloom reported that students who learned a topic with a personal tutor and mastery learning performed about two standard deviations better than students in a conventional class, so that the average tutored student scored higher than about 98 percent of the class. He called the challenge of achieving this effect for everyone, at an affordable cost, the "2 sigma problem". The studies behind the figure were small and short, and later reviews put the effect of human tutoring lower. Kurt VanLehn (2011) reviewed the literature and estimated an effect of about 0.8 standard deviations for human tutoring compared with no tutoring, and about 0.76 for good step-based tutoring systems. Systems that gave feedback only on final answers did less well, about 0.3. Bloom's figure is best seen as an upper bound, but the underlying point stands: individual help with feedback at each step is powerful.

Intelligent tutoring systems

An ITS typically has four parts.

  1. A domain model that breaks a subject (algebra, say) into skills.
  2. A learner model that estimates which skills the student has mastered.
  3. A tutoring module that decides the next problem and what hint to give.
  4. An interface where the learner works, step by step.

The best known family is the Cognitive Tutors developed at Carnegie Mellon University by John Anderson, Kenneth Koedinger and colleagues, which teach mathematics and are used in many schools. A large evaluation by John Pane and colleagues (2014) of a Cognitive Tutor algebra course found no clear effect in the first year, and a modest improvement in the second year of implementation, a reminder that software works through the way teachers and students use it.

Two failure modes have been well documented. Gaming the system: Ryan Baker and colleagues (2004) found that some students click through hints until the answer appears or enter guesses rapidly, and that this behavior predicted lower learning. Wheel spinning: Joseph Beck and Yue Gong (2013) described students who stay stuck on a skill for a very long time without reaching mastery, wasting effort that a human could redirect.

Bayesian knowledge tracing

How can software tell whether you know a skill? You cannot see knowledge, only answers, and answers are noisy: you can guess right without knowing, or slip and get it wrong though you do know. In 1995 Albert Corbett and Anderson proposed Bayesian knowledge tracing (BKT), a model with four numbers per skill.

  • P(L0): the probability that you already knew the skill before starting.
  • P(T): the chance that you learn it from one practice opportunity.
  • P(G): the chance that you answer right by guessing without knowing it.
  • P(S): the chance that you answer wrong by slipping even though you know it.

After each answer, the model first updates its belief using Bayes' rule, then adds the chance of learning from the attempt:

Text
if correct:   P(L | answer) = P(L)(1 - S) / [ P(L)(1 - S) + (1 - P(L)) G ]
if incorrect: P(L | answer) = P(L) S / [ P(L) S + (1 - P(L)) (1 - G) ]
P(L next)     = P(L | answer) + (1 - P(L | answer)) T

When P(L) passes a threshold, often 0.95, the system considers the skill mastered and moves on. The demonstration below runs the model on a sequence of answers. The parameter values are invented for illustration. Real systems estimate them from data.

Python Bayesian knowledge tracing: tracking one skill

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Spaced repetition: Leitner to SM-2

The oldest scheduling method needs no computer. In 1972 the German science journalist Sebastian Leitner described a system of boxes for flash cards. All cards start in box 1. A card you answer correctly moves up to the next box, and a card you miss goes back to box 1. Box 1 is reviewed every day, box 2 every other day, and so on, with each box reviewed less often than the last, as in the picture. Easy cards drift to rarely visited boxes and hard ones return often, so your time is spent where it is needed.

In the late 1980s Piotr Wozniak wrote a computer algorithm that became the foundation of many programs, known as SM-2. It gives each item a numeric ease factor (EF, starting at 2.5) and a number of successful repetitions (n). After each review you rate your recall from 0 (blackout) to 5 (perfect). Then:

  • If the rating is 3 or more: the next interval is 1 day after the first success, 6 days after the second, and after that the previous interval multiplied by EF and rounded. Update EF = EF + (0.1 - (5 - q)(0.08 + (5 - q) 0.02)), but never below 1.3.
  • If the rating is below 3: reset n to 0, so the next interval is 1 day, and leave EF unchanged.

Anki, a popular free flash-card program, uses a modified version of this scheme. Newer algorithms, such as the FSRS scheduler that recent Anki versions offer as an option, model memory more directly and tune the intervals to each user. The demonstration below runs exact SM-2. Rate each review as you would on a real card.

SM-2 spaced-repetition scheduler

What does the evidence say?

  • Spacing and retrieval: SRS tools implement the two highest-rated techniques in the previous chapter, which is a strong theoretical reason to expect them to work. Laboratory studies of flash-card learning support this.
  • Real-world trials: Evidence from long-term use is thinner. Studies of medical students using spaced-repetition flash cards have reported better exam results for users, but those studies are mostly observational, which means that motivated students may be the ones using the tool.
  • Adaptive tutors: Reviews of step-based ITS find moderate average effects, with large differences between systems, subjects and implementations.

Limits and pitfalls

  • Garbage in, garbage out. SRS only helps you remember what is on your cards. Poorly made cards (vague, with multiple facts) waste time. Understanding must come first, since spaced repetition strengthens memory and does not create comprehension.
  • Motivation and habit. Reviews pile up if you skip a few days. Plan a small daily quota.
  • Narrow skills. Mastery models work well for well-defined skills (arithmetic, vocabulary, formulas) and less well for open-ended skills like writing or design.
  • Gaming and wheel spinning. Without human oversight, some learners misuse hints or get stuck.
  • Self-rating bias. In SM-2 the learner chooses the rating, so the illusions of knowing from the last chapter still apply. Test yourself honestly before looking at the answer.

Common misconceptions

"Adaptive means the software adapts to my learning style." Adaptive tutors adapt to what you know and do, not to a style.

"Spaced repetition makes me understand." It helps you remember what you already understand.

"More cards always means more learning." Too many poorly written cards lead to overload and abandonment.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is Bloom's two sigma problem?

    Later reviews suggest that human tutoring is somewhat less powerful than Bloom's figure.
  2. 2. In Bayesian knowledge tracing, what does the slip parameter P(S) represent?

    A known skill can still be answered wrongly because of a careless error.
  3. 3. In the Leitner system, what happens to a card you answer wrongly?

    Missed cards return to the most frequently reviewed box.
  4. 4. In SM-2, what happens when you rate a review below 3?

    The item restarts, and the ease factor is left unchanged.
  5. 5. What is 'gaming the system' in tutoring software?

    Baker and colleagues found that this behavior predicted lower learning.

Summary

  • Bloom's 2 sigma finding is an upper bound. Step-based tutoring has substantial, if smaller, effects.
  • Intelligent tutoring systems model skills and the learner, often with Bayesian knowledge tracing, and choose what to practice.
  • The Leitner box and SM-2 implement spacing and retrieval in software. They help you remember what you understand.
  • Weaknesses: poor cards, habit problems, self-rating bias, gaming and wheel spinning.
  • Use these tools as part of a plan that also includes explanation and application.

Next: AI and learning