Skip to content
Tutorials Center
My learning

Topics

Lesson 23 of 39 ยท Classification

Rule-Based and Associative Classification

Learn IF-THEN rule classifiers: coverage and accuracy, sequential covering (1R, RIPPER-style), and associative classification (CBA) that mines class association rules, with Python code.

A rule-based classifier is a set of IF-THEN rules such as IF age < 25 AND student = yes THEN buys = yes. The part after IF is the antecedent (the condition) and the part after THEN is the consequent (the class). Rules are among the most understandable models: a person can read, check and edit them. You already saw that a decision tree can be turned into rules, one per leaf. This lesson covers rules learned directly from the data and associative classification, which borrows the machinery of association rule mining.

Measuring a rule

Two numbers describe every rule R:

  • Coverage: the fraction of all records that satisfy the antecedent.
  • Accuracy (confidence): among the records covered, the fraction that really have the predicted class.

A rule with high accuracy but tiny coverage may be a coincidence; a rule with huge coverage but modest accuracy is too general. When several rules fire for the same record, a conflict resolution strategy decides: use the rule with the highest accuracy, the most specific antecedent, or the order in the list (a decision list: the first matching rule wins). If no rule fires, a default rule gives the majority class.

Python Coverage and accuracy of a rule

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Sequential covering

Instead of building a whole tree, sequential covering learns one rule at a time:

  1. Learn one good rule for the target class (grow its antecedent condition by condition, keeping the most accurate version).
  2. Remove the records it covers.
  3. Repeat until no good rule remains or all records are covered.

Famous algorithms using this scheme are AQ, CN2, FOIL and RIPPER (which adds pruning). The first rule is learned for one class, then the next class, and the last class becomes the default.

Python A tiny sequential covering learner

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The loop grows a rule greedily (add the condition that makes the covered set most "yes"), removes the covered rows, and continues. Real systems measure rule quality with information gain and prune to avoid overfitting.

Association rules for classification

Associative classification (also called classification by association rules) mines class association rules (CARs): association rules whose consequent is always a class label. Because association mining examines many combinations of attributes at once, it can find rules that greedy tree or rule learners miss. Well-known systems are CBA (Classification Based on Associations), CMAR and CPAR.

The workflow:

  1. Turn every record into a set of items such as age=young, student=yes plus a class item buys=yes.
  2. Mine frequent itemsets with a minimum support, then keep rules conditions -> class above a minimum confidence.
  3. Sort the rules by confidence, then support, then shorter length first.
  4. Select a small, accurate rule set (CBA keeps rules that correctly classify at least one training record) and add a default class.
  5. Classify a new record with the first matching rule (CBA) or by a weighted vote of the matching rules (CMAR).
Python Class association rules from scratch

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The sorted rule list is a decision list. Sorting by confidence first puts the most reliable rules at the top.

Advantages and limits

Approach Strength Weakness
Rules from a tree Easy, mutually exclusive rules Inherits tree instability
Sequential covering Compact rule sets, handles imbalance Greedy, sensitive to noise
Associative classification Often more accurate, finds attribute combinations Can create huge rule sets; needs support/confidence tuning

All three need numeric attributes to be discretized into bins first, or the rule learner must search for thresholds as in the tree builder.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In a rule IF A AND B THEN C, what is the antecedent?

    The antecedent is the condition part; the consequent is the predicted class.
  2. 2. What does rule coverage measure?

    Accuracy is the share of covered records with the right class; coverage is how many records the rule applies to.
  3. 3. What is the key step of sequential covering?

    Rules are learned one at a time and the covered records are removed before learning the next rule.
  4. 4. What makes a rule a class association rule?

    Class association rules are association rules restricted to a class label on the right-hand side.

Summary

  • Rule classifiers use IF-THEN rules judged by coverage and accuracy, with a default rule for uncovered records.
  • Sequential covering learns rules one by one (AQ, CN2, FOIL, RIPPER).
  • Associative classification (CBA, CMAR, CPAR) mines class association rules, ranks them and classifies with the best matching rule.
  • Rules are easy to read but need discretized numeric attributes and careful thresholds.

Next: Ensembles and Support Vector Machines