Skip to content
Tutorials Center
My learning

Topics

Lesson 38 of 63 ยท Sequences and Structure

Sequence Labeling with CRFs and Features

Build a feature-based sequence labeler with scikit-learn: token features, context windows, BIO decoding constraints, and how CRFs add label-to-label structure.

Part-of-speech tagging and named entity recognition share a shape: one label per token, and each label depends on the word, its neighbors and the labels around it. This family is called sequence labeling. Before neural networks took over, the winning recipe was to describe every token with hand-designed features (its suffix, its capitalization, the previous word) and to feed them to a classifier or a conditional random field (CRF). That recipe is still worth learning: it is fast, needs little data, works as a strong baseline, and its ideas, context and structure, live on inside neural taggers.

In this lesson we turn tokens into feature dictionaries, train a plain logistic regression on them with scikit-learn, watch which kinds of features matter, and then see the one weakness of treating each token independently. That weakness is exactly what a CRF fixes.

From tokens to feature dictionaries

A classifier cannot read a word, but it can read a dictionary of named features, which DictVectorizer converts into a sparse matrix (see bag of words). What features help? Think about what a human uses to spot a name in a sentence you have never seen:

Clues inside the word

The word itself Lowercased, for known words.
Capitalization and shape Xxxxxx for Lisbon, dd for 42.
Prefix and suffix -ing, -ly, -ova.

These help even for words never seen in training.

1 / 2

A context window includes features of the neighbors, usually one or two words each side. The shape feature maps every uppercase letter to X, every lowercase letter to x and every digit to d, so Lisbon and Hanoi both become Xxxxxx or Xxxxx. Unseen words still get meaningful features this way.

Python Features for one token

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The feature dictionary for Ines says: capitalized, shape Xxxx, comes right after yesterday, followed by flew. For Tbilisi, the neighbors are to and .. A classifier learns that prev=to points to a place and prev=yesterday to a person or an organization, without ever having to know the word Tbilisi.

Which features matter? An experiment

To measure this, we need data where we control what the model has seen. The code generates sentences from six templates and fills the slots with invented person, organization and place names. The test names come from a different list than the train names, so every entity in the test set is an unseen word. This mimics the real situation: new names appear all the time. We train three versions of the same logistic regression with different feature sets and score them with the F1 over the entity tags (B- and I- tags; O is excluded so that the easy majority does not inflate the score).

Python Three feature sets, one classifier

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The numbers tell a clear story:

Feature set Features Token accuracy Entity-tag F1
word identity only 58 0.648 0.000
+ shape, capitalization, suffix 133 0.756 0.324
+ context window 303 0.921 0.829

With word identity alone, the classifier has memorized the training names and knows nothing about the test names, so it gets no entity tag right (F1 of exactly 0.000, yet 64.8% accurate on tokens, a reminder of why accuracy misleads). Shape features let it notice capitalized words. But it takes the context to decide which kind of entity: in before a capitalized word suggests a place, the before one suggests an organization. Feature engineering is the main lever for this kind of model.

The independence problem

The logistic regression predicts every token on its own. It never asks whether its own predictions fit together, and BIO tags have strict rules: I-ORG can only follow B-ORG or I-ORG. Independent predictions can break those rules. The next experiment counts how often that happens, and then repairs it with constrained decoding: instead of taking each token's best tag, find the best whole sequence of tags that obeys the rules, using the classifier's probabilities as scores. This is the Viterbi idea from part-of-speech tagging again, with forbidden transitions given a score of minus infinity.

Python Independent vs constrained decoding

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The independent tagger produces 34 invalid transitions on the 100 test sentences. In the final sentence, which uses names the model has never seen, it correctly labels Zephyr as B-ORG and then calls Group an I-PER, an impossible continuation (an organization cannot turn into a person mid-name). The constrained decoder removes every invalid transition and raises F1 from 0.829 to 0.848. But look at its answer for Zephyr Group: it fixes the broken sequence by splitting it into two organizations, B-ORG B-ORG, because the rules allow it and the probabilities said Group is not part of a person. A better answer, B-ORG I-ORG, requires knowing from training data that I-ORG often follows B-ORG. A hard rule does not know that; a model that learns it does.

Conditional random fields

A linear-chain CRF is the model that learns label-to-label preferences jointly with the token features. It has two kinds of weights: one for each (feature, label) pair, like our logistic regression, and one for each (previous label, label) pair, the transition weights, such as a high weight for B-ORG followed by I-ORG and a very negative weight for B-ORG followed by I-PER. The model scores a whole label sequence at once, the probabilities are normalized over all possible sequences, and prediction is the same Viterbi search we just wrote. Training learns every weight so that correct sequences score highest.

Logistic regression vs CRF

Per-token classifier Each tag is chosen alone.
CRF The whole tag sequence is scored.
Transitions Learned weights between neighboring tags.

Same features, structured output.

1 / 2

Compared with an HMM, a CRF can use arbitrary overlapping features of the whole input (suffixes, neighbors, shape) instead of only the current word, which is why CRFs replaced HMMs for taggers. The usual library is sklearn-crfsuite, which accepts the same feature dictionaries we built. It is not available in the browser, so the code is read-only:

Python
import sklearn_crfsuite

# X_train: for each sentence, a list of feature dicts (one per token)
# y_train: for each sentence, a list of tags
X_train = [[features(tokens, i) for i in range(len(tokens))] for tokens, _ in train]
y_train = [tags for _, tags in train]

crf = sklearn_crfsuite.CRF(algorithm="lbfgs", c1=0.1, c2=0.1, max_iterations=100,
                           all_possible_transitions=True)
crf.fit(X_train, y_train)

pred = crf.predict([[features(tokens, i) for i in range(len(tokens))]])
print(crf.transition_features_)  # learned label-to-label weights

Notice the change in data layout: a CRF takes whole sentences (lists of lists), not a flat list of tokens, because it must know where each sequence starts and ends. c1 and c2 are the L1 and L2 regularization strengths.

Beyond CRFs

Neural taggers learn their own features. A BiLSTM-CRF reads the sentence in both directions with recurrent networks and puts a CRF layer on top for the label transitions; today a fine-tuned transformer with a token classification head is the standard. The ideas from this lesson, context windows, orthographic clues, and legal tag sequences, are all still there; they are just learned instead of written by hand. Feature-based CRFs remain a good choice for small datasets, tight latency or memory limits, and as an interpretable baseline: you can print the weights and see why a decision was made.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why did the model with only word-identity features get none of the entity tags right in the test set?

    Word identity only memorizes the training names. Shape and context features describe what a name looks like and where it appears, so they carry over to new words.
  2. 2. What does a shape feature such as Xxxxx capture?

    Mapping uppercase to X, lowercase to x and digits to d gives a coarse description that generalizes to words never seen in training.
  3. 3. What problem does independent token classification have for BIO tagging?

    Each token is classified without regard to its neighbors' labels, so impossible tag transitions can appear. A CRF or constrained decoding addresses this.
  4. 4. What extra weights does a linear-chain CRF have compared with a per-token logistic regression?

    Transition weights let the model prefer likely label pairs, like B-ORG then I-ORG, and make sequence-level decoding with Viterbi possible.
  5. 5. Why does sklearn-crfsuite take lists of sentences instead of one flat list of tokens?

    Transitions connect neighboring tokens within a sentence, so the sequence structure has to be part of the training data.

Summary

  • Sequence labeling assigns one label per token; classic systems describe each token with features of the word (shape, suffix, lowercase form) and of its context window.
  • DictVectorizer plus a linear classifier makes a strong baseline; in our experiment, context features lifted entity-tag F1 from 0.324 to 0.829 on unseen names.
  • Classifying tokens independently can produce illegal BIO sequences; constrained Viterbi decoding removes them but cannot learn which legal sequences are likely.
  • A linear-chain CRF learns feature weights and label-transition weights together and decodes with Viterbi; sklearn-crfsuite is the common library, and BiLSTM-CRF and transformers are its neural successors.
  • Always compare feature sets on held-out data with unseen words, and report entity-level scores.

Next, move from tokens to meaning with Distributional Semantics.