Skip to content
Tutorials Center
My learning

Topics

Lesson 63 of 63 ยท Practice and Review

NLP Cheat Sheet

A one-page NLP cheat sheet with the text pipeline, key formulas, scikit-learn snippets, a metrics table and a guide to choosing the right model.

This page packs the whole course into one place: the steps of a text pipeline, the formulas you keep looking up, copy-ready scikit-learn snippets, a table of metrics and a guide for choosing a model. Use it as a quick refresher while coding, before an interview, or next to the practice exercises.

Each section links to the lesson that explains the topic in depth. Two live editors let you check a formula and a complete pipeline on the spot.

The whole course in five steps

Clean Normalize and tidy text.
Tokenize Words or subwords.
Vectorize Counts, TF-IDF, embeddings.
Model Classify, tag, generate.

Then evaluate, inspect errors and repeat.

1 / 2

The pipeline at a glance

Step What it does Typical tools Lesson
Clean Fix case, accents, whitespace, markup str, re, unicodedata Cleaning
Split Sentences, then tokens regex, subword tokenizers Tokenization
Normalize Stop words, stemming, lemmas word lists, stemmers Stemming
Vectorize Text to numbers CountVectorizer, TfidfVectorizer TF-IDF
Model Learn a mapping Naive Bayes, logistic regression, transformers Classification
Evaluate Measure and inspect errors precision, recall, F1 Metrics

Keep the whole chain in one object (Pipeline) so the same steps run for training and prediction. See reusable pipelines.

Text snippets

Python
import re, unicodedata
from collections import Counter

text.lower().strip()                                   # case and whitespace
re.sub(r"\s+", " ", text)                              # collapse whitespace
re.sub(r"https?://\S+", " ", text)                     # drop URLs
re.findall(r"[a-z']+", text.lower())                   # simple word tokens
Counter(tokens).most_common(10)                        # top words
list(zip(tokens, tokens[1:]))                          # bigrams

# remove accents
"".join(c for c in unicodedata.normalize("NFD", text)
        if not unicodedata.combining(c))

Normalize Unicode first (NFC or NFKC), because the same visible character can have several encodings. See levels of language and text basics.

Regex patterns that come up often

Goal Pattern
Words with apostrophes [A-Za-z]+(?:'[a-z]+)?
Numbers with decimals \d+(?:\.\d+)?
Simple email [\w.+-]+@[\w-]+(?:\.[\w-]+)+
ISO date \b\d{4}-\d{2}-\d{2}\b
Hashtag #\w+

Key formulas

TEXT
TF-IDF        tfidf(w, d) = tf(w, d) * idf(w)
              idf(w) = ln(N / df(w))                      textbook
              idf(w) = ln((1 + N) / (1 + df(w))) + 1      scikit-learn default

Cosine        cos(u, v) = (u . v) / (|u| * |v|)
Jaccard       J(A, B) = |A and B| / |A or B|
PMI           pmi(x, y) = log( p(x, y) / (p(x) * p(y)) )

Bigram prob   P(w2 | w1) = count(w1 w2) / count(w1)
Add-one       P(w2 | w1) = (count(w1 w2) + 1) / (count(w1) + V)
Perplexity    PP = exp( -(1/n) * sum( ln P(w_i | context) ) )

Naive Bayes   class = argmax_c  log P(c) + sum_w count(w) * log P(w | c)
Softmax       softmax(z)_i = exp(z_i / T) / sum_j exp(z_j / T)
Attention     softmax(Q K^T / sqrt(d_k)) V
Edit distance D[i][j] = min(D[i-1][j] + 1, D[i][j-1] + 1,
                            D[i-1][j-1] + (a_i != b_j))

Lessons: TF-IDF, similarity, n-grams, language models, smoothing, spelling, attention.

Python Check a few formulas

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

scikit-learn snippets

Vectorizers

Python
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer, HashingVectorizer

CountVectorizer(lowercase=True, stop_words="english", ngram_range=(1, 2), min_df=2)
TfidfVectorizer(sublinear_tf=True, max_df=0.9, min_df=2)
TfidfVectorizer(analyzer="char_wb", ngram_range=(3, 5))   # character n-grams
HashingVectorizer(n_features=2**18, alternate_sign=False)

X = vec.fit_transform(train_texts)    # learn vocabulary on TRAINING data only
X_test = vec.transform(test_texts)    # reuse it on new data
vec.get_feature_names_out()

See bag of words and feature hashing.

Train, search and evaluate

Python
from sklearn.pipeline import make_pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split, cross_val_score, GridSearchCV
from sklearn.metrics import classification_report, confusion_matrix

X_train, X_test, y_train, y_test = train_test_split(
    texts, labels, test_size=0.2, stratify=labels, random_state=0)

model = make_pipeline(TfidfVectorizer(), LogisticRegression(max_iter=1000))
model.fit(X_train, y_train)
print(classification_report(y_test, model.predict(X_test)))

scores = cross_val_score(model, texts, labels, cv=5, scoring="f1_macro")

grid = GridSearchCV(model,
    {"tfidfvectorizer__ngram_range": [(1, 1), (1, 2)],
     "logisticregression__C": [0.1, 1, 10]},
    cv=5, scoring="f1_macro").fit(X_train, y_train)
print(grid.best_params_)

See pipelines.

Other useful pieces

Python
from sklearn.naive_bayes import MultinomialNB
from sklearn.svm import LinearSVC
from sklearn.decomposition import NMF, LatentDirichletAllocation, TruncatedSVD
from sklearn.cluster import KMeans
from sklearn.metrics.pairwise import cosine_similarity

MultinomialNB(alpha=1.0)                  # alpha = Laplace smoothing
LinearSVC(C=1.0)                          # strong linear baseline
NMF(n_components=5, random_state=0)       # topics from TF-IDF
KMeans(n_clusters=4, random_state=0, n_init=10)
cosine_similarity(X_query, X_docs)        # rows = queries, columns = documents
Python A complete pipeline in ten lines

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Metrics at a glance

Metric Formula Use it when
Accuracy correct / total Classes are balanced
Precision TP / (TP + FP) False alarms are costly
Recall TP / (TP + FN) Misses are costly
F1 2PR / (P + R) You need one number for a rare class
Macro F1 mean of per-class F1 Every class matters equally
Micro F1 pooled TP, FP, FN Large classes should dominate
Perplexity exp(mean negative log probability) Comparing language models (lower is better)
BLEU n-gram precision with a brevity penalty Machine translation
ROUGE n-gram recall against references Summarization
Cohen's kappa agreement beyond chance Quality of human labels

Always compare against a majority-class baseline and read the confusion matrix. See evaluating classifiers.

Choosing a model

Task Start with Move to when needed
Spam, topic, intent (labeled data) TF-IDF + logistic regression or Naive Bayes fine-tuned transformer
Sentiment lexicon baseline, then TF-IDF + linear model fine-tuned transformer
Search and ranking inverted index + TF-IDF or BM25 embeddings + reranking
Similar documents TF-IDF + cosine sentence embeddings
Topics without labels NMF or LDA clustering embeddings
Tagging (POS, entities) rules or gazetteers, then a feature-based tagger pre-trained encoder
Spelling edit distance + word frequencies language model scoring
Question answering on your documents retrieval + extractive reader RAG with an LLM
Summarization, translation, chat pre-trained seq2seq or LLM fine-tuning, prompting patterns

Rule of thumb: a linear model on TF-IDF is a hard baseline to beat with fewer than a few thousand labeled examples. Build it first, so every fancier model has to earn its cost.

Common pitfalls

  • Leakage: fit vectorizers and scalers inside the pipeline, on the training fold only.
  • Over-cleaning: removing stop words, punctuation or case can delete signal (negations, emoji, names).
  • Wrong metric: accuracy on imbalanced data hides failure; check recall and macro F1.
  • Vocabulary mismatch: words unseen in training are silently ignored by a fitted vectorizer.
  • Different preprocessing at prediction time: reuse the exact same pipeline object.
  • Trusting generated text: large models can produce fluent, wrong answers; ground and verify them.

Keep going

You now have the full toolkit in one page. To put it to work, build one of the projects, such as the spam filter, or return to the exercises and the interview questions whenever you want to practice.