Skip to content
Tutorials Center
My learning

Topics

Lesson 1 of 37 ยท Getting Started

What Is a Large Language Model?

Learn what a large language model is, how it turns text into next-token predictions, why size matters, and what runs live in this course.

A large language model (LLM) is a neural network trained on a huge amount of text to do one deceptively simple job: given some text, predict what comes next. Chat assistants, code completion, translation, and summarizers are all built on top of that single skill. The "large" refers to two things at once: the number of learned numbers inside the model (its parameters, from millions to hundreds of billions) and the amount of text it learned from (often trillions of tokens).

This course takes the model apart piece by piece. You will see how text becomes numbers, how a Transformer mixes information between words, how training and scaling work, how a base model becomes an assistant, and how to run, evaluate, and use one responsibly. Almost every idea is demonstrated with a small Python program you can run in the page.

The core idea in one paragraph

An LLM never sees letters or words directly. Text is cut into tokens (pieces such as " the", "ing", or "Paris"), and every token is mapped to an integer id. The model reads a sequence of ids and outputs a probability for every token in its vocabulary being the next one. A program picks one token (by taking the most likely one or by sampling), appends it to the text, and asks again. Repeating this loop, one token at a time, produces a whole answer.

The LLM loop

Tokenize Text becomes token ids.
Predict The model scores every token.
Pick Choose one token.
Repeat Append it and go again.

Everything an LLM writes comes from this loop.

1 / 2

Language modeling, formally

A language model assigns a probability to a sequence of tokens. Using the chain rule of probability, the probability of a whole sentence is a product of next-token probabilities:

P(t1, t2, ..., tn) = P(t1) * P(t2 | t1) * P(t3 | t1, t2) * ...

The model only has to learn the conditional P(next | everything so far). Training adjusts the parameters so the real next token in the training text gets a high probability. That is the whole objective, and it is covered in detail in Pretraining Objective and Cross-Entropy.

Here is the smallest possible "language model": it counts which word follows which in a tiny text, then turns the counts into probabilities.

Python A language model made of counts

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

This model has a vocabulary of about ten words and a memory of exactly one previous word. A real LLM has a vocabulary of tens of thousands of tokens and looks at thousands of previous tokens at once, but the output has the same shape: a probability distribution over the next token.

From probabilities to text

The distribution is not text yet. A decoding step turns it into a choice. Greedy decoding always takes the most likely token; sampling draws one at random in proportion to its probability. You will meet temperature, top-k, and top-p in the inference chapter, but you can already see the difference:

Python Greedy versus sampled generation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Greedy decoding gets stuck in a loop because the most likely follower of each word leads back to the same place. Sampling wanders. Real systems balance the two.

Why scale changes what a model can do

Tiny count tables only capture surface patterns. As models get bigger and see more text, they pick up grammar, facts, styles, and even rough reasoning, because predicting the next token well in all kinds of text requires modeling the world that produced it. To predict the word after "The capital of France is", a model benefits from knowing geography. To predict the next line of a function, it benefits from understanding the code. This is why one training objective can yield so many skills. Scaling Laws and Compute shows how performance improves smoothly and predictably with size.

What an LLM is not

  • Not a database. It stores statistical regularities in its weights, not a table of facts, so it can state things confidently and be wrong.
  • Not a person or a search engine. It has no live access to the world unless a program gives it tools or retrieved documents.
  • Not deterministic by nature. The same prompt can give different answers when sampling is enabled.
  • Not magic. The whole system is matrix arithmetic, which is why you can reproduce its key parts in a few dozen lines of NumPy.

Where the code in this course runs

The Python examples run in your browser with real NumPy, SciPy, and matplotlib. They use tiny models you can inspect completely. Production models such as the ones behind chat assistants need GPUs and cannot run in a page, so wherever the real thing cannot run, the lesson says so and shows a clearly labelled small stand-in that demonstrates the same mechanism.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the basic task an LLM is trained to perform?

    Next-token prediction is the training objective; other skills emerge from doing it well on lots of text.
  2. 2. What does the 'large' in large language model refer to?

    Size means parameter count and training data, together with the compute used to train.
  3. 3. Why does greedy decoding in the count model get stuck repeating itself?

    With a one-word memory and always taking the top follower, the path can enter a cycle.
  4. 4. Which statement about an LLM is true?

    The model outputs a distribution; a decoding step picks the token. Models read tokens, not letters.

Summary

  • An LLM is a neural network that predicts the next token from the previous ones.
  • Text is split into tokens, mapped to ids, scored, one token is chosen, and the loop repeats.
  • "Large" means many parameters, a huge training set, and heavy compute.
  • Decoding turns the predicted distribution into a choice; greedy and sampling behave very differently.
  • LLMs store patterns in weights, not facts in a database, so they can be wrong with confidence.

Next: Next-Token Prediction