Skip to content
Tutorials Center
My learning

Topics

Lesson 1 of 53 ยท Getting Started

What Is Generative AI?

Learn what generative AI is, how it differs from predictive models, what large language models can and cannot do, and build a tiny text generator in Python.

Most software you have used so far follows rules that a person wrote down. Most classic machine learning goes one step further: it learns a rule from examples, then uses it to put new inputs into a box ("spam" or "not spam", "cat" or "dog"). Generative AI does something different. It learns what data looks like, and then produces new data of the same kind: a paragraph, a picture, a melody, a function.

This lesson gives you the map for the whole tutorial. You will see how generative models differ from predictive ones, what large language models (LLMs) are good and bad at, which kinds of content can be generated, and you will run a very small text generator yourself. It is tiny, but it is built on the same idea as the models behind modern chatbots.

Predictive versus generative

A predictive (or discriminative) model maps an input to a label or a number. The output space is small and fixed: a class, a price, a probability. If you want a refresher on that style of model, see Machine Learning with scikit-learn.

A generative model produces something whose output space is huge and open-ended: any sequence of words, any image of 512 by 512 pixels. Instead of answering "which label?", it answers "what could come next?" or "what might a sample from this data look like?"

Two kinds of model

Predictive Takes an input and picks a label or a number.
Generative Creates new text, images, audio or code.

Same math toolbox, different question.

1 / 2

The line between the two is blurry. Many generative models are trained by predicting something (the next word, a missing patch of an image). The difference is that we then use the prediction machine repeatedly to build a whole new output, one piece at a time.

Python Predict a label, then generate text

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first function can only ever return two strings. The second can return an enormous number of different sentences, and some of them were never in the training text. That open-endedness is what makes generative models powerful, and also what makes them harder to control and to test.

What can be generated

The same idea works for many kinds of data, as long as you can turn the data into a sequence or a grid of numbers.

  • Text. Chat answers, summaries, translations, emails, stories. This tutorial focuses here.
  • Code. Source code is text with strict rules, so language models write functions, tests and explanations of errors.
  • Images. Diffusion models start from noise and refine it into a picture that matches a text description (see How Image Generation Works).
  • Audio and video. Speech, music and short clips follow the same pattern with different encodings (see Audio, Video and Multimodal Models).
  • Structured data. JSON records, SQL queries, synthetic tables for testing.

What a large language model is

A large language model is a neural network trained on a vast amount of text to predict the next token (a small piece of text, covered in Tokens and Tokenization). To write an answer, the model predicts one token, adds it to the text so far, and predicts the next one, over and over until it decides to stop.

flowchart LR
    P["Prompt so far"] --> M["Model"]
    M --> D["Probabilities for the next token"]
    D --> S["Pick one token"]
    S --> A["Append to the text"]
    A --> P

Generation is a loop: predict, pick, append, repeat.

Nothing in that loop checks whether a statement is true. The model produces text that is likely given everything it learned, and usually likely text is also correct. When it is not, you get a confident mistake, known as a hallucination (see Hallucinations and How to Reduce Them).

Strengths and limits

Language models are strong at tasks where fluent language and broad pattern knowledge matter, and weaker where exactness or fresh facts matter.

Where LLMs shine

Drafting Emails, outlines, first versions.
Transforming Summarize, translate, reformat.
Explaining Break an idea into small steps.
Extracting Pull fields out of messy text.
1 / 2

Many of the later lessons are about closing those gaps: retrieval to add fresh and private data (RAG: Giving Models Your Data), tools to calculate or look things up (Tool Use and Function Calling), and evaluation to find out how often the system is wrong (Evaluating LLM Applications).

Why a bigger corpus makes a better generator

The toy generator above only knows which word followed which in one short sentence, so its output is repetitive and often nonsense. Feed it more text and it improves, but a simple count table never learns grammar or meaning beyond neighbouring words. LLMs replace the count table with a neural network that looks at thousands of previous tokens at once and has billions of adjustable numbers (parameters). Scale and architecture, not a different goal, are what separate the toy from a chatbot.

Python More data, better imitation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that the small corpus can only walk one path, while the larger one has branches, so different seeds give different stories. Branching is where "creativity" comes from in a generator: several continuations are possible and one is picked at random. Logits, Softmax and Temperature shows how you control that choice.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the main difference between a predictive and a generative model?

    Predictive models choose from a small fixed set of answers, while generative models create open-ended outputs like sentences or pictures.
  2. 2. How does an LLM write a long answer?

    Generation is a loop: predict a distribution over the next token, pick one, append it, and repeat.
  3. 3. Why can a language model state something false in a confident tone?

    The training goal rewards plausible continuations. Truth usually correlates with plausibility, but nothing in the loop guarantees it.
  4. 4. In the toy bigram generator, why do different seeds give different sentences from the larger corpus?

    A word that appeared before different words has several options in the table, and the random generator picks among them.

Summary

  • Predictive models choose a label or number; generative models create new text, code, images, audio or structured data.
  • Many generative models are trained by predicting the next piece of data, then used in a loop to build whole outputs.
  • An LLM predicts the next token, appends it and repeats; it optimizes for plausible text, not verified truth.
  • LLMs are strong at drafting, transforming, explaining and extracting, and weaker at exact math, fresh facts and secrets.
  • A table of word-to-word counts is a tiny language model; LLMs replace it with a neural network at massive scale.

Next: see how real models get their abilities in How LLMs Are Trained.