Skip to content
Tutorials Center
My learning

Topics

Lesson 52 of 53 ยท Practice and Review

Generative AI Interview Questions

32 Generative AI interview questions with concise answers on LLMs, attention, sampling, RAG, agents, evaluation, safety and fine-tuning.

Interviews for AI engineering roles test two things: whether you understand how language models behave, and whether you can explain it clearly to people who build products. This page collects 32 questions that come up again and again, grouped by topic, with short, accurate answers you can expand in your own words.

Try to answer each question out loud before you open it. If you cannot explain an idea in two or three sentences, follow the link to its lesson and review it. Three small live editors let you check the claims that are easiest to get wrong, so run them before you repeat the claim in an interview.

A good answer has three parts

Definition One plain sentence.
Mechanism How or why it works.
Trade-off When it helps and when it hurts.

Add a tiny example if you have time.

1 / 2

In a real interview, a short correct answer plus a concrete example beats a long recital. Mention trade-offs ("this is more accurate but slower and costlier"): that is what separates mid-level answers from junior ones.

Fundamentals

1. What is a large language model?

An LLM is a neural network, almost always a transformer, trained on huge amounts of text to predict the next token given the tokens before it. Repeating that prediction one token at a time produces text. Capabilities like translation, summarizing or coding emerge because predicting text well requires modeling grammar, facts and patterns of reasoning. It is a statistical model of text, not a database of facts. See What Is Generative AI?.

2. What is a token, and why not just use words or characters?

A token is the unit of text a model reads and writes: often a word, part of a word or a single character. Whole words need an enormous vocabulary and cannot handle new words; characters make sequences very long and slow. Subword tokenizers such as byte-pair encoding strike a balance: common words are one token, rare words split into pieces, and nothing is out of vocabulary. See Tokens and Tokenization and Byte-Pair Encoding from Scratch.

3. What are the main stages of training a chat model?

First pretraining on a large text corpus with next-token prediction gives a base model. Then supervised fine-tuning on instruction and answer pairs teaches it to follow requests. Finally preference tuning (RLHF, DPO or similar) uses human or model judgments of which answers are better to make it more helpful and safe. See How LLMs Are Trained and Alignment: RLHF, DPO and Preference Data.

4. What is the difference between a base model and an instruct (chat) model?

A base model just continues text, so a question may be continued with more questions. An instruct model has been fine-tuned to treat input as a request and respond helpfully, using a chat format with roles. Base models are useful for research and for custom fine-tuning; instruct models are what most applications call.

5. What is a context window?

It is the maximum number of tokens the model can consider at once, covering both the prompt and the generated answer. Anything outside it is invisible to the model. Bigger windows allow longer documents but cost more and can dilute attention, so models sometimes use information in the middle of a long prompt less reliably. See Context Windows and the KV Cache.

6. Why do LLMs hallucinate?

The training objective rewards plausible continuations, not truth. When the model lacks a fact, the most plausible-sounding continuation is still a fluent sentence, and nothing in the model signals "I don't know" unless it was trained to. Mitigations: ground answers in retrieved sources, ask for citations, allow "I don't know", verify outputs and lower the temperature for factual tasks. See Hallucinations and How to Reduce Them.

Transformers and attention

7. Explain self-attention in a few sentences.

Each token is turned into a query, a key and a value vector. A token's query is compared (dot product) with every key to get scores, a softmax turns the scores into weights that sum to 1, and the token's new representation is the weighted sum of the value vectors. So each token gathers information from the tokens most relevant to it. See Attention from Scratch.

8. Why are attention scores divided by the square root of the key dimension?

Dot products of random vectors grow in size with the dimension. Large scores push softmax into saturation, where one weight is almost 1 and the others almost 0, and gradients become tiny. Dividing by the square root of the dimension keeps the scores at a stable scale. The first live editor below shows the effect.

9. Why does a transformer need positional information?

Attention treats its input as a set: shuffle the tokens and, without positions, the outputs are shuffled in the same way and nothing else changes. Word order matters, so positions are added to or mixed into the vectors, as sinusoidal encodings, learned embeddings or rotary encodings (RoPE). See The Transformer Block.

10. What do multiple attention heads add?

Each head has its own query, key and value projections, so different heads can focus on different relationships at the same time, for example nearby words, matching brackets or a pronoun's subject. Their outputs are concatenated and mixed by one more projection. The total compute stays similar to a single large head.

11. How does attention cost grow with sequence length?

Every token attends to every earlier token, so compute and memory for the score matrix grow with the square of the length. Doubling the prompt roughly quadruples that part of the cost, which is why long contexts are expensive and why techniques such as sparse or sliding-window attention and efficient kernels exist.

12. What is the KV cache and why does it matter?

While generating, keys and values of earlier tokens never change, so the server keeps them in memory instead of recomputing them for every new token. That makes each step cheap but uses a lot of GPU memory, growing linearly with context length and with the number of concurrent users. The second editor below calculates it.

13. What is the difference between encoder-only, decoder-only and encoder-decoder models?

Encoder-only models (BERT-style) see the whole text in both directions and are used for classification and embeddings. Decoder-only models (GPT-style) see only earlier tokens through a causal mask and generate text. Encoder-decoder models read the input with an encoder and generate with a decoder, which suits translation. Most current LLMs are decoder-only.

Sampling and decoding

14. What does temperature do?

The logits are divided by the temperature before softmax. Below 1 the distribution sharpens, so the top token dominates and output is more predictable; above 1 it flattens, so unlikely tokens appear more often. Temperature 0 means always taking the most likely token. See Logits, Softmax and Temperature.

15. What is the difference between top-k and top-p sampling?

Top-k keeps a fixed number of the most likely tokens. Top-p (nucleus sampling) keeps the smallest set of tokens whose probabilities add up to p, so the set is small when the model is confident and larger when it is not. Both cut off the long tail of unlikely tokens before sampling. See Top-k, Top-p and Min-p Sampling.

16. Is output deterministic at temperature 0?

Mostly, but not guaranteed. Floating-point arithmetic on GPUs can differ slightly between runs, batch sizes and hardware, and near-ties between tokens can then flip. Providers often offer a seed for better reproducibility, but treat identical output as likely, not certain. Test with that in mind.

17. When would you use beam search instead of sampling?

Beam search keeps several candidate sequences and prefers the highest total probability. It suits tasks with a single best output, such as translation or speech recognition. For open-ended chat it tends to produce bland, repetitive text, which is why sampling dominates there. See Greedy Decoding, Beam Search and Repetition Penalties.

Check the claims yourself

Two of the answers above are easy to say and easy to get wrong. This editor runs question 8 (why scale by the square root of the dimension) and question 14 (temperature) on small random data.

Python Scaling and temperature

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With a large dimension, the unscaled weights collapse onto a single key (the biggest weight approaches 1), while the scaled ones stay spread out. For temperature, a small value concentrates almost everything on the first token and a large one approaches a uniform distribution.

Prompting

18. What is few-shot prompting and when is it better than zero-shot?

Few-shot prompting puts a handful of input and output examples in the prompt so the model infers the task and format from them (in-context learning). It helps when instructions alone are ambiguous, when you need a strict output format or a particular style. The costs are extra input tokens on every call and sensitivity to which examples you choose and in what order. See Zero-Shot, One-Shot and Few-Shot Prompting.

19. What is chain-of-thought prompting?

Asking the model to reason step by step before answering. Writing intermediate steps gives it more computation per answer and tends to improve arithmetic, logic and multi-step problems. Dedicated reasoning models do this internally, so explicit instructions help less there, and the extra tokens cost money and time. See Reasoning: Chain of Thought and Beyond.

20. How do you get reliable structured output such as JSON?

Describe the schema and give an example, use the provider's structured-output or JSON mode if there is one, validate every reply in code, and retry with the error message when validation fails. Constrained decoding, which only allows tokens that keep the output valid, gives the strongest guarantee. Still validate meaning, not only syntax. See Getting Structured Output (JSON).

Retrieval-augmented generation

21. What is RAG, and when do you choose it over fine-tuning?

Retrieval-augmented generation fetches relevant documents at question time and puts them in the prompt, so the model answers from them. Choose it for knowledge that changes, is private or must be cited. Fine-tuning is better for teaching style, format or a narrow skill; it is a poor way to add facts. They combine well. See RAG: Giving Models Your Data.

22. How do you pick a chunk size?

Small chunks give precise matches but lose surrounding context; large chunks keep context but dilute the embedding and waste prompt space. Start with a few hundred tokens with modest overlap, split on natural boundaries such as headings and paragraphs, add metadata, and then tune by measuring retrieval quality on real questions. See Chunking and Preparing Documents.

23. Why combine keyword and vector search?

Embeddings capture meaning but can miss exact terms such as product codes, names and error messages; keyword search (BM25) is the reverse. Running both and merging the rankings, for example with reciprocal rank fusion, then optionally reranking the top results with a cross-encoder, is usually more robust than either alone. See Hybrid Search and Reranking.

24. How do you evaluate a RAG system?

Separate the two halves. For retrieval, build questions with known relevant documents and measure recall@k and MRR. For generation, check that answers are faithful to the retrieved text, that citations point to real sources and that the system says it does not know when the context lacks the answer. Then measure end-to-end on a held-out question set. See Metrics: Retrieval, Benchmarks and Statistics.

Tools and agents

25. What is an agent, and how is it different from a workflow?

An agent is a loop in which the model chooses actions (tool calls), sees their results and decides what to do next until it finishes. In a workflow, the steps are fixed in code and the model only fills in parts of them. Workflows are cheaper, faster and easier to test; use an agent only when the path genuinely depends on intermediate results. See MCP, Workflows and Multi-Agent Systems.

26. How does function calling work?

You describe tools with names and JSON schemas. The model replies with a tool name and arguments instead of text; your code validates and runs the tool, then sends the result back as a message, and the model continues. The model never executes anything itself, which is why you control permissions. See Tool Use and Function Calling.

27. How do you keep an agent from going wrong?

Put hard limits around it: a maximum number of steps, token and cost budgets, a guard against repeated identical calls, truncated tool outputs and narrow tool permissions. Log a trace of every step, require human approval for irreversible actions, and test the loop with a scripted model. See Building an Agent Loop and Project: A Tool-Using Agent.

Evaluation and safety

28. How do you evaluate an LLM application?

Build a test set of realistic inputs with expected outputs or scoring rules, run it on every change, and track the score over time. Use exact or regex checks where answers are checkable, a judge model or human review where they are not, and report confidence intervals, because small test sets are noisy. See Evaluating LLM Applications.

29. What are the weaknesses of using an LLM as a judge?

Judges favor the answer shown first, favor longer answers, may prefer text written by themselves, and can be inconsistent. Reduce the problems with a clear rubric, swapping the order of pairs, comparing against human labels and measuring agreement with a statistic such as Cohen's kappa. See LLM-as-Judge and Human Review.

30. What is prompt injection and how do you defend against it?

Prompt injection is when untrusted text, such as a web page, email or document, contains instructions that the model follows. There is no complete fix, so use layers: treat all retrieved content as data, give the model least-privilege tools, require approval for sensitive actions, filter and sanitize outputs, and monitor. Never put secrets in a prompt. See Prompt Injection and LLM Security.

31. How should you handle personal data when using LLM APIs?

Send as little as possible, redact or pseudonymize identifiers before the call, check the provider's retention and training terms, and keep logs under the same rules as the original data. Tell users when they are talking to AI and keep a human in the loop for high-stakes decisions. See Bias, Privacy, Copyright and Responsible Use.

Customizing and running models

32. What are LoRA and quantization?

LoRA fine-tunes a model by freezing the original weights and training small low-rank matrices added to them, so only a tiny fraction of parameters changes and memory needs are far lower. Quantization stores weights with fewer bits (for example 4 or 8 instead of 16), shrinking memory and often speeding inference at a small quality cost. Combined, they let large models be tuned and run on modest hardware. See LoRA, Quantization and Efficient Models.

Memory for the KV cache

Question 12 claimed that the KV cache grows linearly with context and with users. This calculator makes it concrete for a model with 32 layers, 8 key-value heads of size 128 and 16-bit numbers.

Python KV cache memory calculator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each token costs 128 KiB, so one 32,000-token conversation needs nearly 4 GiB for its cache alone and 20 of them need about 78 GiB, before counting the model weights. That is why providers charge more for long prompts and offer prompt caching.

Why hybrid search helps

Finally, a quick demo for questions 21 to 23: keyword overlap finds an exact code, while a made-up meaning-based score finds a paraphrase. It shows why you would want both.

Python Why hybrid search helps

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The exact code E4021 is matched only by keywords, and the paraphrase "card was rejected" is matched best by meaning, even though it shares almost no words with the "declined" document. The meaning scores here are invented to illustrate the idea; a real system would compute them with an embedding model.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the main reason attention scores are divided by the square root of the key dimension?

    Without scaling, large dimensions give large scores, softmax becomes nearly one-hot and gradients vanish.
  2. 2. Which statement about RAG versus fine-tuning is the most accurate?

    Retrieval brings facts in at question time and allows citations; fine-tuning changes behavior but is a poor way to store facts.
  3. 3. Why is a step limit essential in an agent loop?

    Models can get stuck repeating actions; a hard cap bounds cost and time.
  4. 4. What is a known weakness of LLM judges?

    Position and verbosity biases are well documented, so swap orders and compare with human labels.

Summary

  • Know the basics cold: tokens, next-token training, context windows and why hallucinations happen.
  • Be able to explain attention, scaling, positions, heads and the KV cache in plain words.
  • Sampling parameters reshape the next-token distribution; temperature 0 is near-deterministic, not guaranteed.
  • RAG is for knowledge, fine-tuning is for behavior, and agents need limits, logs and least privilege.
  • Evaluation needs test sets, honest statistics and awareness of judge biases.
  • Always answer with a definition, a mechanism and a trade-off.

Next: keep the essentials at hand in the Generative AI Cheat Sheet.