Skip to content
Tutorials Center
My learning

Topics

Lesson 37 of 37 ยท Projects and Practice

Practice: Exercises and Review

Practice large language model skills with worked numeric exercises: token cost, KV cache size, LoRA parameter counts, quantization memory, perplexity and a final review quiz.

Reading about language models is not enough: you need to be able to estimate costs, sizes and scores yourself. Each exercise below states a task, gives a runnable solution and explains how to interpret the result. Try to solve each one on paper first, then run the code to check. These are the back-of-the-envelope calculations engineers do every week when planning training or serving.

Exercise 1: how much does a request cost?

Providers bill per token, with a separate price for input and output. A prompt has 1,200 tokens, the answer is 300 tokens, input costs 2 dollars per million tokens and output costs 8 dollars per million. What does one call cost, and what do 50,000 calls a day cost per month (30 days)?

Python Request cost

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Output tokens are usually several times pricier than input tokens, so trimming the answer length matters as much as shortening the prompt. See serving for why.

Exercise 2: size of the KV cache

During generation the model keeps keys and values for every earlier token (KV cache). Memory per token is 2 * layers * kv_heads * head_dim * bytes. For a model with 32 layers, 8 key/value heads (grouped-query attention), head dimension 128 and 16-bit numbers, how big is the cache for a 8,000-token conversation, and how many conversations fit in 20 GB?

Python KV cache memory

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

This is why grouped-query attention exists: fewer KV heads means more simultaneous users per GPU.

Exercise 3: parameters in a LoRA adapter

A weight matrix is 4096 x 4096. Full fine-tuning updates all of it. LoRA instead learns two small matrices A (4096 x r) and B (r x 4096). For r = 16, how many trainable parameters is that, and what fraction of the full matrix?

Python LoRA parameter count

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The adapter is under one percent of the matrix, which is why one base model can serve many small task-specific adapters.

Exercise 4: memory for model weights

How much memory do the weights of a 7-billion-parameter model need at 16, 8 and 4 bits per weight? Remember activations and the KV cache come on top (quantization).

Python Weight memory by precision

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A 16-bit 7B model needs about 14 GB just for weights; at 4 bits it fits in roughly 3.5 GB, which is how large models reach laptops and phones, at a small quality cost.

Exercise 5: perplexity from log-probabilities

A model assigns these probabilities to the five correct tokens of a sentence: 0.5, 0.25, 0.1, 0.4, 0.2. Compute the average cross-entropy (in nats) and the perplexity (metrics). Then say what a perplexity of 1 would mean.

Python Perplexity by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A perplexity near 4 means the model is, on average, as uncertain as if it were choosing uniformly among about four tokens at each step. A perplexity of 1 means it predicted every token with certainty.

Exercise 6: training compute estimate

A common rule of thumb: training needs about 6 * N * D floating-point operations, where N is parameters and D is training tokens (scaling laws). For N = 7 billion and D = 2 trillion tokens, estimate the total FLOPs and the GPU-days on hardware sustaining 200 teraFLOP/s of useful throughput per GPU.

Python Training compute

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The result is thousands of GPU-days, a useful reminder of why pretraining is done only by organisations with large clusters while fine-tuning is within reach of many more teams.

Review: pick the right tool

Match each problem to the technique that usually solves it best.

Final review quiz

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Output tokens usually cost more than input tokens because:

    Generation is sequential, one token per step, while the prompt is processed in parallel.
  2. 2. Which change shrinks the KV cache the most?

    Cache size scales with the number of KV heads, layers, head size and sequence length.
  3. 3. A LoRA adapter at rank 16 on a 4096 x 4096 matrix trains roughly what share of its parameters?

    r * (d_in + d_out) = 131,072, which is under 1% of 16.8 million.
  4. 4. A 7B model at 4 bits per weight needs about how much memory for weights?

    7e9 weights times half a byte is about 3.5 GB.
  5. 5. Perplexity of a model is exp of:

    Perplexity exponentiates average cross-entropy; lower means less surprise.

Summary

  • Cost, memory and compute questions reduce to short formulas: tokens times price, layers times heads times bytes, 6 * N * D.
  • LoRA, grouped-query attention and quantization all attack size and memory in different places.
  • Perplexity is exponentiated average loss and is only comparable under the same tokenizer and data.
  • Choose techniques by the bottleneck: knowledge, behaviour, memory, throughput or trust.

Congratulations: you have covered the whole path from tokens to deployed applications. Return to the track overview to revisit any chapter.

Next: Large Language Models overview