Skip to content
Tutorials Center
My learning

Topics

Lesson 45 of 53 ยท Production

Running Models Locally

Learn how open-weight LLMs run on your own machine: GGUF and quantized formats, local runtimes, memory sizing for weights and KV cache, and local APIs.

You do not need a cloud API to use a language model. Open-weight models can run on a laptop, a workstation or your own server. People do this for privacy (data never leaves the machine), cost (no per-token bill), offline use, control over versions, and plain curiosity about how it works.

The catch is hardware. A model is a very large file of numbers, and it must fit in memory to run at a useful speed. This lesson explains the file formats, the runtimes, how to estimate whether a model fits on your machine, and how to serve it through an API that looks like the hosted ones. The commands are read-only, since the site cannot run a model. The sizing calculator is live.

Open weights, in practice

An open-weight model is one whose trained parameters you can download (see The Model Landscape). Check the license: some are fully permissive, others limit commercial use or very large deployments. "Open weights" is not the same as "open source", because training data and code are often not released.

A model release usually contains:

  • the weights: tensors of numbers, the bulk of the download;
  • a tokenizer that must match the weights exactly (see Tokens and Tokenization);
  • a config describing layers, sizes and context length;
  • often a chat template that formats system, user and assistant messages the way the model was trained to expect. Using the wrong template quietly degrades answers.

Quantized formats and GGUF

Weights are normally trained in 16-bit floating point, which needs 2 bytes per parameter. Quantization stores them in fewer bits, such as 8, 5 or 4, trading a little quality for a much smaller file (the math is in LoRA, Quantization and Efficient Models).

Why quantize?

Smaller files 4-bit weights are about a quarter of 16-bit.
Fits in memory A bigger model can run on the same machine.
Faster decoding Less data to read per token.

The price is a small, usually acceptable, loss of quality.

1 / 2

GGUF is a single-file format used by the llama.cpp family of runtimes. It stores the weights (often quantized), the tokenizer and the model's metadata together, so one file is enough to run it. File names carry the quantization level, for example Q4_K_M (about 4-bit, a "medium" mixed scheme) or Q8_0 (8-bit). As a rule, 8-bit is nearly lossless, 4- to 5-bit is a good balance, and 2- to 3-bit visibly hurts quality.

Runtimes and tools

You rarely run a model by hand. A runtime loads the file, runs the transformer and exposes a command line, a chat UI or an API.

  • llama.cpp is a C/C++ engine that runs GGUF models on CPUs, Apple silicon and many GPUs. Many other tools are built on it.
  • Ollama wraps a runtime in a friendly command line and a local server, with a library of ready-to-download models.
  • LM Studio and similar desktop apps give a graphical way to download and chat with models.
  • vLLM and similar servers target GPU machines and many concurrent users, using batching and efficient KV-cache management.
  • Hugging Face Transformers is the Python library most research and fine-tuning code uses.

These read-only commands show typical usage. Names and flags change between versions, so check each tool's documentation.

Terminal
# Ollama: download a model once, then chat in the terminal
ollama pull llama3.2
ollama run llama3.2 "Explain recursion in two sentences."

# llama.cpp: run a GGUF file directly
llama-cli -m ./models/my-model-Q4_K_M.gguf -p "Write a haiku about caching." -n 64

# llama.cpp: start an HTTP server
llama-server -m ./models/my-model-Q4_K_M.gguf --port 8080

Sizing: will it fit?

Memory is the first question. You need room for three things:

  1. Weights: parameters x bits per weight / 8 bytes. A 7-billion-parameter model is about 14 GB at 16-bit, about 7 GB at 8-bit and about 4 GB at 4-bit (plus a little overhead for scales).
  2. KV cache: the stored keys and values for every token in the context, which grows with context length (see Context Windows and the KV Cache).
  3. Working buffers and the OS: leave one to two gigabytes free.

On a GPU the weights must fit in video memory (VRAM) to be fast. Runtimes can offload some layers to the CPU and system RAM when VRAM is too small, which works but slows generation a lot. Apple silicon uses unified memory shared by CPU and GPU, so total RAM is the limit.

Python Memory-sizing calculator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Try changing the model size to 70 in the calculator. A 70-billion-parameter model at 4 bits needs roughly 35 GB for weights alone, which is why large models need multi-GPU servers or a big-memory workstation.

Speed: what to expect

Generation speed is mostly limited by memory bandwidth, because every new token requires reading all the weights once. A rough upper bound is bandwidth / model size. Halving the size by quantizing roughly doubles the speed. The next sketch makes that concrete with invented bandwidth numbers.

Python Estimating tokens per second

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Treat these as ceilings, not promises. Prompt processing (prefill) is limited by compute instead, so a long prompt can take noticeable time on a CPU.

Serving a local API

Most local runtimes can start an HTTP server that mimics a hosted API. Many follow the same request shape used by popular providers, so existing client code works after you change the base URL. This is a read-only example of the idea:

Python
# Read-only: needs a local server such as Ollama or llama-server running.
import json
import urllib.request

body = {
    "model": "llama3.2",
    "messages": [
        {"role": "system", "content": "You are a concise assistant."},
        {"role": "user", "content": "Name three uses of a hash map."},
    ],
    "temperature": 0.2,
}
request = urllib.request.Request(
    "http://localhost:11434/v1/chat/completions",
    data=json.dumps(body).encode(),
    headers={"Content-Type": "application/json"},
)
with urllib.request.urlopen(request) as response:
    reply = json.load(response)
print(reply["choices"][0]["message"]["content"])

A local server has no authentication by default. Keep it bound to localhost. If you expose it to a network, put it behind a gateway with keys and rate limits (see Deploying and Monitoring LLM Apps).

Local or hosted?

Local models shine for private data, offline tools, predictable cost at steady volume and experimentation. Hosted models usually win on top quality, zero operations and elastic scale. Many products use both: a small local model for sensitive or high-volume steps and a hosted frontier model for the hardest requests. Whichever you choose, evaluate on your own tasks first (Evaluating LLM Applications).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. About how much memory do the weights of a 7-billion-parameter model need at 16-bit precision?

    16 bits is 2 bytes per parameter, so 7 billion parameters need about 14 GB before the KV cache and buffers.
  2. 2. What does a GGUF file contain?

    GGUF packs everything a runtime such as llama.cpp needs into one file, often with quantized weights.
  3. 3. Why does quantizing a model usually make local generation faster?

    Decoding is limited by memory bandwidth because all weights are read for every token. Smaller weights mean less to read.
  4. 4. What grows with context length and must be added to your memory estimate?

    The keys and values stored for every token in the context make up the KV cache, which scales with context length.

Summary

  • Open-weight models can run on your own hardware for privacy, cost control and offline use; read the license first.
  • Quantization stores weights in fewer bits; GGUF is a single-file format popular with llama.cpp-based runtimes.
  • Weight memory is parameters x bits / 8; add the KV cache, buffers and room for the operating system.
  • Decoding speed is capped by memory bandwidth divided by model size, so smaller quantized models are faster.
  • Local runtimes can serve a hosted-style HTTP API; keep it on localhost or protect it with a gateway.
  • Use the correct chat template and evaluate on your own tasks before committing.

Next: put the pieces together in your first project, Project: A Chatbot with Memory.