Skip to content
Tutorials Center
My learning

Topics

Lesson 1 of 36 ยท Getting Started

What Is a Distributed System?

Learn what a distributed system is, why we spread work over many machines, what it costs, and build a first toy cluster of nodes in Python.

A distributed system is a set of independent computers that cooperate by sending messages to each other, and that present themselves to users as one coherent service. When you search the web, stream a video or pay with a card, dozens or thousands of machines take part in that single request, and you never notice.

This tutorial is about the problems that appear the moment one program no longer lives on one machine. They are different from the problems of ordinary programming: parts of the system can fail while other parts keep running, messages can be lost or arrive late, and no machine has a perfect, shared view of what is happening. You will learn the ideas (time, replication, consensus, partitioning) and then watch each one work in small simulations you can run and change right in your browser.

A working definition

Three properties together make a system distributed:

  • Several nodes. A node is any independent unit that runs code: a server, a container, a phone, a process. Each node has its own memory and its own clock.
  • Communication only by messages. Nodes do not share memory. They send bytes over a network, and that network is slow and unreliable compared with a function call.
  • A shared goal. The nodes work together to deliver one service, such as a database, a chat system or a search engine.

A web app and its database running on two servers already form a distributed system. So does a mobile app talking to a backend. The interesting questions start when the same data or the same job is spread over several nodes that must agree.

What makes it distributed?

Many nodes Independent machines, each with its own memory.
Messages They talk over a network, never shared memory.
One goal Users see a single service.

Take away any one of the three and it is just a normal program.

1 / 2

Why distribute?

Nobody builds a distributed system for fun. You do it because a single machine cannot meet a requirement.

Scale. A machine has a fixed amount of CPU, memory, disk and network bandwidth. You can buy a bigger one (vertical scaling), but prices climb steeply and there is a ceiling. Adding more ordinary machines (horizontal scaling) has no hard ceiling, if you can split the work.

Availability. Every machine eventually fails: disks die, power goes out, kernels panic, people deploy bad code. If the service lives on one machine, it is down whenever that machine is. With copies on several machines, the service survives individual failures.

Latency. Light needs roughly 65 ms to travel from Europe to Australia and back, even in a perfect cable. If your users are everywhere, the only way to answer them quickly is to keep data and computation close to each of them.

Organization. Large products are built by many teams. Separate services let teams deploy and fail independently.

The availability arithmetic

Replication is the cleanest example of why this works. Suppose one server is up 99% of the time, and servers fail independently. The probability that all of three copies are down at the same moment is 0.01 ร— 0.01 ร— 0.01, so at least one is up with probability 1 - 0.01ยณ.

Python Availability of replicated servers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice how quickly the downtime shrinks. But the calculation hides an assumption: that failures are independent. Two servers in one rack that share a power supply are not independent. Real designs spread copies across racks, zones and regions for exactly this reason.

The cost of distribution

Everything you gain has a price, and the price is complexity. On one machine a function call either returns or throws. Across a network there is a third outcome: you do not know. You sent a request and nothing came back. Did the server never receive it? Did it process it and the reply got lost? Is it just slow? You cannot tell from the outside.

This one uncertainty creates most of the field:

  • Partial failure. Some nodes are broken while others work. A single machine usually fails all at once.
  • Unreliable, slow networks. Messages are delayed, duplicated, reordered or dropped. See the fallacies.
  • No shared clock. Each node has its own clock and they disagree. See physical clocks.
  • No global state. No node can instantly see what all others currently hold. Agreement needs communication, and communication takes time.

A good rule: do not distribute until you must. A well-tuned single database server handles far more traffic than most products ever see. Distribute when a measured requirement (load, uptime, geography) demands it, and then expect to pay for it in design effort.

Nodes and messages in code

Let us build the smallest useful model of a cluster. Our model has the same shape as every simulation in this tutorial: nodes with state, a network that carries messages, and a loop that delivers them one at a time. Real networks are concurrent, but a single deterministic loop lets us see exactly what happens and repeat it.

A node has a name, a store of key/value pairs, an alive flag and a method to handle a message. The network holds a queue of messages in flight.

Python A first toy cluster

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Two of three nodes answered. From the client's point of view, node b simply never replied; the client cannot know whether b is dead, slow, or whether its reply was lost. A real client would wait for a timeout and then decide what to do, perhaps ask another node. That decision, repeated everywhere, is what the rest of the tutorial studies.

What the client can do now

Because two nodes still hold the data, the service is still available despite a crash. That is the promise of replication. The next question is what happens when copies disagree, for example if a write reached a and c but not b, and b comes back later with the old value. The lessons on replication and consistency answer it.

Goals of a distributed system

A well-designed distributed system aims at a handful of goals. They pull against each other, which is where most design decisions come from.

  • Resource sharing: many users and programs reach the same data, printers, services and compute.
  • Transparency: hide the distribution so the system looks like one machine (details below).
  • Openness: use published, standard interfaces and protocols so parts from different vendors and teams can be added or replaced.
  • Heterogeneity: cope with different hardware, operating systems, languages and networks, usually through middleware and agreed message formats.
  • Scalability: keep working well as users, data and nodes grow.
  • Performance and availability: use parallelism for speed, and replicas so a failure does not stop service.

Types of transparency

Transparency What the user or programmer does not need to know
Access Whether a resource is local or remote, or how data is represented
Location Where a resource physically lives (files.example.com instead of an IP)
Migration That a resource moved to another node
Relocation That a resource moved while in use
Replication That there are several copies
Concurrency That other users are using the same resource at the same time
Failure That a component failed and was recovered
Performance That the system reconfigured itself to keep response times steady
Scalability That the system grew or shrank

Full transparency is a goal, not a promise. The network can fail in ways local calls never do, and hiding that completely is both costly and misleading: a remote call that pretends to be a local one invites code with no timeouts. The fallacies lesson explains why. Good designs are transparent where it is cheap (naming, load balancing) and honest where it is not (timeouts, errors, stale reads).

Scalability and elasticity

Scalability has several dimensions: size (more users or data), geography (users far apart, higher latency) and administration (more organizations involved). Typical metrics are throughput (requests per second), latency percentiles, and speedup (how much faster n nodes finish a task) with efficiency (speedup divided by n). The limit is the work that cannot be parallelized, summarized by Amdahl's law:

Python Speedup and efficiency with a serial part

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Adding nodes to a task with 5% serial work stops paying off long before 256 nodes: the ceiling is 20 times. Elasticity is a related but different idea. Scalability is the ability to handle more load by adding resources; elasticity is doing so automatically and in both directions, growing during a traffic peak and shrinking afterwards to save cost. A system can be scalable without being elastic (you add servers by hand), and cloud platforms and autoscalers are what make elasticity practical.

Scalable designs decentralize: no single node may hold all the data, do all the work or see all the traffic. The next section shows how the architectural styles do this.

Architectural styles

  • Client-server: clients request, a server responds. Simple and easy to secure, but the server is a bottleneck and a single point of failure unless it is replicated.
  • Peer-to-peer: every node is both client and server, there is no central authority, and capacity grows with the number of peers (file sharing, blockchain networks). Finding data and keeping it consistent is harder.
  • Service-oriented and microservices: the system is split into independently deployed services that talk over the network; see Microservices Architecture.
  • Event-driven: components communicate through events on a broker, which decouples producers from consumers; see Queues, Pub/Sub and Streams.

Real systems mix them: a search engine, a social network or an online shop typically has client-server entry points, services behind them, an event stream between services, and replicated, partitioned storage underneath. For the system-design view of the same material, see What is system design?.

A map of the territory

The tutorial follows the problems in the order you meet them:

  1. Models and failures: what can go wrong, and what we assume about it.
  2. Time and ordering: how to order events without a shared clock.
  3. Communication: messages, retries and queues.
  4. Replication and partitioning: keeping copies in step and splitting data.
  5. Consensus: getting nodes to agree despite failures (Paxos, Raft).
  6. Building blocks and operations: locks, caches, load balancing, resilience, testing.

You do not need prior distributed-systems knowledge, but comfort with Python helps because the simulations are written in it. If you need a refresher, start with the Python tutorial.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which of these is NOT part of the working definition of a distributed system?

    Nodes in a distributed system have their own memory and clock. Sharing memory would make it one machine, not a distributed system.
  2. 2. What is the new outcome a network call has that a local function call does not?

    When a request gets no answer, the request may have been lost, processed with a lost reply, or still be running. This uncertainty is the root of many distributed-systems problems.
  3. 3. Three servers each are up 99% of the time and fail independently. About how available is the service if any one of them is enough?

    All three must be down together to lose the service, which has probability 0.01 cubed, or 0.000001. Availability is 1 minus that, 99.9999%.
  4. 4. Why is "do not distribute until you must" good advice?

    Distribution can buy scale and availability but it adds partial failure, unreliable networks and coordination work. Pay that cost only for a real requirement.

Summary

  • A distributed system is several independent nodes cooperating through messages to deliver one service.
  • We distribute for scale, availability, latency and organizational independence.
  • The price is partial failure, unreliable networks, unsynchronized clocks and no global state.
  • Replication improves availability sharply, provided failures are independent.
  • Simulations with nodes, a message queue and a deterministic loop let us study all of this safely.

Next: The Fallacies of Distributed Computing, eight false beliefs that trip up almost every newcomer.