Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 36 ยท Practice and Review

Interview Questions

Common distributed systems interview questions with concise answers: CAP, consensus, replication, idempotency, sharding, plus how to walk through a design prompt.

Distributed-systems interviews usually take two forms. In concept questions you explain an idea precisely and give an example ("what is a quorum?"). In design prompts you sketch a system and defend its trade-offs ("design a URL shortener / chat / job scheduler"). This lesson gives short model answers for the first kind, a repeatable method for the second, and two small runnable checks that show how to back an answer with numbers.

The answers are deliberately short: in an interview, one clear paragraph with a concrete example beats a lecture. Each links to the lesson where the topic is explained in full.

How to answer

Define it One sentence, no jargon loops.
Give an example A tiny scenario shows you understand.
Name the trade-off Every design choice costs something.
Say how it fails Mention the failure mode unprompted.

Interviewers reward reasoning about failure.

1 / 2

Fundamentals

What makes a system "distributed", and what is hard about it? Several machines that cooperate over a network to look like one system. The difficulty is partial failure and uncertainty: a node can crash, a message can be lost or delayed, and you cannot tell "slow" from "dead". There is no shared memory and no perfect shared clock. See what is a distributed system and the fallacies.

What is the difference between a fault and a failure? A fault is one component going wrong; a failure is the system as a whole no longer meeting its specification. Fault tolerance means faults do not become failures. See failure models.

Synchronous, asynchronous, partially synchronous: what is the difference? Synchronous: known bounds on message delay and processing time. Asynchronous: no bounds at all. Partially synchronous: bounds exist but are unknown or hold only eventually. Real networks are closest to partially synchronous, which is why practical consensus protocols use timeouts for liveness but never for safety.

Time and ordering

Why can't you rely on timestamps from different machines? Physical clocks drift and are only synchronized approximately (NTP can leave milliseconds to seconds of error, and clocks can jump). Two events 5 ms apart may appear in the wrong order. See clocks.

What is the happens-before relation? a โ†’ b if a and b are on one process with a first, or a is a send and b its receive, or by transitivity. Events not related either way are concurrent.

Lamport clock versus vector clock? Lamport clocks give one number per event: if a โ†’ b then L(a) < L(b), but not the reverse, so they cannot detect concurrency. Vector clocks keep one counter per process and can tell "before", "after" and "concurrent", at the cost of space proportional to the number of processes. See Lamport and vector clocks.

Communication

Can you have exactly-once delivery? Not at the network level. A sender that gets no acknowledgement cannot know whether the message arrived, so it either retries (at-least-once, possible duplicates) or does not (at-most-once, possible loss). What you can build is exactly-once processing: at-least-once delivery plus an idempotent receiver. See delivery guarantees.

What is idempotency and why does it matter? An operation is idempotent if doing it twice has the same effect as once. Retries are only safe for idempotent operations, so you attach an idempotency key and store the result per key. See idempotency and retries.

What is a retry storm and how do you avoid it? When a service slows down, clients time out and retry, multiplying load on the already struggling service. Defences: exponential backoff with jitter, retry budgets, circuit breakers and load shedding. See resilience patterns.

Replication and consistency

Leader-follower versus leaderless replication? With a leader, all writes are ordered at one node, which is simple but makes failover (and replication lag) the hard parts. Leaderless systems write to several replicas and need quorums, read repair and conflict handling, in exchange for higher write availability. See replication and multi-leader and leaderless.

Explain quorums. Why does R + W > N give consistency? With N replicas, a write must be acknowledged by W and a read must contact R. If R + W > N any read set shares at least one replica with the last write set, so it sees the newest version (when versions can be compared). It does not give linearizability by itself: sloppy quorums, concurrent writes and failed partial writes can still leak stale data. See quorums.

Linearizable versus eventual consistency? Linearizable: the system behaves as if there were one copy and every operation took effect instantly at some point between its start and end. Eventual: if writes stop, replicas converge, but reads may be stale meanwhile. There is a whole range in between (causal, read-your-writes, monotonic reads). See consistency models.

What is a CRDT? A data type designed so that replicas can be updated independently and merged with a function that is commutative, associative and idempotent, which guarantees convergence without coordination. Examples: G-Counter, OR-Set. See CRDTs.

Partitioning and transactions

Range versus hash partitioning? Range keeps neighbouring keys together (good for scans, risk of hot spots such as time-ordered keys). Hash spreads load evenly but loses range queries. Consistent hashing limits how many keys move when nodes join or leave. See sharding.

What is the problem with two-phase commit, and what is a saga? 2PC makes several nodes commit atomically, but if the coordinator crashes after participants voted yes, they are blocked holding locks until it returns. A saga splits the work into local transactions, each with a compensating action, trading isolation for availability. See distributed transactions.

CAP and consensus

State the CAP theorem correctly. When a network partition happens, a system must choose between consistency (linearizability) and availability (every non-failing node answers). It is not "pick two of three": partitions happen whether you like it or not, so the real choice is what to give up during a partition. PACELC adds that even without a partition there is a trade-off between latency and consistency. See CAP and PACELC.

What is consensus, and what does FLP say? Consensus gets nodes to agree on a single value despite crashes. FLP proves that in a fully asynchronous system no deterministic protocol can guarantee termination if even one node may crash. Practical protocols stay safe always and ensure progress when the network behaves well enough. See consensus and Paxos.

How does Raft keep a log consistent? A single elected leader (per term) appends entries and replicates them; an entry is committed once stored on a majority; a candidate cannot win unless its log is at least as up to date as a majority's, so a committed entry is never lost. See Raft.

What is split brain and how do fencing tokens help? Two nodes both believe they are leader. A paused old leader may wake up and keep writing. A fencing token is a number that grows with each leadership term; the storage rejects any request with a smaller token than one it has already seen. See leader election and locks and leases.

Reliability and performance

Why is tail latency a bigger deal in distributed systems? A request that fans out to 100 backends is as slow as its slowest call. If each call is slow 1% of the time, a fan-out of 100 hits a slow call in about 63% of requests. See observability and tail latency.

What is a cache stampede? A popular entry expires and many requests rebuild it simultaneously, flooding the database. Remedies: request coalescing, TTL jitter, early refresh. See caching.

Two runnable checks

Numbers make an answer convincing. These two snippets are the kind of quick calculation that strengthens a design discussion.

Check 1: does replication really help availability?

"If each machine is up 99% of the time, how available is a service that needs any one of 3 replicas, compared with a service that needs all 3?"

Python Availability of 'any of three' versus 'all of three'

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Needing any replica gives 99.9999% availability, a majority gives 99.97%, and needing all of them drops to about 97%, worse than a single machine. The lesson for interviews: the quorum you require decides whether replication helps or hurts.

Check 2: a unique ID generator without coordination

A classic design prompt: "generate unique, roughly time-ordered 64-bit IDs on many machines without a central counter." The usual answer packs time, machine id and a per-millisecond sequence into the number.

Python A Snowflake-style ID generator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

IDs never collide because machines have different machine ids, and a single machine never reuses a (time, sequence) pair. The trade-off to mention aloud: it trusts clocks. If a machine's clock jumps backwards, it could repeat an ID, so real implementations refuse to issue IDs until time catches up.

Walking through a design prompt

For "design X", follow the same six steps every time:

  1. Clarify requirements: what operations, how many users, read/write ratio, latency target, and what must never happen (lose a payment? show stale likes?).
  2. Estimate scale: requests per second, data size, growth. A quick calculation justifies sharding or not.
  3. Sketch the simplest design: one service, one database.
  4. Find the first bottleneck and fix it: cache reads, replicate for availability, partition for write scale.
  5. Choose consistency per operation: payments need strong consistency; a like counter does not.
  6. Walk through failures: a node dies, the network splits, a message is duplicated, a dependency is slow. Say what the user sees.

Take "design a job scheduler" as an example. Requirements: jobs run at least once, ideally once. Simplest design: one database table and one worker. Bottleneck: single worker, so add many workers and a queue with visibility timeouts. Consistency: duplicates are possible, so make jobs idempotent (see the job queue project). Failures: a worker crashes mid-job (the job reappears), a poison job (dead letters), two schedulers fire the same job (a lease or leader election so only one runs). That order of reasoning is what interviewers are looking for.

Check your understanding

Quick check ยท 3 questions

of 3 answered You scored / 3 ยท perfect!

  1. 1. A candidate says 'we use exactly-once delivery'. What is the best clarifying response?

    A lost ack leaves the sender unable to know if the message arrived. Idempotent processing of retried messages gives the effect of exactly once.
  2. 2. If each of 100 backend calls is slow 1% of the time, roughly how often is a request that calls all 100 slow?

    The chance that none is slow is 0.99^100, about 37%, so about 63% of requests hit at least one slow call.
  3. 3. What does CAP actually say?

    Partitions are not optional, so the choice is only what to sacrifice while one is happening.

Summary

  • Answer concept questions with: definition, tiny example, trade-off, failure mode.
  • Know the pairs well: Lamport vs vector clocks, at-least-once vs exactly-once processing, leader vs leaderless, 2PC vs sagas, CP vs AP.
  • Back claims with quick numbers (availability, fan-out latency, keys moved).
  • For design prompts: requirements, scale, simplest design, bottleneck, consistency per operation, failure walk-through.

Next: Distributed Systems Cheat Sheet.