Skip to content
Tutorials Center
My learning

Topics

Lesson 11 of 30 ยท How Cassandra Works Inside

Replication and tunable consistency

Understand replication factor and Cassandra consistency levels (ONE, QUORUM, ALL), the R+W>N rule, hinted handoff, read repair and anti-entropy repair.

Cassandra keeps several copies of every row, on purpose, so that a machine can die without losing data or taking the service down. Several copies immediately raise a question: when a client writes or reads, how many of those copies must take part? Cassandra does not decide this once for the whole database. You decide it per query, with a setting called the consistency level. That is what "tunable consistency" means, and it is the practical form of the CAP and PACELC trade-offs from the first lesson.

This lesson explains the vocabulary, proves the famous R + W > N rule by brute force in JavaScript, and then simulates a three-node cluster that loses a node, so you can see hinted handoff and read repair at work. The simulator is a model of the ideas: it has no network, no real timing and a much simpler coordinator than Cassandra's. Real cluster commands appear in read-only blocks.

Replication factor

The replication factor (RF) is how many nodes hold a copy of each partition. It is set per keyspace:

SQL
CREATE KEYSPACE shop
WITH replication = {'class': 'NetworkTopologyStrategy', 'dc1': 3};

With RF = 3 the partition's token (see Cassandra architecture) chooses one primary replica and the next two nodes around the ring hold the other copies. There is no master among them: all three replicas are equal peers. Any node in the cluster can receive a client request and act as the coordinator, forwarding the work to the replicas.

flowchart LR
    CL["Client"] --> CO["Coordinator<br/>(any node)"]
    CO --> R1["Replica 1"]
    CO --> R2["Replica 2"]
    CO --> R3["Replica 3<br/>(down)"]
    R1 -- "ack" --> CO
    R2 -- "ack" --> CO
    CO -- "2 of 3 acks = QUORUM met" --> CL

With RF = 3 and consistency level QUORUM, two acknowledgements are enough, so one dead replica does not stop the write.

Consistency levels

The consistency level (CL) says how many replicas must respond before the coordinator reports success. In cqlsh you set it with a command; drivers set it per statement.

SQL
CONSISTENCY QUORUM;
SELECT * FROM shop.orders WHERE user_id = ? AND order_day = '2026-03-03';
Level Replicas that must respond (RF = 3) Typical use
ONE 1 Fast, tolerates two failures, may read stale data
TWO, THREE 2, 3 Rarely needed
QUORUM a majority: floor(RF / 2) + 1, so 2 The common balanced choice
ALL every replica (3) Strongest, but one failed node blocks the request
LOCAL_ONE, LOCAL_QUORUM same, counted only in the local datacenter Multi-datacenter clusters, avoids cross-region latency
ANY (writes only) any node, even just a stored hint Highest availability, weakest guarantee

Two behaviors are worth memorizing. First, the coordinator always sends a write to all replicas; the consistency level only decides how many acknowledgements it waits for. Second, if fewer replicas are alive than the level demands, the coordinator fails the request immediately with an "unavailable" error instead of trying.

The R + W > N rule

Let N be the replication factor, W the number of replicas that must acknowledge a write, and R the number that must answer a read. If R + W > N, then every read set overlaps every write set in at least one replica, so a read always includes at least one copy of the latest acknowledged write, and the timestamps let the coordinator pick it. For RF = 3, writing and reading at QUORUM gives 2 + 2 = 4 > 3, so reads see the latest successful write. Writing at ONE and reading at ONE gives 1 + 1 = 2, which is not enough.

You do not have to trust the arithmetic. The program below brute-forces it: for every possible set of replicas that acknowledged the write and every possible set that a read consults, it checks whether the read set always contains an acknowledging replica.

JS Proving R + W > N by brute force

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Every row agrees with the rule. The pair ONE / ALL and the pair ALL / ONE are also safe, but each has a cost: the first makes reads fragile, the second makes writes fragile. QUORUM / QUORUM is the usual compromise because it tolerates one failed replica for both operations when RF = 3.

Repairing divergence

Replicas will sometimes disagree: a node was down, a message was lost. Cassandra has three layers to bring them back in step.

  1. Hinted handoff. When a replica is down, the coordinator stores a small hint (the missed write) and replays it once the replica returns. Hints are kept only for a limited window (three hours by default), after which the node needs a proper repair.
  2. Read repair. When a read consults several replicas and notices that one returned an older version, the coordinator sends the newest version to the stale replica. Reading fixes what it touches.
  3. Anti-entropy repair. The command nodetool repair compares replicas using hash trees (Merkle trees) and streams the differences. It is the only way to fix data that is never read, and it must run regularly, at least once within every gc_grace_seconds period, for a reason you will see in Deletes, TTL and tombstones.
Terminal
nodetool repair -pr shop

The -pr flag repairs only the node's primary ranges, so running it on every node covers the whole ring once.

Stand-in: a three-node cluster that loses a node

The simulator keeps a clock for write timestamps, stores hints for nodes that are down, answers reads by asking a chosen number of live replicas in a preferred order, and applies read repair. A read picks the replicas in the order you give, so you can see exactly which ones answered.

JS Hints and read repair in a 3-node cluster

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Study the second scenario. While n3 was down, a QUORUM write still succeeded with two acknowledgements, and the ALL write was refused up front, which is why the price stayed 12 and never became 13. When n3 returned without its hint, a ONE read from it served the stale 10. A QUORUM read, which included an up-to-date node, returned 12 and repaired n3 as a side effect, so the next ONE read from n3 was correct. In the first scenario, replaying the hint did the same job before any read happened.

Choosing levels in practice

  • Default to LOCAL_QUORUM for reads and writes in production: strong enough for most applications and tolerant of one failed replica.
  • Use ONE or LOCAL_ONE for data where a stale read is harmless, such as view counters or caches.
  • Avoid ALL: a single slow or failed node then affects every request.
  • Remember that a higher level costs latency, because the coordinator waits for the slowest of the required replicas.
  • QUORUM does not give you transactions. Two clients writing the same row at the same time still resolve by timestamp. For compare-and-set you need lightweight transactions.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. With RF = 3, how many replicas must respond for a QUORUM operation?

    QUORUM is a majority of the replicas: floor(3 / 2) + 1 = 2.
  2. 2. Which combination guarantees that a read sees the latest acknowledged write when RF = 3?

    The read and write sets must overlap: R + W > N. Only QUORUM / QUORUM (2 + 2 > 3) of these satisfies it; ONE / TWO sums to exactly 3.
  3. 3. A replica was down for 20 minutes. Which mechanism usually catches it up first?

    The coordinator stored hints for the missed writes and replays them when the node returns. Read repair and nodetool repair handle what hints miss.
  4. 4. Why must nodetool repair run regularly?

    Anti-entropy repair is the only mechanism that reconciles data nobody reads, and it must complete within gc_grace_seconds to keep deletes safe.
  5. 5. In the simulator, why was the ALL write refused while n3 was down?

    If fewer replicas are alive than the level requires, the coordinator reports unavailable immediately and writes nothing.

Summary

  • The replication factor sets how many copies exist; the consistency level, chosen per query, sets how many must respond.
  • QUORUM is a majority. R + W > N guarantees a read overlaps the latest acknowledged write.
  • Higher levels trade latency and availability for stronger guarantees; LOCAL_QUORUM is a common production default.
  • Hinted handoff, read repair and nodetool repair bring diverged replicas back together, each covering a different gap.
  • The simulator models the ideas with a simplified coordinator; real Cassandra adds timeouts, digests, speculative retries and more.

Next: Deletes, TTL and tombstones.