Skip to content
Tutorials Center
My learning

Topics

Lesson 26 of 35 ยท Replication and Clusters

Replication and ClickHouse Keeper

Learn how ClickHouse replicates data with ReplicatedMergeTree and ClickHouse Keeper: replication log, macros, adding a replica, quorum inserts and read-after-write settings.

One server is a single point of failure. Replication keeps several copies of the same table on different servers, so a machine can fail without losing data or availability, and reads can be spread across copies. In ClickHouse, replication is a property of the table engine, not of the server: only tables using a Replicated*MergeTree engine are replicated, and you choose per table.

ClickHouse cannot run on this site, so the cluster snippets are read-only. A SQL stand-in shows the replication-log idea. If you have not yet met parts and merges, review the sparse index lesson first.

The moving pieces

flowchart TB
    K["ClickHouse Keeper (ZooKeeper-compatible, 3 nodes)"]
    R1["Replica 1"] <--> K
    R2["Replica 2"] <--> K
    R1 <-->|"fetch parts"| R2
    C["Client INSERT"] --> R1

Keeper stores only metadata and the replication log. The data itself travels directly between replicas.

  • ClickHouse Keeper is a coordination service speaking the ZooKeeper protocol, written in C++ and shipped with ClickHouse (it can run inside the server process or standalone). It stores the table's metadata, a replication log of operations and the list of parts per replica. Run three (or five) Keeper nodes so a majority survives a failure; an even number adds nothing.
  • Replicas are ordinary ClickHouse servers that hold a copy of the table and subscribe to the log.
  • Parts, not rows, are what get replicated. An insert creates a part on one replica, records "part X exists" in Keeper, and other replicas fetch that part over HTTP. Merges are replicated as instructions: a replica that is told to merge parts A and B either does the merge itself or downloads the result.

Replication is asynchronous and multi-master: any replica accepts inserts, and they converge. Inserts are acknowledged after the data is written to the replica that received them and the part is registered in Keeper (unless you ask for a quorum).

Creating a replicated table

SQL
CREATE TABLE events ON CLUSTER main
(
    ts       DateTime,
    site     LowCardinality(String),
    user_id  UInt64
)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/events', '{replica}')
PARTITION BY toYYYYMM(ts)
ORDER BY (site, ts);

The two arguments are the Keeper path (identifies the table; all replicas of one shard share it) and the replica name (unique per server). They come from macros defined in each server's configuration, such as and , so the same CREATE TABLE text works on every node.

On ClickHouse Cloud and in newer open-source setups using the Replicated database engine or SharedMergeTree, you can omit the arguments: the defaults (ReplicatedMergeTree with no parameters) fill in path and name from server settings. The engine family includes ReplicatedReplacingMergeTree, ReplicatedSummingMergeTree, ReplicatedAggregatingMergeTree and so on: just prefix Replicated.

ON CLUSTER main runs the DDL on every host of the cluster main (defined in remote_servers config), through a queue in Keeper, so you write the statement once.

What replicates and what does not

Replicated Not replicated
inserts (as parts) SELECT queries
merges, mutations (ALTER ... UPDATE/DELETE) Settings and users (unless stored in Keeper)
ALTER of columns, TTL, partitions ON CLUSTER DDL for databases/users: you must use it explicitly
DROP PARTITION Non-replicated engines such as plain MergeTree or Log

Adding or recovering a replica

To add a replica, create the same ReplicatedMergeTree table on the new server with a new replica name. It reads the log, sees all existing parts and downloads them from other replicas. No manual copy is needed. If a replica's data is damaged, you can detach the table or use SYSTEM RESTORE REPLICA when its Keeper metadata was lost; for the whole story see the operations lesson.

Useful system tables: system.replicas (queue size, absolute delay, read-only state, whether the session to Keeper is alive), system.replication_queue (pending tasks) and system.zookeeper (browse Keeper). readonly = 1 on a replica usually means it lost its Keeper connection.

SQL
SELECT database, table, is_readonly, absolute_delay, queue_size, active_replicas, total_replicas
FROM system.replicas;

Consistency knobs

Because replication is asynchronous, a query on replica 2 immediately after an insert into replica 1 may not see the new part yet. Options:

  • insert_quorum = 2 makes an insert wait until two replicas have the part (a failed quorum raises an error, and the data may still land on one).
  • select_sequential_consistency = 1 makes SELECT read only data that a quorum already confirmed.
  • SYSTEM SYNC REPLICA table makes a replica catch up with its queue before you read.
  • Route a session to one replica (for example with a load balancer using "first or random") for read-your-writes in a simple way.

These trade latency and availability for consistency. Most analytics workloads accept eventual consistency.

Stand-in: a replication log

This is a model, not ClickHouse: a log table records which parts exist, and each replica applies the log entries it has not yet applied. It shows why two replicas converge and how lag looks.

SQL Stand-in: replicas catching up from a shared log

Press Run (or Ctrl + Enter) to see the result here.

replica_2 is two entries behind, so it shows fewer rows until it fetches the missing parts. That is the "queue" in system.replicas.

SQL Stand-in: a new replica downloads everything it is missing

Press Run (or Ctrl + Enter) to see the result here.

Common mistakes

  • Using a plain MergeTree on a cluster and assuming data is copied.
  • Running Keeper on two nodes (it loses quorum when one fails) or on the same disk as heavy ClickHouse I/O.
  • Giving two servers the same replica name, or different Keeper paths for what should be one table.
  • Forgetting ON CLUSTER, creating the table on one host only.
  • Expecting read-after-write across replicas without a quorum or sync setting.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does ClickHouse Keeper store?

    Parts move directly between replicas; Keeper holds coordination data, the log and part lists.
  2. 2. How many Keeper nodes should a production ensemble have?

    A majority must stay alive. Three nodes tolerate one failure, five tolerate two; an even count adds no tolerance.
  3. 3. A SELECT on replica B misses rows just inserted on replica A. Which is a valid fix?

    Replication is asynchronous; quorum inserts, sequential consistency or syncing the replica give stronger guarantees.
  4. 4. What is replicated between replicas?

    An insert creates a part that other replicas fetch; merges and mutations are replicated as log entries.

Summary

  • Replication is per table via Replicated*MergeTree; it is asynchronous and multi-master.
  • Keeper stores the log and metadata; replicas fetch parts directly from each other.
  • Macros and ON CLUSTER keep DDL identical across nodes.
  • Monitor with system.replicas and system.replication_queue; tune consistency with quorum settings.

Next: Sharding and Distributed Tables