Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 39 ยท Operations

Multi-Cluster Kafka: MirrorMaker 2 and Geo-Replication

Copy topics between Kafka clusters with MirrorMaker 2: active-passive and active-active layouts, topic prefixes, offset translation and failover, with runnable stand-ins.

Replication inside one cluster protects you from a broker dying. It does not protect you from losing the whole data center, and it does not put data close to users on another continent. For that you run more than one cluster and copy topics between them. Kafka's own tool for this is MirrorMaker 2 (MM2).

This lesson covers what MM2 is, the common multi-cluster layouts, and the hardest part of failover: consumer offsets that mean different things on two clusters.

Real versus stand-in. The properties files and commands below are real MirrorMaker 2 usage, shown read-only because the site cannot reach Kafka clusters. The TryIt blocks are plain JavaScript stand-ins that imitate topic naming and offset translation. They are not MirrorMaker.

What MirrorMaker 2 is

MM2 is built on Kafka Connect. It ships with Kafka and runs a set of connectors that read from a source cluster and write to a target cluster:

  • MirrorSourceConnector copies records and topic configuration.
  • MirrorCheckpointConnector translates consumer group offsets so groups can move between clusters.
  • MirrorHeartbeatConnector emits heartbeats so you can check that replication is alive and measure its delay.

It replaced the first-generation MirrorMaker, which was a plain consumer plus producer with no config sync and no offset translation.

Replication between clusters is asynchronous. A record is acknowledged on the source before it reaches the target, so after a disaster the target can be missing the last few seconds of data. That gap is your recovery point, and you should measure it.

A minimal setup

The dedicated driver reads one properties file:

PROPERTIES
clusters = primary, backup
primary.bootstrap.servers = kafka-a1:9092,kafka-a2:9092
backup.bootstrap.servers  = kafka-b1:9092,kafka-b2:9092

# direction: primary -> backup
primary->backup.enabled = true
primary->backup.topics = orders.*, payments
primary->backup.groups = .*

backup->primary.enabled = false

replication.factor = 3
sync.group.offsets.enabled = true
Terminal
bin/connect-mirror-maker.sh mm2.properties

Topics are chosen by regular expression. MM2 also copies topic configs and partition counts, and it keeps ACLs in step if you enable that.

Remote topic names

By default MM2 prefixes the source cluster's alias onto the topic name on the target. The topic orders on primary becomes primary.orders on backup.

This looks odd but solves a real problem: in a two-way setup, a record copied from A to B must never be copied back to A, or you get an endless loop. Because the copy carries the prefix, MM2 can recognize it as a remote topic and skip it. It also tells consumers where the data originated.

The default policy can be replaced with an "identity" policy that keeps names unchanged. That is convenient for simple disaster recovery, but then you must make sure yourself that no loop exists.

JS Remote topic names and loop prevention

Press Run (or Ctrl + Enter) to see the console output here.

No output.

(The stand-in's backup-local topic has no cluster prefix, so it is treated as local data. A topic literally named primary.orders created by hand would confuse the real policy too, so avoid dots that look like aliases.)

Layouts

Active-passive

Primary All writes and reads happen here.
Standby Receives a copy, idle until disaster.

Simple. The standby wastes capacity.

1 / 3

Active-passive is the simplest disaster recovery plan. Applications use the primary; MM2 copies to a standby; if the primary region is lost, applications switch to the standby.

Active-active runs applications in both regions. Each region writes its own topics, and MM2 copies them both ways, so a consumer in region B sees orders (local) and primary.orders (remote) and can subscribe to both with a pattern. The catch is conflicts: if both regions update the same entity, you need a rule such as "last timestamp wins" or partitioning ownership by region.

Hub and spoke (aggregation) copies many small edge clusters into one central cluster for analytics. Data flows one way.

An alternative to MM2 is a single stretched cluster across three nearby data centers, which gives synchronous replication but needs low latency between sites. MM2 suits regions that are far apart.

Offsets do not match

When MM2 copies a record, the record gets a new offset in the target topic. The target might have been created later, or records might have been skipped by compaction, so offset 500 on the source is not offset 500 on the target.

A consumer group that fails over cannot simply keep its committed offset. If it did, it would skip or reread data. The checkpoint connector solves this by writing, for each group and partition, a mapping from a source offset to the matching target offset into a <alias>.checkpoints.internal topic. With sync.group.offsets.enabled=true, MM2 also commits the translated offsets into the target cluster's consumer groups automatically.

Java clients can translate manually with RemoteClusterUtils.translateOffsets(...).

JS Translating offsets for failover

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Because translation picks a point at or below the committed offset, a failed-over consumer may process some records twice. Design consumers to be idempotent, as discussed in delivery semantics.

Failing over, and failing back

A disaster-recovery runbook for active-passive looks like this:

  1. Check the heartbeat topic and checkpoint lag to estimate how much data is missing on the standby.
  2. Stop writers if the primary is still partly alive, to avoid split brain.
  3. Point producers and consumers at the backup bootstrap servers.
  4. Consumers read the remote topics (primary.orders), starting from translated offsets.
  5. New writes go to local topics on the backup.

Failing back is harder than failing over. The primary now lacks the data written during the outage, so you reverse the mirroring direction, let it catch up, and only then switch applications back. Practice the whole drill before you need it.

Operating notes

  • Throughput: MM2 tasks are Connect tasks; raise tasks.max and use more partitions for more parallelism.
  • Monitoring: watch replication latency metrics and the lag of the MM2 consumers on the source.
  • Security: each cluster alias can have its own security.protocol, SASL and TLS settings, for example backup.security.protocol=SSL.
  • Transactions: MM2 copies records, not transactions. A transactional write on the source becomes plain records on the target, so exactly-once does not span clusters.
  • Compacted topics are copied with their keys, and the target compacts on its own.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does MirrorMaker 2 prefix the source cluster alias onto topic names by default?

    The prefix marks a topic as a remote copy, so the reverse flow can skip it. It also records the origin of the data.
  2. 2. Why can a consumer group not reuse its committed offset after failing over to the target cluster?

    Copied records get new offsets in the target log, so offsets must be translated using checkpoints.
  3. 3. What is true of replication between clusters with MM2?

    MM2 reads from the source after the write was acknowledged, so there is always some delay and a possible gap.
  4. 4. Which layout needs conflict rules for updates to the same entity?

    When both regions accept writes and copy each other's data, concurrent updates can conflict.

Summary

  • MirrorMaker 2 runs on Kafka Connect and copies topics, configs, and consumer group checkpoints between clusters.
  • Replication is asynchronous; measure the gap, since it is your recovery point.
  • Default naming prefixes the source alias (primary.orders), which prevents loops.
  • Common layouts: active-passive (disaster recovery), active-active (local serving, needs conflict rules), hub and spoke (aggregation).
  • Offsets differ between clusters; checkpoints translate them, and failover can reread some records, so keep consumers idempotent.
  • Rehearse both failover and failback.

Next: protect the connection between clients and brokers in Security and Quotas.