Skip to content
Tutorials Center
My learning

Topics

Lesson 15 of 30 ยท Cassandra in Practice

Multi-datacenter setups and security

Configure Cassandra across datacenters with NetworkTopologyStrategy and LOCAL_QUORUM, then lock it down with roles, permissions, TLS and audit logging.

One of Cassandra's selling points is that a single cluster can span several datacenters: users in Europe read from European nodes, users in Asia from Asian nodes, and if a whole region goes dark the others keep serving. Two things make that work in practice. The first is telling Cassandra how many copies of the data each datacenter should hold, and choosing consistency levels that do not wait for a slow link between continents. The second is security: a database that talks across the internet needs logins, permissions and encrypted connections.

The real Cassandra cannot run on this site, so all CQL, configuration and shell commands below are read-only. The JavaScript stand-ins compute quorums and check permissions; they are models of the rules, not the server's code.

Replication per datacenter

For a single datacenter, SimpleStrategy is enough in a lab. For anything real, use NetworkTopologyStrategy and name the replica count per datacenter:

SQL
CREATE KEYSPACE shop
WITH replication = {
  'class': 'NetworkTopologyStrategy',
  'eu_west': 3,
  'us_east': 3
};

Every row is now stored on three nodes in eu_west and three in us_east: six copies in total. Within a datacenter Cassandra also tries to spread the replicas over different racks, so a failed rack takes out at most one copy.

How does Cassandra know which node sits in which datacenter and rack? That is the job of the snitch. The common choice, GossipingPropertyFileSnitch, reads a small file on each node and shares it by gossip:

PROPERTIES
# cassandra-rackdc.properties on a node in Frankfurt
dc=eu_west
rack=rack1

The names in the file must match the names used in the keyspace definition exactly (eu_west above). A typo means the keyspace asks for replicas in a datacenter that does not exist.

flowchart LR
    subgraph EU["Datacenter eu_west (RF 3)"]
        E1["Node E1"]
        E2["Node E2"]
        E3["Node E3"]
    end
    subgraph US["Datacenter us_east (RF 3)"]
        U1["Node U1"]
        U2["Node U2"]
        U3["Node U3"]
    end
    EC["EU app"] --> E1
    UC["US app"] --> U1
    E2 <-->|"replication over the WAN"| U2

Each datacenter holds a full set of replicas; apps talk to their local datacenter and replication crosses the wide-area link in the background.

Consistency across datacenters

The consistency rules from Replication and tunable consistency still apply, but the cluster now has two groups of replicas, and waiting for the remote group is slow. The multi-datacenter levels exist for this:

Level Waits for
QUORUM A majority of all replicas in all datacenters (4 of 6 here)
LOCAL_QUORUM A majority of replicas in the coordinator's datacenter (2 of 3)
EACH_QUORUM A majority in every datacenter (writes only)
LOCAL_ONE One replica in the local datacenter

With QUORUM and two equal datacenters, a request needs at least one remote answer, so it pays the cross-ocean latency every time. LOCAL_QUORUM reads and writes stay inside one datacenter and still give the R + W > N guarantee within that datacenter. The other datacenter catches up asynchronously, so a client switching regions right after a write may briefly see older data.

JS How many replicas does each level wait for, and how long?

Press Run (or Ctrl + Enter) to see the console output here.

No output.

LOCAL_QUORUM needs only two local acknowledgements. QUORUM over six replicas needs four, so even in the best case one answer must come from the far datacenter. The latencies are made-up numbers; the shape of the result is what matters.

Two practical rules follow. Use LOCAL_QUORUM (or LOCAL_ONE) as the default for latency-sensitive traffic, and configure drivers with a local datacenter so they never route to the wrong region (see Drivers, nodetool and production pitfalls). And when you add a datacenter, change the keyspace replication and then run nodetool rebuild on the new nodes to stream the existing data to them.

Authentication and roles

A fresh Cassandra install accepts any connection. Production clusters switch on authentication and authorization in cassandra.yaml:

YAML
authenticator: PasswordAuthenticator
authorizer: CassandraAuthorizer

Accounts are roles. A role can log in, own permissions, and be granted to other roles:

SQL
CREATE ROLE reporting;
GRANT SELECT ON KEYSPACE shop TO reporting;

CREATE ROLE ada WITH PASSWORD = 'use-a-long-random-secret' AND LOGIN = true;
GRANT reporting TO ada;

CREATE ROLE shop_app WITH PASSWORD = 'another-long-secret' AND LOGIN = true;
GRANT SELECT ON KEYSPACE shop TO shop_app;
GRANT MODIFY ON KEYSPACE shop TO shop_app;

Permissions such as SELECT, MODIFY, CREATE, ALTER, DROP and AUTHORIZE can be granted on ALL KEYSPACES, one keyspace, or one table. Granting a role to another role passes its permissions on, so you can build "reporting" and "writer" roles and give people the groups they need.

Three cautions for real clusters. The default superuser is cassandra with the password cassandra: create your own superuser, log in with it and disable the default. The system_auth keyspace stores the roles, so raise its replication factor (and, in multi-datacenter clusters, give it replicas in every datacenter), or a lost node can lock you out. And give each application its own low-privilege role instead of sharing one superuser.

JS Role inheritance and permission checks

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Ada can read through the reporting role she was granted, but cannot write; the application role can read and write shop but nothing else. Cassandra permissions are purely additive (there is no DENY), so the only way to take a permission away is to revoke the grant that provides it.

Encryption and auditing

Cassandra has two separate encryption settings, both in cassandra.yaml:

  • client_encryption_options protects the connection between applications and nodes (TLS).
  • server_encryption_options protects node-to-node traffic, which matters most between datacenters.
YAML
client_encryption_options:
  enabled: true
  keystore: /etc/cassandra/conf/keystore.p12
  keystore_password: "<kept in your secret store>"
  require_client_auth: false
server_encryption_options:
  internode_encryption: all
  keystore: /etc/cassandra/conf/keystore.p12
  truststore: /etc/cassandra/conf/truststore.p12

Encrypting at rest is usually done by the disk or the cloud provider. Finally, Cassandra 4.0 and later include audit logging that records who ran which statement, configurable by category, keyspace and user, so that you can answer "who changed this table?" after the fact. Audit logs grow quickly, so scope them to what you need.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A keyspace uses NetworkTopologyStrategy with 3 replicas in each of two datacenters. How many replicas does LOCAL_QUORUM wait for?

    A quorum of 3 replicas is 2, counted only in the datacenter of the coordinator. No cross-datacenter wait is needed.
  2. 2. What does the snitch do?

    Replica placement needs the topology. The snitch (for example GossipingPropertyFileSnitch) supplies datacenter and rack names.
  3. 3. Why should you change the default 'cassandra' superuser?

    Everyone knows the default credentials. Create your own superuser, then disable the default one.
  4. 4. Which pair of settings protects client connections and node-to-node traffic?

    The two encryption blocks cover the client-to-node and node-to-node channels. authenticator and authorizer handle logins and permissions.

Summary

  • Use NetworkTopologyStrategy with a replica count per datacenter; the snitch tells Cassandra where nodes live, and names must match.
  • LOCAL_QUORUM gives quorum behaviour inside one datacenter without a wide-area wait; plain QUORUM over two datacenters needs a remote answer.
  • Turn on PasswordAuthenticator and CassandraAuthorizer, manage access with roles and grants, replace the default superuser and replicate system_auth widely.
  • Use TLS for clients and between nodes, and enable audit logging where you need an accountability trail.
  • The stand-ins model quorum arithmetic and permission lookup; the CQL and configuration are not run here.

Next: Drivers, nodetool and production pitfalls.