Skip to content
Tutorials Center
My learning

Topics

Lesson 16 of 30 ยท Cassandra in Practice

Drivers, nodetool and production pitfalls

Connect to Cassandra with the Python and Java drivers, use prepared statements and token-aware routing, read nodetool output and avoid common anti-patterns.

Knowing CQL is half the job. The other half is how your application talks to the cluster and how you keep the cluster healthy once it is running. A well-configured driver sends each request straight to a node that owns the data, reuses prepared statements and survives a node failure without errors. A well-run cluster is watched with nodetool, repaired on schedule, backed up and kept free of a handful of well-known anti-patterns.

Cassandra cannot run on this site, so the driver code, nodetool commands and CQL below are read-only; they show the real calls but were not executed here. The Node stand-ins at the end of each section model the ideas (routing, statement caching, reading a status table) in a single file with no network.

Connecting with a driver

Cassandra drivers keep a pool of connections to every node, learn the cluster layout from the nodes themselves and handle retries and failover. You give them a few contact points, not the whole list. A minimal Python program with the DataStax cassandra-driver package:

Python
from cassandra.cluster import Cluster, ExecutionProfile, EXEC_PROFILE_DEFAULT
from cassandra.policies import DCAwareRoundRobinPolicy, TokenAwarePolicy
from cassandra import ConsistencyLevel

profile = ExecutionProfile(
    load_balancing_policy=TokenAwarePolicy(DCAwareRoundRobinPolicy(local_dc="eu_west")),
    consistency_level=ConsistencyLevel.LOCAL_QUORUM,
)
cluster = Cluster(["10.0.0.11", "10.0.0.12"], execution_profiles={EXEC_PROFILE_DEFAULT: profile})
session = cluster.connect("shop")

insert = session.prepare("INSERT INTO orders_by_user (user_id, order_id, total) VALUES (?, ?, ?)")
session.execute(insert, (user_id, order_id, 19.90))

rows = session.execute(
    session.prepare("SELECT order_id, total FROM orders_by_user WHERE user_id = ?"), (user_id,)
)

The same idea in Java with the 4.x driver:

Java
try (CqlSession session = CqlSession.builder()
        .addContactPoint(new InetSocketAddress("10.0.0.11", 9042))
        .withLocalDatacenter("eu_west")
        .withKeyspace("shop")
        .build()) {

    PreparedStatement insert = session.prepare(
        "INSERT INTO orders_by_user (user_id, order_id, total) VALUES (?, ?, ?)");
    session.execute(insert.bind(userId, orderId, new BigDecimal("19.90")));
}

Both programs do the same three things that matter: they name the local datacenter, they set a default consistency level (see Multi-datacenter setups and security), and they prepare statements. (The Java 4.x driver is token-aware by default, so no policy is configured there.)

Prepared statements

A prepared statement is parsed once by the cluster. After that the driver sends only the statement id and the bound values. That saves parsing and lets the driver compute the partition key from the values you bind, which is what makes token-aware routing possible. Prepare each statement once at start-up and reuse it. Building a new CQL string with the values pasted in for every request is slower and, if the text comes from user input, open to injection.

Node.js Why prepare once: counting parses

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The prepared path parses once. In the real driver the statement id and the bound values travel on the wire, and the server keeps the parsed form in a cache.

Token-aware routing

Any node can coordinate any request, but if the coordinator is not one of the replicas it must forward the request, which costs a network hop. A token-aware policy hashes the partition key on the client, finds the replicas and sends the request to one of them (preferring the local datacenter).

flowchart LR
    subgraph NA["Without token awareness"]
        A1["Client"] --> A2["Any node"] --> A3["Replica"]
    end
    subgraph TA["With token awareness"]
        B1["Client"] --> B3["Replica (coordinator)"]
    end

Token-aware routing removes one hop for most requests because the coordinator is already a replica.

Node.js Hops with round-robin and token-aware routing

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Round-robin lands on a non-replica about half the time (a coordinator is a replica with probability 3/6), while token-aware routing never needs the extra hop. The real hash is Murmur3 and the real ring is richer, but the effect is the same.

Watching the cluster with nodetool

nodetool is the command-line tool for inspecting and operating a node. The ones you will use most:

Command Use
nodetool status Which nodes are up, how much data each owns
nodetool tpstats Thread-pool queues; pending or blocked tasks show overload
nodetool tablestats shop.orders_by_user Per-table sizes, partition sizes, tombstones, latencies
nodetool compactionstats Running compactions
nodetool repair -pr shop Anti-entropy repair of the ranges this node owns as primary
nodetool snapshot -t before_upgrade shop Hard-link backup of SSTables
nodetool drain Flush and stop accepting writes before a restart

A healthy nodetool status looks like this (illustrative output):

TEXT
Datacenter: eu_west
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address      Load       Tokens  Owns   Host ID    Rack
UN  10.0.0.11    251.3 GiB  16      33.4%  3f1c...    rack1
UN  10.0.0.12    248.9 GiB  16      33.3%  a92b...    rack1
DN  10.0.0.13    0 bytes    16      33.3%  c07e...    rack1

The first two letters are the key: U/D for up or down, N/L/J/M for normal, leaving, joining, moving. UN is what you want; the DN line is a node that is down and needs attention.

Node.js Reading a nodetool status table

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Real monitoring uses JMX metrics exported to Prometheus or a similar system rather than parsing text, but watching the same signals (down nodes, pending compactions, dropped messages, read and write latency percentiles, partition sizes, tombstone counts) is the heart of it.

Repair and backups

Replicas drift apart whenever a node misses writes. Hinted handoff and read repair fix some of that; anti-entropy repair (nodetool repair) compares data between replicas and streams the differences. Run it regularly on every node, and at least once within gc_grace_seconds so that deleted data does not come back (see Deletes, TTL and tombstones). Many teams automate it with a repair scheduler.

Backups are SSTable-level: nodetool snapshot creates hard links to the immutable files, which is nearly instant and takes no extra space at first. Copy snapshots off the node, keep the schema (DESCRIBE KEYSPACE) next to them, and test a restore; an untested backup is a guess.

Common anti-patterns

  • Unbounded partitions: a partition key that grows forever (see Time-series and bucketing patterns).
  • Using Cassandra as a queue: insert-then-delete creates tombstones that slow every read of that partition.
  • SELECT count(*) or full scans on big tables, and ALLOW FILTERING (Secondary indexes, SAI and materialized views).
  • Large multi-partition batches used as bulk loaders.
  • Unbounded collections and very large cells.
  • Treating it as a relational database: joins, ad hoc queries and many secondary indexes belong elsewhere.
  • Ignoring repair, or running with different time on nodes (keep clocks synchronized, because last-write-wins relies on timestamps).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the benefit of preparing a statement once and reusing it?

    Prepared statements avoid repeated parsing and enable token-aware routing. They also keep values out of the CQL text.
  2. 2. What does a token-aware load-balancing policy do?

    The driver hashes the partition key, so the coordinator is already one of the replicas for most requests.
  3. 3. In nodetool status, what does the code 'DN' mean?

    The first letter is up or down, the second is the state. A DN node needs investigation.
  4. 4. Why should repair run at least once within gc_grace_seconds?

    If a replica missed a delete and the tombstone is purged elsewhere, the old value can be copied back. Repair spreads the tombstone first.

Summary

  • Drivers connect to a few contact points, learn the rest of the ring, and need the local datacenter and a default consistency level.
  • Prepare statements once; token-aware routing then sends requests straight to replicas.
  • nodetool status, tpstats, tablestats and compactionstats are the first tools to reach for; scrape JMX metrics for real monitoring.
  • Repair regularly, snapshot and copy backups off the node, and test restores.
  • Avoid unbounded partitions, queue-like tables, big batches, scans and unsynchronized clocks.
  • The stand-ins model routing and parsing; the driver code and commands are not run here.

Next: Cypher: MATCH, CREATE and patterns.