Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 42 ยท Operations, Security and Performance

Monitoring and Troubleshooting

Monitor Redis with INFO: hit rate, evictions, connected clients, memory fragmentation, replication lag, CLIENT LIST, OOM and other errors, plus Prometheus.

A Redis server that nobody watches fails in boring ways: memory creeps up until writes are refused, the hit rate of a cache quietly falls from 95% to 40%, a replica lags minutes behind, a client leak uses up all connections. None of these needs a clever tool to detect. Redis reports everything about itself through one command, INFO, and a handful of its numbers answer most operational questions.

This lesson tours INFO, names the metrics worth alerting on and what each one means, shows CLIENT LIST, and then goes through the errors you will meet in real life and what to do about each. It ends with the standard way to feed all of this into Prometheus. You have already met some of the settings behind these numbers in Memory Limits and Eviction, Replication and Performance and Latency.

Redis cannot run on this page, so redis-cli transcripts and config are read-only. The runnable TryIts are plain JavaScript stand-ins: they parse a pasted INFO text exactly as an exporter would, which is real parsing logic applied to sample data (the sample numbers are invented, not from a real server).

Five numbers to watch

Hit rate Hits divided by hits plus misses.
Evictions Keys removed to make room.
Memory Used versus maxmemory.
Clients Connected, blocked, rejected.

Plus replication lag if you have replicas.

1 / 2

What INFO tells you

INFO returns plain text, grouped into sections that start with # Name; each line is field:value. You can ask for one section (INFO memory), several, or everything (INFO everything, which also includes commandstats and latencystats).

Section What it holds
server Version, uptime, process id, config file, TCP port.
clients connected_clients, blocked_clients, maxclients.
memory used_memory, used_memory_rss, maxmemory, mem_fragmentation_ratio.
persistence Last RDB and AOF status, fork time, whether a rewrite is running.
stats total_commands_processed, instantaneous_ops_per_sec, keyspace_hits, keyspace_misses, evicted_keys, expired_keys, rejected_connections.
replication Role, replica list with offset and lag, master_repl_offset, link status.
commandstats Calls and total time per command.
errorstats Counts of each error type returned to clients.
keyspace Per database: key count, keys with expiry, average TTL.

Most values are counters that only ever grow (total_commands_processed, keyspace_hits, evicted_keys), so one reading says little: take two readings and divide the difference by the elapsed time to get a rate. Others are gauges (connected_clients, used_memory) that you read as they are. The same distinction applies in any monitoring system.

Text
127.0.0.1:6379> INFO stats
# Stats
total_connections_received:812
total_commands_processed:9941203
instantaneous_ops_per_sec:412
rejected_connections:0
expired_keys:55012
evicted_keys:0
keyspace_hits:7810442
keyspace_misses:640118

The metrics that matter

Hit rate. For a cache, keyspace_hits / (keyspace_hits + keyspace_misses) is the headline number. A drop means the cache is too small (evictions), keys expire too early, or the application asks for different keys than before. It is meaningless for data that is not a cache, such as queues and sessions.

Memory. used_memory is what Redis has allocated; compare it to maxmemory. When it reaches the limit the eviction policy kicks in, or with noeviction writes start failing. used_memory_rss is what the operating system gives the process. Their ratio is mem_fragmentation_ratio: a value around 1.0 to 1.5 is normal, a clearly higher value means lots of unused but reserved memory (fragmentation after many deletes, or a big value freed), and a value below 1 is a red flag that the OS is swapping Redis memory to disk. Alert before you reach the limit, not after (see Memory Limits and Eviction).

Evictions and expirations. evicted_keys should be zero for data you cannot lose and low and stable for a cache. A rising rate means memory pressure. expired_keys growing is normal, but a sudden spike followed by latency can be an expiry storm.

Clients. connected_clients close to maxclients (default 10,000) leads to rejected_connections and the error max number of clients reached. A steadily growing count usually means a client that does not close or pool connections. blocked_clients counts clients waiting in BLPOP, BRPOP, XREAD BLOCK and similar; a large number is normal for a worker pool, odd otherwise.

Persistence. rdb_last_bgsave_status and aof_last_write_status should read ok. rdb_last_save_time tells you how old your last snapshot is, which is your worst-case data loss for RDB-only setups (see Persistence: RDB and AOF).

Replication. On the master, each slaveN: line shows state, offset and lag (seconds). The lag in bytes is master_repl_offset minus the replica's offset. On a replica, check master_link_status:up and master_last_io_seconds_ago.

The first TryIt is a miniature exporter: it parses an INFO dump into sections and fields, then applies the rules above.

JS Parse INFO and check health

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The parser is the same idea as the Prometheus exporter below: split on # headers, split fields on the first : (some values contain more colons), and unpack the comma-separated slaveN and dbN lines. The sample is a typical unhealthy cache: memory over 90%, evictions running, a hit rate that has dropped and one replica nine seconds behind.

Rates from counters

Counters need two samples. The second TryIt takes two INFO stats readings 10 seconds apart and turns them into rates, and then reads CLIENT LIST lines.

CLIENT LIST prints one line per connection as key=value pairs. The fields you use most: id, addr (the peer), name (set with CLIENT SETNAME, which you should do in every application), age and idle (seconds), cmd (the last command), omem (memory used by the output buffer), qbuf (input buffer) and flags (N normal, M master link, S replica, b blocked).

Text
127.0.0.1:6379> CLIENT LIST
id=7 addr=10.0.0.7:51234 name=web fd=8 age=3600 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=26 omem=0 cmd=get user=default
127.0.0.1:6379> CLIENT KILL ID 7
(integer) 1

CLIENT KILL ID, CLIENT KILL ADDR and CLIENT KILL USER disconnect clients; a kill is a blunt tool, but it is the right one for a runaway job that opened thousands of connections.

JS Rates from two INFO samples, and a CLIENT LIST audit

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The recent hit rate (computed from deltas) is what you graph; the lifetime hit rate from a single reading changes very slowly and hides a sudden problem for days. The unnamed idle connection and the subscriber with a 64 MB output buffer are typical findings: Redis protects itself from the second with client-output-buffer-limit, which disconnects clients whose buffers stay too big.

Common errors and what to do

Error Meaning and usual cause What to do
OOM command not allowed when used memory > 'maxmemory' At the memory limit with noeviction (or nothing evictable). Raise maxmemory, choose an eviction policy, add TTLs, delete big keys.
MISCONF Redis is configured to save RDB snapshots, but it is currently not able to persist on disk A BGSAVE failed (disk full, permissions, fork failed) and stop-writes-on-bgsave-error is on. Fix the disk or permissions; check INFO persistence.
NOAUTH Authentication required The client did not authenticate. Configure the password or ACL user in the client.
WRONGPASS / NOPERM Bad credentials, or an ACL denies the command or key. Check ACL LOG (see Securing Redis).
WRONGTYPE Operation against a key holding the wrong kind of value You used a list command on a string, for example. Check key naming and TYPE key.
READONLY You can't write against a read only replica The client reached a replica, often after a failover. Make the client discover the master (Sentinel or Cluster).
LOADING Redis is loading the dataset in memory A starting server is still reading its RDB or AOF. Wait; add a readiness check on loading:0.
BUSY Redis is busy running a script A Lua script is running too long. Use SCRIPT KILL; make scripts shorter.
ERR max number of clients reached maxclients reached. Use connection pools, close leaks, inspect CLIENT LIST.
MOVED / CROSSSLOT / CLUSTERDOWN Cluster redirects and slot problems. Use a cluster-aware client; see Redis Cluster and Sharding.

INFO errorstats counts how many times each error prefix was returned, which turns "the app feels broken" into "we returned 4,000 OOM errors in the last minute".

A short checklist for "Redis is slow or failing":

  1. PING and redis-cli --latency from the application host: is the server reachable and quick?
  2. INFO stats rates, INFO clients, INFO memory: is it saturated, near its memory limit, or swapping?
  3. SLOWLOG GET: any slow commands (see Performance and Latency)?
  4. INFO persistence: is a fork or a failing save involved?
  5. INFO replication: is a replica resyncing and consuming memory and CPU?
  6. CLIENT LIST: a flood of connections or one client with a giant buffer?

Prometheus and dashboards

You do not have to run INFO by hand. The usual setup is a Redis exporter (for example the open-source redis_exporter by oliver006, which listens on port 9121 by default) that connects to Redis, runs INFO and similar commands, and exposes the values as Prometheus metrics, plus a Grafana dashboard. Exact metric names depend on the exporter version, so check its /metrics page, but the common ones look like this:

Text
redis_up
redis_connected_clients
redis_memory_used_bytes
redis_memory_max_bytes
redis_evicted_keys_total
redis_keyspace_hits_total
redis_keyspace_misses_total
Text
# Prometheus alert examples (read-only)
- alert: RedisDown
  expr: redis_up == 0
  for: 1m
- alert: RedisMemoryHigh
  expr: redis_memory_used_bytes / redis_memory_max_bytes > 0.85
  for: 5m
- alert: RedisHitRateLow
  expr: rate(redis_keyspace_hits_total[5m]) / (rate(redis_keyspace_hits_total[5m]) + rate(redis_keyspace_misses_total[5m])) < 0.8
  for: 15m

Notice the rate(...) in the last rule: it is the counter-to-rate conversion the TryIt just performed by hand. Managed Redis services (see Running Redis in Production) expose similar metrics through their own dashboards.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. keyspace_hits and keyspace_misses only ever grow. How do you get the current hit rate?

    Counters accumulate since startup. Differences between two samples give the rate over that interval, which is what reveals a recent drop.
  2. 2. mem_fragmentation_ratio is 0.7. What is the most likely explanation?

    The ratio is RSS divided by used_memory. Below 1 means less is resident than Redis thinks it uses, typically because pages were swapped out, which hurts latency badly.
  3. 3. A write fails with 'OOM command not allowed when used memory > maxmemory'. Which setting probably caused it?

    With noeviction, Redis refuses writes once it reaches maxmemory instead of removing keys.
  4. 4. After a failover a client gets 'READONLY You can't write against a read only replica'. What does that mean?

    Writes are only accepted by the master. Clients should rediscover the master through Sentinel or Cluster instead of using a fixed address.
  5. 5. Why should every application set a client name with CLIENT SETNAME?

    Names turn a list of IP addresses and ports into something you can attribute to an application, which speeds up finding leaks and heavy users.

Summary

  • INFO reports everything by section; counters need two samples to become rates, gauges are read directly.
  • Watch hit rate (for caches), memory against maxmemory, mem_fragmentation_ratio (below 1 means swapping), evictions, connected and blocked clients, persistence status and replication lag.
  • CLIENT LIST shows who is connected; name your clients with CLIENT SETNAME; CLIENT KILL removes runaways.
  • Learn the common errors: OOM, MISCONF, NOAUTH, WRONGTYPE, READONLY, LOADING, BUSY, max number of clients reached; INFO errorstats counts them.
  • Use an exporter (such as redis_exporter) with Prometheus and Grafana and alert on memory, hit rate, evictions, replication and availability.

Next, deploying all of this in real environments: Running Redis in Production.