Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 37 · Patterns and Comparisons

Cache Anti-Patterns and Security

Avoid common caching mistakes such as missing user IDs in keys, caching errors and unbounded keys, and lock down Memcached: private bind, firewall, no UDP, SASL and TLS.

Caching bugs have a nasty property: they usually look like someone else's bug. A customer sees another customer's name, a price is wrong "sometimes", a deploy breaks only for users who visited earlier. The cause is a pattern in how keys are built, how values are stored or how the server is exposed. This lesson lists the anti-patterns that cause the most trouble and then covers hardening the server itself. Demonstrations are stand-ins in Python; the real server cannot run in the live editors.

Anti-pattern 1: keys that miss part of the identity

The classic data leak: caching a personalised response under a key that does not include the user.

Python The key forgot who is asking

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Every factor that changes the output must be in the key: user or tenant, language, currency, device class, feature flags, and the version of the code or schema. If that makes the key space too large to cache efficiently, cache the shared part and add the personal part after the lookup.

Anti-pattern 2: user input straight into keys

Memcached's text protocol uses spaces and line breaks to separate commands. A key containing \r\n or a space can break the protocol or, in a vulnerable client, inject a second command. Keys also have a maximum length of 250 bytes. Treat any user-influenced value as hostile: hash it or validate it strictly.

Python Build safe keys from untrusted input

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Hashing makes the key opaque but safe and bounded. It also removes the ability to find all keys of a given user-supplied name, which is rarely needed.

Anti-pattern 3: other common mistakes

  • Caching without a TTL "because it never changes". Eventually something changes. Always set an upper bound.
  • Unbounded key sets: a key per search term, per timestamp or per random query string. They fill memory with items that are never read twice. Normalise the input and cap cardinality.
  • Caching errors and empty results forever. A transient failure cached for an hour becomes an hour-long outage. Use short TTLs for failures, or do not cache them. (Caching "not found" briefly is fine, as a defence against repeated lookups.)
  • Treating the cache as a database. Evictions and restarts are normal. If losing a key causes data loss, it does not belong only in Memcached.
  • Huge values and chatty small ones. A multi-megabyte object per request wastes bandwidth; 500 sequential gets waste round trips. Batch with multi-get.
  • Caching before measuring. If the database call takes 1 ms and the cache lookup 1 ms, you added complexity for nothing. Cache the expensive, repeated, tolerant-of-staleness work.
  • Silent failures. Swallowing every cache exception hides outages and falling hit ratios. Log errors and count them (see Hit Ratio and Metrics).
  • Different key formats in different services. Centralise key construction in one function or library.

Hardening the server

By default Memcached trusts its network completely: anyone who can connect can read, write and flush everything. The server used to listen on all interfaces and enable UDP, and that made it a headline amplification attack tool in 2018, when attackers sent small spoofed UDP requests that triggered very large responses. Modern releases disable UDP by default, but the lessons remain.

  1. Never expose it to the internet. Bind to a private address (-l 10.0.0.5) or localhost, and verify with a port scan from outside.
  2. Firewall port 11211 (TCP and UDP) with security groups or iptables/nftables, allowing only application servers.
  3. Disable UDP unless you need it (-U 0).
  4. Run as an unprivileged user (-u memcache), never as root.
  5. Authentication: Memcached supports SASL authentication (a build option, with the binary protocol) and newer releases have proxy-level and TLS options. If the data is sensitive and the network is shared, enable them.
  6. Encryption in transit: modern versions can be built with TLS (-Z with certificate options). Use it when traffic crosses untrusted links or when compliance asks for it; measure the CPU cost.
  7. Limit connections (-c) and watch listen_disabled_num so one noisy client cannot starve the rest.
  8. Keep it updated. Past versions have had memory-safety and denial-of-service bugs. Subscribe to release notes.
  9. Do not store secrets in plain form. Anyone who can read the cache can read the values. Session tokens, API keys and personal data should be encrypted or hashed before caching, or left out.

An open Memcached on a public IP is one of the most common data exposures found by scanners. If you can reach it from your laptop without a VPN, so can everybody else.

flowchart LR
  I["Internet"] -. "blocked by firewall" .-> M["memcached :11211"]
  A["App servers (private subnet)"] --> M

Only the application subnet may reach the cache; the internet cannot.

Multi-tenant caches

When several customers or teams share a cache, a missing tenant prefix leaks data between them, and one tenant's heavy usage evicts another's items. Namespace keys by tenant, consider separate pools for high-value or noisy tenants, and monitor evictions by namespace. Hit-ratio dashboards per prefix catch both problems early.

A short checklist before going live

  • Keys include every variant of the response and a version.
  • User-supplied parts of keys are validated or hashed.
  • Every set has a TTL.
  • Failures are logged and short-lived; misses are not errors.
  • The server is private, firewalled, UDP off, updated.
  • Sensitive values are encrypted or excluded.
  • Failure of the cache has been rehearsed (see Node Failure and Failover).

Check your understanding

Quick check · 4 questions

of 4 answered You scored / 4 · perfect!

  1. 1. A page shows one user's name to another user. What is the most likely cache bug?

    A personalised response was stored under a shared key. Every factor that changes the output belongs in the key.
  2. 2. Why hash or validate user input before using it in a key?

    Control characters can corrupt or inject protocol commands; keys are limited to 250 bytes.
  3. 3. Which is the most important network rule for Memcached?

    Without authentication, anyone who can connect can read, change or flush all data.
  4. 4. Why can caching an error response for an hour be harmful?

    The cache keeps returning the failure after the backend has recovered.

Summary

  • Put every output-changing factor (user, tenant, language, version) in the key.
  • Validate or hash untrusted key parts; keys are limited to 250 bytes.
  • Avoid unbounded key sets, forever-keys, cached failures and silent errors.
  • Memcached has no safe defaults for open networks: bind privately, firewall, disable UDP, update, and use SASL or TLS where needed.
  • Keep secrets out of the cache or encrypt them, and namespace multi-tenant data.

Next: Project: A Cache Cluster Simulator