Skip to content
Tutorials Center
My learning

Topics

Lesson 22 of 37 ยท Distributed Caching

Client-Side Sharding and Server Pools

See how Memcached clients shard keys across a pool, batch multi-gets per server, and how proxies like mcrouter and twemproxy move that logic out of the app.

Memcached is deliberately simple: a server holds some items and answers get, set and a few other commands. It has no cluster mode. All "distribution" is done by a client, which holds a list of servers (a pool), hashes each key and sends the command to the right one. This is called client-side sharding.

In this lesson you will build a small sharding client in Python on top of fake in-memory servers, see why multi-get needs special handling, and learn when it makes sense to move sharding into a proxy. The Memcached servers themselves cannot run inside the live editors, so a dictionary stands in for each one.

What a sharded client does

For every command that names a key, the client:

  1. Computes the placement (consistent hashing, as in Consistent Hashing and the Hash Ring).
  2. Opens or reuses a connection to that server.
  3. Sends the command and returns the reply.

Because servers never talk to each other, a pool of ten servers behaves like ten independent caches that happen to share a key namespace.

Python A sharded cache client over fake servers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Multi-get and the fan-out problem

A page often needs many items at once: ten product records, twenty user names. Memcached has a multi-get command that asks for several keys in one round trip (get key1 key2 key3). In a pool, however, those keys are spread over several servers, so the client has to group keys by server, send one multi-get per server, and merge the replies. Good clients do this in parallel.

Python Group keys by server for multi-get

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The saving is large, but it also shows a subtle cost: with more servers, a single multi-get touches more of them, so one slow server can delay the whole page. This is why very wide pools make tail latency worse (see Hit Ratio and Metrics for how to watch it).

Connection pools and keep-alive

Opening a TCP connection per command is wasteful. Real clients keep persistent connections to each server and reuse them. If you run many app processes, remember that each process holds connections to every server: 200 PHP workers times 10 servers means 2,000 connections. Memcached's connection limit (default 1024) is a classic cause of mysterious errors, so check the -c setting and the connection counters.

Moving sharding into a proxy

Client-side sharding has downsides: every language needs a good client, configuration changes need a coordinated rollout, and connections multiply. A proxy speaks the Memcached protocol to your app and does the sharding itself:

  • twemproxy (nutcracker) shards across a pool and reduces connection counts.
  • mcrouter adds replication, failover, warm-up of fresh nodes, and flexible routing prefixes (for example send keys that start with session: to a separate pool).
  • Some managed services present a single endpoint and do the placement behind it.
flowchart LR
  A1["App server 1"] --> P["Proxy"]
  A2["App server 2"] --> P
  P --> M1["memcached 1"]
  P --> M2["memcached 2"]
  P --> M3["memcached 3"]

With a proxy, apps keep one connection and one address; the proxy owns the server list and routing rules.

The price is an extra network hop and one more component to run and monitor. Many teams start with client-side hashing and add a proxy once the fleet grows.

mcrouter and twemproxy are real tools, but they are not available in the live editors here. The code on this page is a stand-in that shows the placement logic, not their implementation.

Namespacing keys

All apps in a pool share one flat key space. Prefix keys to avoid collisions: shop:v3:product:42 rather than product:42. Include a version in the prefix so you can invalidate a whole family of keys by bumping the number, a technique covered in the invalidation chapter. Remember that Memcached keys are at most 250 bytes and cannot contain spaces or control characters; hash long keys first.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Who decides which Memcached server stores a given key?

    Memcached servers are independent. The client's placement algorithm picks the server.
  2. 2. Why does a multi-get over a sharded pool need grouping?

    A single server only knows its own keys, so the client splits the key list by owner.
  3. 3. What problem does a proxy such as mcrouter or twemproxy help with?

    The proxy owns the pool configuration, so apps need one connection and routing changes do not require redeploying every client.
  4. 4. What is a good way to avoid key collisions between apps sharing a pool?

    Prefixes separate namespaces, and a version in the prefix also lets you invalidate a whole family at once.

Summary

  • Memcached has no clustering; the client shards keys over a pool of independent servers.
  • Multi-get must group keys by server; wide pools raise tail latency.
  • Persistent connections matter, and many app processes multiply connection counts.
  • Proxies (twemproxy, mcrouter) move routing out of the app at the cost of an extra hop.
  • Prefix and version your keys, and keep them short.

Next: Hot Keys and Replication