Skip to content
Tutorials Center
My learning

Topics

Lesson 138 of 147 ยท Projects

Project: Key-Value Store with TCP Protocol

Build a Redis-like key-value server in Go in four stages - a thread-safe store, a text protocol over TCP with a server, a client with key expiry using timers, and JSON snapshots with many clients.

Databases like Redis, Memcached and etcd feel like magic until you build a miniature one. At the heart of each is a surprisingly small idea: a map of keys to values, held in memory, that many programs can reach over the network by sending short text commands. In this project you build exactly that. You'll write a key-value store, give it a Redis-like text protocol over TCP, write a client for it, add keys that expire, save everything to a JSON snapshot and load it again after a restart, and then hammer the server with twenty clients at once.

It is also a tour of Go's networking and concurrency in one program. You'll use the net package for sockets, bufio for reading lines, goroutines to serve many connections at once, mutexes to protect the data, timers for expiry, encoding/json for snapshots and context-style shutdown ideas for stopping cleanly. There are four stages, and each is a complete program you can run and change.

Our runner allows networking only on the loopback interface (127.0.0.1), so every program starts its own server and connects to it from the same process. Port 0 asks the operating system for any free port, which is why the examples never print a port number. In your own projects you would run the server and the clients as separate programs, and the code would be the same.

The plan

The server understands these commands:

Command Reply Meaning
PING [message] +PONG or the message Check that the server is alive
SET key value [EX seconds | PX millis] +OK Store a value, optionally with a lifetime
GET key the value, or nil Read a value
DEL key [key ...] number deleted Remove keys
INCR key, INCRBY key n the new number Atomic counter
EXPIRE key s, PEXPIRE key ms 1 or 0 Give an existing key a lifetime
TTL key, PTTL key time left, -1 (no expiry) or -2 (no such key) Ask how long a key still lives
KEYS [pattern] list of keys Find keys with *, ? and [a-z]
DBSIZE number of keys Count keys
SAVE number of keys written Write a snapshot file
QUIT +OK, then the server closes End the session
Stage What you build Main Go ideas
1 The store: a locked map with atomic INCR and pattern search sync.RWMutex, check-then-act, path.Match
2 The wire protocol and a TCP server net.Listen, bufio, goroutine per connection, pipelining, graceful Close
3 A client library, and keys that expire time.AfterFunc, item identity, Pipeline, ServerError
4 Snapshots, many clients, shutdown, idle timeouts JSON, atomic file writes, WaitGroup.Go, deadlines

Stage 1: the store

Start with the part that has nothing to do with networks. A Store wraps a map[string]string and a sync.RWMutex. Every goroutine that serves a client will call the store, so every method locks. Get takes a read lock, so many reads can proceed together; the methods that change the map take the write lock.

The interesting method is Incr. Read the current value, add a number, store the result: that is a check-then-act sequence, and it must happen under one lock, or two clients could both read 41, both add 1, and both store 42, losing an increment. The method also handles bad input without panicking: a value that is not a number gives ErrNotInteger, and an addition that would pass the largest int64 gives ErrOverflow, checked before the addition because the addition itself would silently wrap around.

Keys supports glob patterns with path.Match: * matches any run of characters, ? one character, [a-z] a range. (path.Match treats / specially, so * does not match across a slash; for keys like user:1 that makes no difference. Redis uses its own matcher.) A malformed pattern such as [ returns an error, which the server will turn into a protocol error.

Go Stage 1: a concurrency-safe in-memory store

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

The last line is the proof that the lock covers the whole read-modify-write cycle: 200 goroutines each add one a hundred times, and the total is exactly 20,000. Try removing the lock from Incr and run again: on a multi-core machine the number comes out lower, and the race detector (go run -race) reports the problem immediately.

Stage 2: a text protocol and a TCP server

Now give the store a voice. A protocol is an agreement about what bytes mean. Ours has two directions.

Requests are single lines: COMMAND arg arg ..., ending in a line break. Arguments that contain spaces are written in double quotes, with \", \\, \n, \r and \t as escapes inside quotes. That lets any string, even one with newlines, travel on one line. splitArgs parses a line into words with a small state machine, and quoteArg is its inverse for the client.

Replies follow the shape of Redis's RESP protocol: the first byte says what kind of reply it is, and every line ends with \r\n:

TEXT
+OK\r\n                      a simple string
-ERR something broke\r\n     an error
:42\r\n                      an integer
$5\r\nhello\r\n              a bulk string: its length, then exactly that many bytes
$-1\r\n                      "nothing there" (nil)
*2\r\n$1\r\na\r\n$1\r\nb\r\n an array of two bulk strings

The length prefix on bulk strings is the key design idea: the reader knows exactly how many bytes to take, so the data can contain anything, including line breaks, without being confused with the protocol. The Go types Simple, Failure, Integer, Bulk, Null and Array represent the reply kinds, writeReply encodes them and readReply decodes them.

The server has a classic shape: net.Listen opens the port, and Accept returns a new connection for each client. Every connection is served in its own goroutine, so a slow client never blocks the others. The goroutine's loop is: read a line, split it, run the command, write the reply.

Four details separate a toy server from a sound one:

  • Lines have a size limit. readLine uses bufio.Reader.ReadLine on a 64 KiB buffer, which reports a line that does not fit instead of growing without bound. A client that sends gigabytes without a newline cannot eat all memory.
  • Pipelining. A client may send many commands without waiting for answers. The server writes each reply into a bufio.Writer and flushes only when no more input is buffered (r.Buffered() == 0), so a burst of commands gets its replies in one network write.
  • Replies for bad commands. Unknown commands, wrong argument counts and unbalanced quotes get an -ERR reply, and the connection stays open. Errors in the protocol of one client must never kill the server.
  • Shutdown. Close marks the server closed, closes the listener (which makes Accept fail, ending Serve) and then wakes every connection by setting its read deadline to "now". A connection that is waiting for input fails its read and exits, while one that is running a command finishes it and writes the reply first. A sync.WaitGroup counts live connections and Close waits for it, with a two-second fallback that force-closes the stragglers. The counter is incremented in track, under the same mutex that Close uses, so a connection can never be added after Close has started waiting.

The program starts a server, connects with a plain net.Dial, and sends a script of commands. It records the exact bytes the server sends back with an io.TeeReader, so you can see the protocol itself.

Go Stage 2: the protocol and a TCP server

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Read the raw replies. PING hello returns a bulk string, "$5\r\nhello\r\n": five bytes of data. GET greeting returns $11 and the text with its space intact. GET nothing is the nil reply $-1. KEYS * is an array of three bulk strings, sorted. INCR color fails with -ERR value is not an integer, and the connection carries on: the next commands still work. After QUIT the server says +OK and closes the connection, so the next read returns io.EOF.

You can talk to this server by hand. If it were running as a real program on port 7777, nc localhost 7777 or telnet localhost 7777 would let you type SET color teal, press Enter and see +OK. Because the protocol is plain text, it is easy to debug, which is the reason so many protocols (HTTP/1.1, SMTP, Redis) are text-based.

Stage 3: a client and keys that expire

Talking to the server with raw strings gets old quickly. Stage 3 adds a client library, and the big new feature in the store: expiry.

The client

Dial connects. Do(args...) sends one command and waits for one reply, turning an -ERR reply into a Go error of type ServerError. Typed helpers (Set, Get, Incr, PTTL, Keys) check the reply type and return plain Go values. A mutex makes a Client safe for several goroutines at once, since two commands must not interleave on one connection. Pipeline writes many commands first and reads all the replies afterwards; with a real network that saves one round trip per command.

Expiry with timers

Caches and sessions need keys that disappear. There are two classic ways to expire a key, and the store uses both:

  1. Lazily, on access. Every read first asks "is this item past its deadline?". If so, it is deleted and treated as absent. This is always correct, even if nothing ever cleans up.
  2. Actively, with a timer. When a key gets a lifetime, the store calls time.AfterFunc(ttl, ...). At the deadline Go runs the function on its own goroutine, and the function deletes the item. This frees memory even for keys that nobody reads again.

The subtle part is what happens when the key changes before the timer fires. SET key v2 replaces the value, EXPIRE key 1000 moves the deadline. The old timer can still wake up and delete the wrong thing. The defenses are layered:

  • The old timer is stopped whenever an item is replaced or its deadline changes.
  • A timer that is already running (it fired just before Stop) is handled by the callback itself. expire(key, it) receives the item pointer that the timer belongs to. It deletes only if s.data[key] == it (the key still holds the same item, not a replacement) and the item is still overdue (its deadline was not moved). Comparing pointers is a cheap identity check.

Because Get can now delete, the store uses a plain sync.Mutex instead of an RWMutex: even a "read" can modify the map.

TTL returns the remaining lifetime in whole seconds, rounded up, or -1 for a key without expiry and -2 for a missing key, the same convention as Redis. PTTL returns milliseconds. SET key value EX 30 is seconds and PX 1500 milliseconds. The demonstration uses milliseconds so that it finishes quickly.

Go Stage 3: client library and expiring keys

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Go through the output line by line. A value with a newline, quotes and a backslash arrives unchanged, which tests quoteArg and splitArgs together. The session key had at most 150 ms to live; after 300 ms it is not found, and the store holds only the two remaining items (note and forever), so the timer really removed it from memory. token was set with a 100 ms lifetime and then replaced after 40 ms by a value without one: it is still there after 250 ms, so the first timer did no harm. cache had its lifetime extended by PEXPIRE and survived. Errors from the server appear as ServerError values, and the pipeline returned five replies in order, including an error in the middle (a ServerError stored as a value, so the later replies were not lost).

Deadlines are computed with time.Now().Add(ttl), and a time.Time from time.Now() carries a monotonic clock reading. Comparisons between such values ignore changes of the wall clock (daylight saving, NTP corrections), so keys cannot expire too early or too late because someone adjusted the system time. A monotonic reading cannot be saved to a file, so the snapshots in the next stage hold wall-clock times, and the restart logic compares them with time.Now().

Stage 4: snapshots, many clients, shutdown

A cache that forgets everything on restart is fine; a store you rely on is not. Stage 4 adds a snapshot: the SAVE command writes all live keys to a JSON file, and the server loads that file at startup.

Design choices:

  • The snapshot holds absolute deadlines ("expires": "2026-..."), not "seconds left". That way a key keeps its true deadline across a restart: an hour later it still expires at the same moment. Entries whose deadline has passed while the server was down are skipped when loading, and the count is reported.
  • The lock is held only to copy. Snapshot copies the live entries under the lock and returns them sorted. JSON encoding and file writing happen afterwards, without the lock, so a big save does not block clients.
  • The file is written atomically: first to dump.json.tmp, then renamed over dump.json. A crash in the middle of the write leaves the old snapshot intact.
  • A missing file is normal (first start); a damaged one is an error that names the file and wraps the JSON error with %w. The server refuses to start on a damaged snapshot instead of starting empty and then overwriting the good data on the next save.
  • SaveFunc keeps the server independent of files. The server just calls a function; main decides it saves to file.

The final program tells a whole story. It starts a server, then twenty clients, each with its own connection, add 1 to the same counter fifty times each, and store a key of their own. The total must be exactly 1,000. A client then saves a snapshot, and the program shows the end of the file with the expires times blanked out (they change every run). The server is closed while a second client is still connected and idle: that client must be disconnected and the close must return quickly, which is the shutdown design from stage 2 at work. After a short pause, during which the key short expires, a new server starts and loads the snapshot. Finally, the new server runs with an idle timeout of 250 ms and the program tests it, and a deliberately broken file shows the error message.

Go Stage 4: snapshots, 20 clients and a restart

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

All the guarantees show up in the output. hits = 1000 with no errors, although twenty connections ran in parallel; the save wrote 24 keys, the 20 client keys plus hits, language, session and short. The restart loaded 23 entries and skipped the one that had expired while the server was down. After the restart hits is still 1000, language is Go, short is gone, and session still has between 3,500 and 3,600 seconds left, nearly its original hour. The idle client got an error when the server closed, and so did the client that stayed silent for twice the idle limit.

What a real server would add

This is a teaching server; production stores add a lot more:

  • A binary protocol or RESP itself, so that existing Redis clients and tools work. Our request format is the "inline command" style of Redis, which its server also accepts.
  • A write-ahead log that records every change as it happens, so a crash loses at most the last moment, not everything since the last SAVE.
  • Authentication and TLS (crypto/tls) so that only trusted programs can connect, and encrypted traffic. Never expose a server like this to the internet.
  • Memory limits and eviction (least recently used) so the store cannot grow without bound.
  • Replication and sharding across several machines.
  • Metrics such as connections, commands per second and memory.

Ideas for extending the project

  • Add LPUSH, LRANGE or hash commands, which means values that are not strings. A Value interface with a type tag would be the natural design.
  • Run a background goroutine that saves a snapshot every minute, stopped by a context.
  • Add a SUBSCRIBE/PUBLISH pair with a channel per subscriber, as in the chat server.
  • Write a test suite for the store and for the protocol parser (see table-driven tests) and a fuzz test for splitArgs (see benchmarks and fuzzing).
  • Replace the mutex-protected map with 16 shards, each with its own lock, and compare the throughput with a benchmark.
  • Implement the full RESP request format (*3\r\n$3\r\nSET\r\n...) so that real Redis clients can talk to your server.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why must Incr read, add and write under one lock instead of calling Get and then Set (each locking separately)?

    Two separate critical sections leave a gap. The combined read-modify-write must be one critical section to be atomic.
  2. 2. What does the bulk-string reply $11\r\nhello world\r\n let the client do that a plain line could not?

    The length prefix tells the reader how much data follows. The data itself is never scanned for a terminator, so it can hold any bytes.
  3. 3. In the expiry code, why does the timer callback check s.data[key] == it?

    A later Set stores a new item under the same key. The old timer must not delete it, so the callback compares item identity (pointer equality) before deleting.
  4. 4. How does Server.Close wake up a connection that is blocked waiting for the next command?

    A read deadline in the past makes a blocked Read return a timeout error. The handler then exits, after finishing any command it was running.
  5. 5. Why does the snapshot store expires as an absolute time instead of the seconds remaining?

    Remaining time is only meaningful at the moment of saving. An absolute deadline stays correct however long the server was offline.

Summary

  • A key-value store is a map plus a lock; make each operation, especially read-modify-write operations such as INCR, one critical section.
  • A text protocol needs a framing rule. Requests here are one line with quoting for spaces; replies carry a type byte and, for data, a length prefix so that payloads can contain any bytes.
  • A TCP server accepts in a loop and serves each connection in its own goroutine. Limit line sizes, answer bad commands with errors instead of dropping connections, and flush replies only when no pipelined input remains.
  • Shut down gracefully: close the listener, wake blocked readers with a read deadline, let running commands finish, wait with a WaitGroup, and force-close after a timeout. Add idle timeouts with SetReadDeadline.
  • Expire keys both lazily (check on access) and actively (time.AfterFunc), and make timer callbacks safe against later changes by checking item identity and the current deadline.
  • Persist with absolute deadlines in JSON, copy the data under the lock and write the file outside it, write through a temporary file and os.Rename, treat a missing file as normal and a damaged file as an error.
  • Test the whole system with real clients on the loopback interface, and check totals that must hold for any interleaving, such as twenty clients adding up to exactly 1,000.

Next: Project: Real-Time Chat Server