Skip to content
Tutorials Center
My learning

Topics

Lesson 36 of 39 ยท Projects and Practice

Project: A Key-Value Store Service

Design and build a gRPC key-value store: proto contract, versioned values, compare-and-swap, TTL, watch streams, auth and errors, with a fully runnable Go implementation of the logic.

In this project you design a small but realistic gRPC service: a key-value store with versioned values, optimistic concurrency (compare-and-swap), expiring keys and a watch stream that tells clients when keys change. Along the way you use the ideas from this chapter group: status codes, metadata and auth, interceptors and schema evolution.

The real gRPC server and generated code cannot run on this site, so the .proto and wiring code are read-only. The store itself, the part with all the interesting logic, is plain Go and runs here. In a real project the handlers would be thin wrappers that call these same methods.

Step 1: the contract

Start with the .proto. Think about evolution from the first line: versioned package, request and response messages for every RPC, an UNSPECIFIED zero value for enums.

PROTO
syntax = "proto3";

package kv.v1;

import "google/protobuf/timestamp.proto";
import "google/protobuf/duration.proto";

option go_package = "example.com/kv/gen/kv/v1;kvv1";

service KeyValueService {
  rpc Get(GetRequest) returns (GetResponse);
  rpc Put(PutRequest) returns (PutResponse);
  rpc Delete(DeleteRequest) returns (DeleteResponse);
  rpc Watch(WatchRequest) returns (stream WatchResponse);   // server streaming
}

message Entry {
  string key = 1;
  bytes value = 2;
  int64 version = 3;                      // increases on every change
  google.protobuf.Timestamp updated_at = 4;
}

message GetRequest  { string key = 1; }
message GetResponse { Entry entry = 1; }

message PutRequest {
  string key = 1;
  bytes value = 2;
  // 0 = write unconditionally; N = only if the stored version is N.
  // Use -1 to mean "only if the key does not exist".
  int64 expected_version = 3;
  google.protobuf.Duration ttl = 4;       // optional expiry
}
message PutResponse { Entry entry = 1; }

message DeleteRequest { string key = 1; int64 expected_version = 2; }
message DeleteResponse {}

message WatchRequest { string prefix = 1; }
message WatchResponse {
  enum EventType {
    EVENT_TYPE_UNSPECIFIED = 0;
    EVENT_TYPE_PUT = 1;
    EVENT_TYPE_DELETE = 2;
    EVENT_TYPE_EXPIRE = 3;
  }
  EventType type = 1;
  Entry entry = 2;
}

Design notes: every RPC has its own request and response, so you can add fields later; version enables optimistic concurrency; the optional ttl and expected_version fields are additions that old clients simply do not set.

Step 2: error design

Decide the status for each failure before writing code:

Situation Status
Empty key or key too long INVALID_ARGUMENT
Key missing on Get NOT_FOUND
expected_version does not match ABORTED (conflict: re-read and retry)
expected_version = -1 but key exists ALREADY_EXISTS
Value too large RESOURCE_EXHAUSTED
No/invalid token UNAUTHENTICATED

Step 3: the store (runnable)

This program contains the complete logic: versioned put with compare-and-swap, get with expiry, delete, and a watch that fans events out to subscribers by prefix. A fake clock makes expiry deterministic. The final main is a scenario test.

Go The key-value store core

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Study what the scenario proves: compare-and-swap catches a lost update (ABORTED), expected = -1 implements "create only", expired keys disappear and generate an EXPIRE event, a Watch("config/") sees only matching keys, and a slow watcher is dropped rather than blocking all writers.

Step 4: wiring it to gRPC (read-only)

Each handler converts between protobuf messages and the store, and maps store errors to real status codes:

Go
type server struct {
    kvv1.UnimplementedKeyValueServiceServer // forward compatibility: new RPCs return UNIMPLEMENTED
    store *Store
}

func (s *server) Put(ctx context.Context, req *kvv1.PutRequest) (*kvv1.PutResponse, error) {
    e, err := s.store.Put(req.GetKey(), string(req.GetValue()), req.GetExpectedVersion(), req.GetTtl().AsDuration())
    if err != nil {
        return nil, toStatus(err) // StatusError -> status.Error(codes.X, msg)
    }
    return &kvv1.PutResponse{Entry: toProto(e)}, nil
}

func (s *server) Watch(req *kvv1.WatchRequest, stream kvv1.KeyValueService_WatchServer) error {
    events, stop := s.store.Watch(req.GetPrefix())
    defer stop()
    for {
        select {
        case <-stream.Context().Done():
            return stream.Context().Err() // client left or deadline passed
        case ev := <-events:
            if err := stream.Send(toWatchResponse(ev)); err != nil {
                return err
            }
        }
    }
}

Embedding UnimplementedKeyValueServiceServer is required in current generated code and lets you add RPCs to the proto without breaking existing servers. The Watch handler shows the key streaming pattern: loop until the stream's context ends, and always stop watching with defer.

Then the server main assembles the pieces from earlier lessons: TLS credentials, a chain with recovery, logging, auth interceptors, health and reflection services, and graceful shutdown on SIGTERM:

Go
srv := grpc.NewServer(
    grpc.Creds(creds),
    grpc.ChainUnaryInterceptor(recovery, logging, auth),
    grpc.ChainStreamInterceptor(streamRecovery, streamLogging, streamAuth),
)
kvv1.RegisterKeyValueServiceServer(srv, &server{store: NewStore(time.Now)})
healthpb.RegisterHealthServer(srv, health.NewServer())
reflection.Register(srv) // development only

Step 5: stretch goals

  1. Add List(prefix, page_size, page_token) with pagination: return next_page_token so the contract can grow.
  2. Persist to disk with an append-only log and replay it at startup.
  3. Add a Watch resume token (from_version) so clients can reconnect without missing events.
  4. Add per-user key namespaces using the authenticated identity from the auth interceptor.
  5. Write the tests: handler unit tests plus an in-memory bufconn integration test (see testing gRPC services).
  6. Run buf lint and buf breaking in CI (Buf lesson).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which status does a failed compare-and-swap (stale expected_version) return in this design?

    ABORTED signals a concurrency conflict where the caller should re-read and retry the whole operation.
  2. 2. Why does every RPC get its own request and response message?

    Dedicated messages leave room for compatible evolution.
  3. 3. What should the Watch handler do when the client disconnects?

    The stream context is cancelled when the client goes away; the handler must return to free resources.
  4. 4. Why does the store drop events for a slow watcher instead of blocking?

    Blocking under the store lock would let one slow client freeze all writes. A real system would let the watcher resume from a version.

Summary

  • Start from the contract: versioned package, per-RPC messages, UNSPECIFIED enum zeros, optional fields for compatible growth.
  • Map failures to precise status codes before coding.
  • Optimistic concurrency with a version number gives safe updates without locks held across calls.
  • Watch is a server stream: loop on stream.Context().Done() and the event channel.
  • Keep the core logic transport-agnostic so it is easy to test; handlers stay thin.

Next: Project: A Chat Service with Bidirectional Streaming