Skip to content
Tutorials Center
My learning

Topics

Lesson 136 of 147 ยท Projects

Project: URL Shortener Service

Build a URL shortener in Go in four stages - JSON handlers and base-62 codes, custom codes with stats and middleware, a real server with graceful shutdown, and httptest-based tests.

A URL shortener turns https://go.dev/doc/effective_go into something like sho.rt/1000. When someone opens the short address, the service looks up the long one and sends the browser there with an HTTP redirect. The idea is simple, which makes it an excellent project for learning how Go web services are built: a few routes, a store shared by many requests, input that must be validated because it comes from strangers, and behavior around the edges (wrong methods, missing links, crashes, shutdown) that separates a toy from a service.

You'll build it in four stages, and each stage is a complete program. In stage 1 you write the store, a base-62 code generator and two handlers, and you call them with net/http/httptest instead of a network. In stage 2 you add custom codes, click statistics, deletion and a chain of middleware (request ids, access logging, panic recovery). In stage 3 you run the service on a real loopback port and shut it down gracefully while a request is still being handled. In stage 4 you write a table-driven test suite with httptest.

You'll use HTTP basics, the net/http routing patterns of Go 1.22, JSON, the sync package, context and testing.

The first line of every program, //go:debug httpmuxgo121=0, switches on the Go 1.22 router patterns such as GET /{code}. You need it only because our runner compiles a single file without a go.mod; in a real module whose go.mod says go 1.22 or newer, the patterns work without it. See what's new in Go for GODEBUG settings.

The plan

The service speaks four small routes:

Method and path What it does Success Errors
POST /api/links Create a short link from {"url": "...", "code": "optional"} 201 with the code 400 bad input, 409 code taken
GET /{code} Redirect to the long URL 302 and a Location header 404 unknown code
GET /api/links/{code} Show the link and its hit count 200 404
DELETE /api/links/{code} Remove a link 204 404
Stage What you build Main Go ideas
1 Store, base-62 codes, create and redirect handlers http.ServeMux patterns, PathValue, httptest, RWMutex
2 Custom codes, stats, delete, middleware Function types, context.WithValue, atomic.Int64, response wrappers, recover
3 A real server, graceful shutdown net.Listen, http.Server, Shutdown, http.Client
4 A test suite Table-driven tests, httptest.NewRecorder, concurrency test

Stage 1: store, codes and two handlers

The store is a map from code to URL, guarded by a sync.RWMutex. Handlers run concurrently, one goroutine per request, so a plain map would be a data race. A read-write lock lets many redirects read at once and makes the rare create wait for exclusive access. See the sync package.

Generating codes. A common trick is to count: link number 1, 2, 3, and so on, and write each number in base 62, using the digits 0-9, the letters a-z and the letters A-Z. Base 62 is compact (a six-digit code covers 56 billion links) and URL-safe. encode62 repeatedly takes n % 62 as the next digit and divides by 62, which produces the digits from least to most significant, so the result is reversed at the end. The counter starts at 62ยณ = 238,328 so that codes have at least four characters; the first link gets 1000.

Validating the URL. The URL comes from outside, so the program never trusts it. It must parse, use the http or https scheme and have a host. This also blocks dangerous schemes such as javascript: and file:.

The handlers. Routes builds a ServeMux with the Go 1.22 patterns POST /api/links and GET /{code}. The method is part of the pattern, so the mux itself answers 405 Method Not Allowed for a wrong method. is a wildcard, and r.PathValue("code") returns what matched. The redirect handler calls http.Redirect with status 302 Found.

Calling the handlers without a network. httptest.NewRequest builds a request in memory and httptest.NewRecorder is a ResponseWriter that remembers what was written. Passing them to handler.ServeHTTP runs the whole routing and handler code with no sockets, no ports, and in microseconds. This is how Go web code is tested, and how our demo program drives the service.

Go Stage 1: base-62 codes, create and redirect

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Every line of output is worth reading. The first two requests create links 1000 and 1001 (201 Created). The next two follow them: 302 with a Location header holding the long URL. /nope gives the router's plain 404. ftp://... and the non-JSON body both give 400 with a message, and POST /1000 shows that a path which exists for GET answers 405 Method Not Allowed for other methods, with no code from us.

Notes on the code:

  • The zero check in encode62 matters: the loop for ; n > 0; n /= 62 runs zero times for 0, so without the special case the code would be an empty string.
  • http.Error writes a plain-text message and sets the status. It is perfect for stage 1; stage 2 will switch to JSON errors because API clients like to parse them.
  • url.Parse is forgiving, which is why the code checks scheme and host afterwards. url.Parse("hello") succeeds with only a path.
  • 302 or 301? A 301 Moved Permanently lets browsers and search engines cache the redirect, which saves traffic but means you never see repeat visits and cannot change or delete the link. 302 Found is the right default for a service that counts clicks and allows deletion.

Stage 2: custom codes, stats and middleware

Stage 2 makes the service more useful and sturdier. Because these programs get long, the code is organized in four blocks: the store, the handlers, the middleware, and main.

The store grows up

  • Injected dependencies. NewStore(newID, now) takes the code generator and the clock as functions. Production passes randomCode and time.Now; tests and demos pass deterministic ones. This is the same trick as the today field in the to-do project.
  • Unguessable default codes. Sequential codes let anyone enumerate every link by counting. randomCode uses crypto/rand.Text() (Go 1.24), which returns random base-32 text, and keeps seven lowercase characters, about 35 bits. Collisions are possible, so Create loops until it finds a free code, and it does so while holding the lock, so two requests can never pick the same one.
  • Custom codes ("code": "pkg") are checked against a regular expression and a list of reserved words. A code like api must be refused, or it would shadow the /api/... routes. A taken code is a different kind of failure: ErrCodeTaken, wrapped with the code for context.
  • Hit counts live in an atomic.Int64 inside each entry. Resolve only needs a read lock to find the entry, and then increments the counter without any lock. Because the counter is atomic, thousands of simultaneous redirects never lose a count. See atomics.

Handlers with JSON in and out

The create handler limits the body to 64 KB with http.MaxBytesReader, rejects unknown fields with DisallowUnknownFields (a misspelled "cod" is an error, not silently ignored), and maps store errors to status codes with errors.Is: 400 for invalid input, 409 Conflict for a taken code, 500 for anything unexpected, with a generic message so that internal details never leak. It sets the Location header of the 201 response to the new short URL, as REST convention suggests.

Middleware

Middleware is a function that takes an http.Handler and returns a new http.Handler that does something before and/or after calling the original. It is the standard way to add cross-cutting behavior without touching handlers. The type Middleware func(http.Handler) http.Handler and a tiny Chain function are all you need:

Go
handler := Chain(top, RequestID(newID), Logging(logger), Recover(logger))

Chain wraps from the inside out, so the first middleware listed is the outermost: a request passes through RequestID, then Logging, then Recover, then the real handler, and the response travels back the same way. The order matters:

  • RequestID creates an id, stores it in the request's context under a private key type, and sets an X-Request-ID response header. Later layers read it with requestID(r). Keys of an unexported type cannot collide with keys from other packages. See context.
  • Logging has to know the response status, which handlers write through the ResponseWriter. It wraps the writer in a statusRecorder that remembers the code passed to WriteHeader. The wrapper has an Unwrap method (Go 1.20), so features such as http.ResponseController still reach the real writer.
  • Recover uses defer and recover() to catch a panic inside a handler and answer 500 instead of letting the connection drop. It sits inside Logging so that the access log records the 500. (net/http already recovers panics per connection and logs them, but it closes the connection without a response; a middleware lets you send a proper error.)

The demo below defines an extra route /boom whose handler panics (it writes to a nil map), then runs a script of requests and prints the access log at the end. Time never appears in the output, which keeps it deterministic.

Go Stage 2: custom codes, stats, delete and middleware

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Read the responses against the rules. The third request fails with 409 because pkg exists; code: "api" is refused as reserved; javascript:alert(1) fails URL validation; the misspelled field colour fails the strict JSON decoder. The two visits to /pkg show up as "hits":2 in the stats. /boom returns the generic 500 body, and the log shows the panic message assignment to entry in nil map on the line before the request line, so you can find the cause without the client ever seeing it. PUT on the stats route gives 405 with Allow: DELETE, GET, HEAD, built by the mux. After DELETE, /pkg is 404.

A public URL shortener is an attractive tool for abusers: short links hide phishing destinations. Real services add rate limiting per client, blocklists such as Safe Browsing, account requirements, and a preview page before the redirect. The scheme check in validateURL is the minimum, not the whole answer.

Stage 3: a real server with graceful shutdown

So far requests never left the process. Stage 3 starts a real HTTP server on a loopback port and talks to it with http.Client, which is exactly how production code runs, just on 127.0.0.1.

The pieces:

  • net.Listen("tcp", "127.0.0.1:0") opens a listening socket. Port 0 asks the operating system for any free port, and ln.Addr() tells you which one it picked. Using your own listener, instead of ListenAndServe, makes the port known before the server starts.
  • http.Server with ReadHeaderTimeout set. Servers without timeouts can be held hostage by slow clients; this field protects against the "Slowloris" attack, and security linters such as gosec flag its absence. Set ReadTimeout, WriteTimeout and IdleTimeout as well in production.
  • A client that does not follow redirects. By default http.Client follows up to ten redirects. Returning http.ErrUseLastResponse from CheckRedirect makes it hand back the 302 response itself, so we can inspect the Location header.
  • Graceful shutdown. When a service is redeployed, killing it would cut off requests that are in progress. srv.Shutdown(ctx) stops accepting new connections, closes idle ones, and waits for active requests to finish, up to the deadline in ctx. Serve then returns http.ErrServerClosed, which is not an error condition but the signal that shutdown was requested.

The demo adds a /slow route that tells main when it has started and then blocks until main releases it. While that request is in flight, main calls Shutdown in a goroutine and checks that it does not return for 100 milliseconds, proving it is waiting. Then the slow handler is released, its response arrives intact, and only then does Shutdown return. The port number differs on every run, so the program replaces the base address with BASE before printing.

Go Stage 3: real loopback server and graceful shutdown

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

The sequence of messages is the point. Shutdown is waiting for the request in flight is printed first, the slow request completes with 200 (it was not cut off), Shutdown and Serve then return, and a new request fails because the listener is closed. Contrast this with srv.Close(), which closes all connections immediately.

In a real main, shutdown is triggered by an operating-system signal, usually SIGINT (Ctrl+C) or SIGTERM (sent by Docker and Kubernetes). signal.NotifyContext turns the signal into a context that becomes done. This is the shape of a production main (read-only, since the editor cannot send signals):

Go
func main() {
	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
	defer stop()

	srv := &http.Server{
		Addr:              ":8080",
		Handler:           handler,
		ReadHeaderTimeout: 5 * time.Second,
	}

	go func() {
		<-ctx.Done() // a signal arrived
		shutdownCtx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
		defer cancel()
		if err := srv.Shutdown(shutdownCtx); err != nil {
			log.Printf("shutdown: %v", err)
		}
	}()

	if err := srv.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {
		log.Fatal(err)
	}
	log.Println("stopped cleanly")
}

Two details deserve attention. ListenAndServe returns immediately with ErrServerClosed the moment Shutdown is called, not when it finishes, so main must not exit until the shutdown goroutine is done. In real code, wait for it with a channel or sync.WaitGroup; otherwise the process exits and the waiting is cut short. And the timeout on shutdownCtx bounds how long a stuck request can delay the deploy.

Stage 4: testing the service

Stage 2's demo proves the service works once. A test suite proves it keeps working after every change. Because the handlers are plain http.Handler values, tests need no network: httptest.NewRecorder captures the answer and the test inspects status, headers and body.

Each test builds its own handler from newHandler(), with deterministic codes and clock (c001, c002, ...), so results never depend on earlier tests. The suite covers:

  • TestCreate, a table of nine inputs, each with expected status and a fragment of the body. Adding a case is one line.
  • TestDuplicateCodeIsConflict and TestRoutingErrors, the 409, 404 and 405 paths, including the Allow header.
  • TestRedirectCountsHits, three redirects followed by a stats call that must report exactly 3 hits.
  • TestPanicBecomes500, which also checks the panic text does not leak to the client.
  • TestConcurrentCreateGivesUniqueCodes, 200 goroutines create links at once with the real random generator, and every code must be distinct. Note that it calls t.Error and not t.Fatal from the goroutines, because FailNow may only run on the test's own goroutine. Run it with go test -race to check the store's locking too.

As in the earlier projects, the TestXxx functions are what you would put in a main_test.go, and main here calls testing.Main as a stand-in for go test -v.

Go Stage 4: httptest test suite

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

All tests pass. The names printed under TestCreate/... are the case names from the table, so a failure reads, for example, --- FAIL: TestCreate/reserved_code. You could also test the whole stack over a real socket with httptest.NewServer(handler), which starts a server on a random loopback port and gives you its URL. Use it when the behavior you test depends on the network layer (TLS, keep-alive, redirects followed by a client). For handler logic, the recorder is faster and simpler.

On your machine, run the suite with the race detector and several repetitions:

Terminal
go test -race -count=5 ./...

Ideas for extending the service

  • Keep links across restarts: save them in a JSON file or SQLite behind a Store interface (see databases with database/sql).
  • Add an expiry time per link and a background goroutine, stopped through a context, that removes expired links.
  • Rate-limit creation per client IP with a token bucket (see concurrency patterns).
  • Count visits per day, and expose them as GET /api/links/{code}/stats.
  • Return the Retry-After header together with 429 Too Many Requests when the limit is hit.
  • Use slog for structured access logs (see logging) and a /metrics endpoint.
  • Serve a small HTML form at / with html/template (see text and HTML templates).

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does the pattern "GET /{code}" do in a Go 1.22+ ServeMux?

    Go 1.22 added methods and wildcards to ServeMux patterns. Other methods on the same path get 405 Method Not Allowed automatically.
  2. 2. Why does Chain(h, A, B, C) make A the outermost wrapper?

    The loop wraps with C first, then B, then A. A request therefore passes through A, then B, then C, then the handler.
  3. 3. What is http.ErrServerClosed when Serve or ListenAndServe returns it?

    After Shutdown starts, Serve returns ErrServerClosed. Treat it as success and let Shutdown finish waiting for active requests.
  4. 4. Why does the redirect handler use 302 Found instead of 301 Moved Permanently?

    A permanent redirect may be cached by browsers and search engines. A service that counts clicks and supports deletion should use a temporary redirect.
  5. 5. In TestConcurrentCreateGivesUniqueCodes, why do the goroutines call t.Error instead of t.Fatal?

    FailNow (used by Fatal) stops only the calling goroutine and must run on the test goroutine. Error and Errorf are safe to call from any goroutine.

Summary

  • A Go web service is a set of http.Handlers behind a ServeMux. Since Go 1.22 patterns include the method and wildcards (GET /{code}), and the mux answers 404 and 405 for you.
  • Guard the shared store with a sync.RWMutex, count hits with atomic.Int64, and inject the code generator and the clock so that tests are deterministic.
  • Validate everything that comes from the client: URL scheme and host, code format, body size and unknown JSON fields. Map errors to precise status codes (400, 404, 409) and never leak internals in a 500.
  • Middleware is func(http.Handler) http.Handler. Wrap the ResponseWriter to observe the status, pass request-scoped values through context, and recover from panics to return a clean 500.
  • Run servers with timeouts, and shut them down with srv.Shutdown(ctx) so requests in flight can finish; ErrServerClosed is the expected result of Serve.
  • httptest.NewRecorder tests handlers without a network, and table-driven tests with named cases keep the suite readable. Run concurrent code under go test -race.

Next: Project: Concurrent Web Crawler