Skip to content
Tutorials Center
My learning

Topics

Lesson 111 of 147 ยท Web and Networking

TCP Servers and Custom Protocols

Build TCP servers in Go with net.Listen and Accept, design a line protocol, add framing and deadlines, shut down gracefully and write the core of a multi-user chat server.

HTTP is not the only way programs talk to each other. Databases, message brokers, game servers, SSH, SMTP and Redis all speak their own protocols directly over TCP, the reliable byte stream that sits underneath HTTP. Go is exceptionally good at this kind of code. A TCP server is a loop that accepts connections and starts one goroutine per connection, and because goroutines are cheap and blocking reads are cheap, you can write straightforward sequential code and still serve thousands of clients.

This lesson goes from the basic Listen/Accept loop to a custom command protocol, then covers the three things that make network code robust: framing (where does a message end?), deadlines (what if the peer goes silent?) and graceful shutdown. The finale is the core of a chat server. Everything runs on the loopback address 127.0.0.1, which is the only network the browser sandbox allows. The quick tour in URLs and Network Basics introduced the same API, so this page goes deeper.

The accept loop

Every TCP server in Go has the same skeleton:

  1. net.Listen("tcp", addr) reserves a port and returns a net.Listener. Use 127.0.0.1:0 to let the operating system pick a free port, and read it back from ln.Addr().
  2. ln.Accept() blocks until a client connects, then returns a net.Conn.
  3. The loop starts a goroutine for the connection and immediately calls Accept again.
  4. The connection handler reads and writes, and closes the connection when it is finished.

A net.Conn implements io.Reader, io.Writer and io.Closer, so everything you know from Bytes and Buffers and bufio applies. TCP is a stream: it delivers bytes in order and without loss, but it does not remember how you split them into Write calls. Anything that looks like a "message" is something your protocol defines.

A line protocol

The simplest protocol is text, one command per line, one reply per command. Redis, SMTP and FTP all started that way, and you can test such a server by hand with telnet or nc. The one below is a tiny key-value store with the commands PING, SET key value, GET key and QUIT. Replies start with + for success and - for errors.

Go A key-value server with a line protocol

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Two connections are served at the same time by two goroutines, and the shared map is protected by a mutex, as in any concurrent Go code. After QUIT, the handler returns and its deferred conn.Close() runs, so the client's next read fails with EOF. Some design details are worth copying:

  • Use bufio.Scanner (or bufio.Reader) on the connection instead of calling conn.Read yourself. A single Read may return half a line or three lines.
  • The client reads the reply to each command before it sends the next, which keeps the conversation in lock step. Protocols that allow pipelining (sending many commands before reading replies) need the reader and writer to run independently.
  • Errors in the protocol (-ERR ...) are ordinary replies; errors in the connection end the handler.

bufio.Scanner rejects lines longer than 64 KB by default and stops with bufio.ErrTooLong. That is a useful safety limit against clients that never send a newline, but make it deliberate: call scanner.Buffer(make([]byte, 0, 4096), 1<<20) to choose your own maximum. Never read unbounded input into memory.

Deadlines: never wait forever

A TCP read has no time limit by default. If a client connects and then says nothing (a crashed laptop, a lost Wi-Fi signal, an attacker opening thousands of silent connections), your goroutine blocks until the operating system gives up, which can take hours. Every server must decide how long it is willing to wait.

conn.SetReadDeadline(t) sets an absolute point in time. Any read after that time fails with a timeout error (which matches os.ErrDeadlineExceeded). Because it is absolute rather than a duration, you must set it again before every read if you want an idle timeout. SetWriteDeadline limits writes in the same way (useful against clients that stop reading), and SetDeadline sets both.

Go An idle timeout that is renewed on every line

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

The server keeps extending its deadline as long as lines keep arriving. After the client falls silent for more than 100 ms, the next Scan fails, the handler sends a farewell and returns; the deferred Close makes the client see EOF. In production, idle timeouts are typically seconds or minutes, and long-lived protocols add heartbeats (a periodic PING) so that silence means "dead" and not just "quiet". The operating system also has TCP keep-alive probes; Go enables them by default for connections it creates (net.Dialer.KeepAlive and net.ListenConfig.KeepAlive tune them), but they work on a timescale of minutes and are not a substitute for application-level timeouts.

Framing messages

Lines are fine for text, but what if a message can contain newlines, or is binary (an image, a protobuf)? You need a framing rule that tells the receiver where each message ends. The two common choices:

  • Delimiters: end each message with a marker (\n, \r\n, a zero byte). Simple, human-readable, but the payload must never contain the marker or must be escaped.
  • Length prefix: send the payload size first (for example four bytes, big-endian), then exactly that many bytes. Works for any binary data and lets the reader allocate the right buffer. This is what most binary protocols (including Postgres, Kafka and gRPC) do.

The important primitive on the reading side is io.ReadFull(r, buf): it keeps calling Read until buf is full, so a frame split across several TCP segments is reassembled. Equally important is a maximum frame size. If you trust the length in the header and allocate that many bytes, a hostile client can make you allocate 4 GB with a 4-byte message.

Go Length-prefixed frames over a TCP connection

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

The server loop sees exactly the messages that were sent: the three frames that shared one Write arrive as three frames (TCP does not care about write boundaries), the empty frame is a valid zero-length message, and the frame sent in two pieces is reassembled by io.ReadFull. The final bogus header is rejected before any memory is allocated. Notice the two kinds of end-of-input: io.EOF when the peer closes cleanly between frames, and io.ErrUnexpectedEOF when it disappears in the middle of a frame. Handle them differently: the first is normal, the second is a protocol error worth logging.

Also, the writer builds header and payload into one buffer and writes once. Two separate Write calls could be interleaved with another goroutine's frame on the same connection; a connection should have only one goroutine writing, or a mutex around each frame.

Graceful shutdown

When you deploy a new version, you do not want to cut every connection in the middle of a conversation. A graceful shutdown does the following:

  1. Stop accepting new connections (ln.Close() makes Accept return an error).
  2. Tell the existing handlers to finish: here, by setting a read deadline of "now", which wakes every handler that is blocked reading.
  3. Wait for the handlers with a sync.WaitGroup, but only up to a limit; after that, force-close what is left.

The structure mirrors http.Server.Shutdown, which you will use for HTTP servers in Building an HTTP Server. Writing it once for TCP shows what happens behind it.

Go A server with Shutdown and a deadline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Look at the order of events. Shutdown closes the listener, so Serve returns an error that wraps net.ErrClosed; that is how you recognise a deliberate stop. It sets a read deadline on the idle connection, the handler's Scan fails and the handler returns, and Wait completes well within the one-second limit. The client sees EOF, and further connections are refused. The closing flag, checked under the same mutex, closes a subtle race: a client that connected a moment before the listener closed can never slip into the map after Shutdown has already looked at it.

The same pattern appears at program level. Call Shutdown when your process receives SIGINT or SIGTERM, using signal.NotifyContext (The context Package).

The core of a chat server

Now combine everything. A chat server needs one goroutine per connection for reading, and a way to send a message from one connection's goroutine to all the others. Writing to other clients' sockets directly from the sender's goroutine would let a single slow client block everybody, so each client gets an outgoing channel and a dedicated writer goroutine. The hub only puts messages into channels; if a client's buffer is full, it is too slow and is disconnected (the same backpressure rule as in Streaming, SSE and WebSockets).

The protocol is simple: the first line is NICK name; after that, every line is a message for everyone else. The server announces joins and leaves. The test client at the bottom is scripted so that the output is the same on every run: each step waits until the previous message has arrived.

Go A multi-user chat server on loopback

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Each user's list shows what the chat delivered to them, in order. Ada never sees her own message, Bob sees that Cy joined after him, and Cy only sees what happened after he arrived. Because the server code uses locks and channels in a particular order, the script above is deterministic; on a real network, messages from different clients have no global order and only the order per connection is guaranteed.

Study the structure, because it is the template for almost every networked server in Go:

  • One reader goroutine per connection (handle) turns bytes into events.
  • One writer goroutine per connection consumes a channel, so writes never race and a slow socket only delays its own queue.
  • A hub with a mutex guards the shared map. broadcast is non-blocking and runs while the lock is held, so it must only touch channels, never sockets.
  • Cleanup always runs: leave removes the client, closes its channel (which ends the writer loop), and Close releases the socket.

To run this as a real server, replace 127.0.0.1:0 with a fixed port such as :4000, add idle and write deadlines, limit line length with scanner.Buffer, and use Shutdown as shown earlier. The Chat Server project builds the full version.

Other things to know

  • TLS: tls.Listen and tls.Dial wrap the same API with encryption; the code above works unchanged on the resulting net.Conn. Load a certificate with tls.LoadX509KeyPair.
  • UDP: net.ListenPacket("udp", addr) returns a net.PacketConn with ReadFrom/WriteTo. UDP keeps message boundaries but may lose, duplicate or reorder packets.
  • Unix sockets: net.Listen("unix", "/tmp/app.sock") has the same API and is useful for talking to a local daemon.
  • Half-close: conn.(*net.TCPConn).CloseWrite() tells the peer "I am done sending" while you keep reading, which protocols like "send a request, then wait for the reply until EOF" rely on.
  • Accept errors: a failing Accept is not always fatal. Running out of file descriptors, for example, is temporary. http.Server retries with a short backoff; a production TCP server should do the same instead of exiting.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A client calls conn.Write three times, each with a short message. What can the server be sure of when it reads from its side of the TCP connection?

    TCP is a reliable, ordered byte stream. It does not preserve the boundaries of individual Write calls, so protocols use delimiters or length prefixes.
  2. 2. Why is conn.SetReadDeadline(time.Now().Add(idleTimeout)) called inside the read loop rather than once before it?

    A deadline is a fixed moment, not a duration. Setting it once makes every read after that moment fail. Renewing it before each read means the connection only times out after being idle that long.
  3. 3. In a length-prefixed protocol, why should the reader check the length against a maximum before allocating the buffer?

    The length comes from an untrusted peer. Without a limit, four bytes on the wire can make the server allocate gigabytes.
  4. 4. What does errors.Is(err, io.ErrUnexpectedEOF) from io.ReadFull tell you while reading a frame?

    ReadFull returns io.EOF if no bytes were read and io.ErrUnexpectedEOF if some, but not all, of the requested bytes arrived. The first is normal, the second means the frame was cut off.
  5. 5. In the chat server, why does each client have its own writer goroutine and outgoing channel?

    Broadcasting straight to sockets from the sender's goroutine would let one slow reader stall all senders. Channels decouple the producers from each socket, and a single writer goroutine serializes writes.

Summary

  • A TCP server is net.Listen, then a loop of Accept with one goroutine per connection; net.Conn is an io.Reader, io.Writer and io.Closer. Use 127.0.0.1:0 for a free port.
  • TCP is an ordered byte stream with no message boundaries. Define your own framing: delimiters such as newline, or a length prefix read with io.ReadFull, and always enforce a maximum size.
  • Wrap connections in bufio.Scanner or bufio.Reader; set scanner.Buffer if you need lines longer than 64 KB.
  • Deadlines are absolute times. Renew SetReadDeadline before each read for an idle timeout, use SetWriteDeadline against clients that do not read, and add heartbeats for long-lived connections.
  • Graceful shutdown closes the listener, wakes idle handlers (a read deadline of "now"), waits with a WaitGroup and force-closes after a context timeout. errors.Is(err, net.ErrClosed) recognises a deliberate stop.
  • For chat-style servers, use one reader goroutine and one writer goroutine per connection, a hub guarded by a mutex, and non-blocking broadcasts that drop slow clients.

Next: RPC, gRPC and Protocol Buffers