Skip to content
Tutorials Center
My learning

Topics

Lesson 74 of 79 ยท How the Web Works and Real-World Projects

Project: A Tiny HTTP Server from Raw TCP

Build an HTTP/1.1 server from raw TCP sockets: parse request lines, headers and bodies, route requests, send status codes and support keep-alive.

Every web framework, however large, starts with the same few moves: read bytes from a socket, recognize an HTTP request in them, decide what to answer, and write bytes back. In this project you do exactly that with nothing but Node's net module. You will not use the http module for the server, so no part of the protocol is hidden from you. At the end, real clients (fetch and http.get) will talk to your server as if it were any other web server.

HTTP/1.1 is a text protocol on top of TCP, which makes it a wonderful teaching subject. You already know its shape from HTTP fundamentals and the persistent connections of HTTP/1.1 and later. The chat-server project taught you to frame a byte stream into lines; here the framing rules are a little richer: a head that ends with a blank line, followed by a body whose length is announced in a header.

What a request looks like on the wire

A request is a few lines of text separated by \r\n (carriage return and line feed), ended by an empty line:

Text
POST /echo HTTP/1.1\r\n
Host: localhost\r\n
Content-Type: text/plain\r\n
Content-Length: 5\r\n
\r\n
hello

The first line is the request line: method, target and protocol version. Then come header lines of the form Name: value. A blank line ends the head. If a Content-Length header is present, exactly that many bytes of body follow. Nothing else marks where the message ends, so a server that gets this wrong either hangs, waiting for bytes that will not come, or reads the next request as part of the body.

flowchart TD
    A["Bytes arrive in a buffer"] --> B{"Blank line<br/>(CRLF CRLF) seen?"}
    B -- "no" --> A
    B -- "yes" --> C["Parse request line<br/>and headers"]
    C --> D{"Content-Length<br/>bytes present?"}
    D -- "no" --> A
    D -- "yes" --> E["Route the request<br/>and write a response"]
    E --> F{"Keep the<br/>connection?"}
    F -- "yes" --> B
    F -- "no" --> G["Close"]

The per-connection loop. After each response the server checks the buffer again, because the next request may already be waiting in it.

Step 1: the request parser

The parser is a pure function. It takes the bytes buffered so far and returns one of three things: null ("I need more bytes"), a parsed request together with the number of bytes it consumed, or it throws an HttpError carrying the status code to send back. Keeping it free of sockets makes it easy to test. Notice the defensive details, because the parser reads untrusted data:

  • Limits. The head may not exceed 4 KB (431) and the body may not exceed 1 KB (413). The size check happens as soon as the Content-Length header is known, so the server refuses a huge upload before receiving it.
  • Strict syntax. The request line must have exactly three parts and the method must be capital letters. A header name must not contain whitespace, which prevents a classic request-smuggling trick.
  • Case-insensitivity. Header names are case-insensitive, so they are lowercased on the way in.
  • Host. HTTP/1.1 requires a Host header, because one server may host many sites on one IP address.
  • Unsupported features produce honest answers: 505 for an unknown version and 501 for Transfer-Encoding (chunked uploads), which this tiny server does not implement.
Node.js Step 1: parse HTTP requests

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The "two requests in a row" line shows the key idea for keep-alive and pipelining: the parser reports that it used 28 of 56 bytes. The rest stays in the buffer, and the server loop will parse it next.

Step 2: routing, responses and keep-alive

With parsing done, the server needs four more pieces.

A router. A plain object maps "METHOD /path" to a handler function. If the exact key is missing, the server checks whether the path exists with another method. If it does, the answer is 405 Method Not Allowed with an Allow header listing the methods that work. Otherwise it is 404 Not Found. The difference matters for APIs and for debugging. HEAD is treated as GET with the body removed, which the spec requires.

A response writer. A response is a status line, headers and a body. The most important header is Content-Length, computed in bytes, not characters (a single รฉ is two bytes in UTF-8). Without it, the client cannot tell where the body ends on a persistent connection. A production server would also send a Date header; this one leaves it out so that the output below is the same on every run.

Keep-alive rules. In HTTP/1.1 the connection stays open after a response unless either side sends Connection: close. HTTP/1.0 is the opposite: it closes unless the client asks for Connection: keep-alive. Reusing a connection saves a TCP handshake (and a TLS handshake on HTTPS) for every request.

A connection loop. On each data event the server appends the chunk to a buffer and then parses in a while loop, because one chunk may contain several complete requests (pipelining) or only part of one. Handler exceptions become 500 responses instead of crashing the process. An idle timeout of two seconds closes connections that say nothing.

The client below is also hand-written and also reads responses by looking for the blank line and then Content-Length bytes. The same framing logic appears on both ends of the wire.

Node.js Step 2: the server and a raw client

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Read the output with these points in mind.

  • 404 and 405 are different failures: /missing exists nowhere, while DELETE /hello hits a path that only accepts GET, so the reply carries Allow: GET.
  • HEAD returns the same Content-Length as the matching GET (14 for Hello, Grace!) but no body at all.
  • A crashing handler produces a clean 500, and the connection survives.
  • Keep-alive: the three requests were answered over a single TCP connection, so the counter went up by one.
  • Pipelining: both requests were in one write, and both responses came back, in order. The second request asked for Connection: close, so that response says so.
  • HTTP/1.0 gets Connection: close by default. (The server still answers with HTTP/1.1 in the status line, which is allowed.)

Pipelining needs responses in the same order as requests, so one slow response blocks all the later ones. That is head-of-line blocking at the HTTP level, and it is why browsers never enabled pipelining and why HTTP/2 introduced multiplexed streams.

Step 3: real clients and hostile input

A protocol implementation is only correct if other people's software can use it. This run points Node's built-in fetch and an http.get request with a keep-alive agent at your server. They know nothing about your code; they simply speak HTTP. The server logs what it saw.

After that, the same script sends broken and oversized input over raw sockets. The parser should answer every one of those with the right 4xx or 5xx code rather than hang or crash.

Node.js Step 3: fetch, http.get and bad requests

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The 404 for fetch is a perfectly valid HTTP answer, not an error in the program, which is why fetch resolves normally and the code checks status. The 431, 413 and 501 replies were produced before the server read or allocated anything large, which is what protects a real server from memory-exhaustion attacks.

What a real server adds

You now have the essential engine. A production server such as nginx, or Node's own http module, adds a long list of features around it:

  • Date header, ETag and caching. See caching and cookies.
  • Chunked transfer encoding for bodies of unknown length (streaming responses), in both directions.
  • Compression with Content-Encoding: gzip, negotiated through Accept-Encoding.
  • TLS, so the same bytes travel encrypted (TLS and HTTPS).
  • Timeouts for slow clients that send a byte every few seconds (the "slowloris" attack), plus limits on connections per client.
  • URL decoding and path safety. If you ever serve files, normalize the path and refuse .. segments.
  • Protocol upgrades to WebSocket (real-time web) and the newer HTTP/2 binary framing.

Ideas to extend it

  • Static files. Serve a folder with correct Content-Type values, and answer 304 Not Modified when If-None-Match matches an ETag.
  • Chunked responses. Send a body of unknown length as Transfer-Encoding: chunked pieces (size in hex, CRLF, data, CRLF, and a final 0 chunk).
  • Cookies and sessions. Parse the Cookie header and send Set-Cookie with HttpOnly and SameSite.
  • Route parameters. Support GET /users/42 with a pattern like /users/:id.
  • Access log. Print one line per request with method, path, status and duration.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. How does an HTTP/1.1 server know where a request body ends?

    The blank line ends the head, not the body. The body length is announced by Content-Length or by chunked transfer encoding; otherwise a keep-alive connection could not carry the next request.
  2. 2. What is the difference between 404 and 405?

    A 405 response must include an Allow header listing the supported methods, which tells the client how to fix the request.
  3. 3. Two complete requests arrive in one TCP segment. What must the server do?

    TCP has no message boundaries. A parser that loops over the buffer handles any mixture of partial and multiple requests.
  4. 4. Why does the server check Content-Length against its limit as soon as the headers are parsed?

    Rejecting early (413) means an attacker cannot make the server buffer megabytes before being turned away.
  5. 5. By default, what happens to a connection after an HTTP/1.0 response?

    HTTP/1.1 made persistent connections the default, while HTTP/1.0 closes after each response unless keep-alive is requested explicitly.

Summary

  • HTTP/1.1 is text over TCP: a request line, headers, a blank line and an optional body whose length comes from Content-Length.
  • A parser should return "need more bytes", a request plus the count of bytes consumed, or an error status. It enforces limits early and treats all input as untrusted.
  • Routing maps method and path to a handler; a missing path is 404, a wrong method is 405 with an Allow header, and a crashing handler becomes 500.
  • Persistent connections are the HTTP/1.1 default: loop over the buffer, answer each request in order, and close only on Connection: close or after an error.
  • Interoperating with fetch and http.get proves the protocol work is right. Real servers add caching headers, compression, TLS, chunked bodies and timeouts on top.

Last project of the chapter: simulate a whole network with routers, links and Dijkstra routing.