Skip to content
Tutorials Center
My learning

Topics

Lesson 108 of 147 ยท Web and Networking

The HTTP Client

Learn the Go HTTP client with net/http - Get, Post and Do, custom timeouts, request building, JSON bodies, closing response bodies, connection reuse and safe retries.

Almost every modern program talks to another program over HTTP: it calls a payment API, fetches a configuration file, or asks another microservice for data. Go's standard library makes this easy, so easy that the default code you find in many examples is also dangerously incomplete. A request without a timeout can hang forever, and a response body that is never closed leaks a connection.

This lesson teaches the http.Client the way production code uses it: building requests with a context, setting timeouts, sending and reading JSON, closing bodies, reusing connections and retrying safely. Because the sandbox on this site can only reach itself, the examples call a small local server started with httptest.NewServer (introduced in Building an HTTP Server) or a stand-in transport inside the same program. Against a real service you would only replace the URL.

The simplest request

http.Get(url) sends a GET request and returns an *http.Response. The response has a Status line, a numeric StatusCode, a Header map and a Body that you read like any other io.Reader.

Go Your first GET request

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Three rules are visible in this short program:

  • err means the request could not be completed: DNS failed, the connection was refused, a timeout fired. A server that answers 404 or 500 is not an error from Get's point of view. The round trip worked, so you get a normal response and must check StatusCode yourself.
  • Always defer resp.Body.Close() after you have checked err. If err != nil, resp is nil and there is nothing to close.
  • resp.Header.Get canonicalizes the key (content-type and Content-Type are the same header).

http.Get, http.Post and friends use http.DefaultClient, which has no timeout at all. A server that accepts the connection and then goes silent blocks your goroutine forever. Fine for a five-line experiment, wrong for anything that runs in production. The next section shows how to fix it.

Timeouts: the most important setting

Create your own http.Client and set Timeout. It is a single deadline for the entire exchange: connecting, sending the request, waiting for the response headers and reading the body.

Go A client timeout in action

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

The slow handler needs 300 ms, the client allows 100 ms, so client.Get returns an error. The message mentions Client.Timeout exceeded and contains the full URL with a random port, which is why this program prints only booleans. Check timeouts with errors.As and net.Error (or errors.Is(err, context.DeadlineExceeded)), as described in Wrapping and Inspecting Errors, not by searching the error text.

A single number is a blunt tool. For finer control, give the client an http.Transport and set the individual phases:

Go
client := &http.Client{
	Timeout: 30 * time.Second, // whole exchange, including reading the body
	Transport: &http.Transport{
		DialContext:           (&net.Dialer{Timeout: 5 * time.Second}).DialContext,
		TLSHandshakeTimeout:   5 * time.Second,
		ResponseHeaderTimeout: 10 * time.Second,
		MaxIdleConns:          100,
		MaxIdleConnsPerHost:   10,
		IdleConnTimeout:       90 * time.Second,
	},
}

Client.Timeout is simple and always a good default. Per-request deadlines, however, are better expressed with a context, which also lets the caller cancel the request.

Building requests with a context

client.Get is a shortcut. The general form is to create a request with http.NewRequestWithContext, adjust it, and send it with client.Do. This is the form you will use most often, because it is the only way to set headers, choose methods other than GET and POST, and attach a context.

Go Headers, query parameters and context

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Build query strings with url.Values, never by string concatenation; URLs and Network Basics explains why. The context belongs to the request: when it expires or is cancelled, Do returns an error and the connection is abandoned, even if the client has no Timeout. Combine the two when you want both a general safety net and tighter per-call limits. The context Package covers cancellation in depth.

Use the constants http.MethodGet, http.MethodPost, http.MethodPut, http.MethodDelete and so on instead of string literals; a typo in "GTE" compiles, a typo in http.MethodGTE does not.

Sending and receiving JSON

JSON is the usual body format for APIs. To send it, encode a value into a buffer (or any io.Reader), pass the reader as the request body and set Content-Type. To receive it, decode straight from resp.Body with json.NewDecoder, as shown in Working with JSON.

Go POST a JSON body and decode the reply

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Notice the shape of createUser. It takes the client and the context as parameters (so tests can substitute a fake and callers control cancellation), checks the status code explicitly, and wraps errors with context using %w. That structure repeats in every API client you will write.

The example also shows a neat testing trick. http.Client.Transport is an interface (http.RoundTripper) with a single method, so you can replace the network with any code you like. handlerTransport hands each request straight to an http.Handler and returns the recorded response: no sockets, no port numbers, no flakiness. The same pattern works for simulating timeouts and server failures in unit tests. (It is also what this page needs: the sandbox's size limit for compiled programs is too small for a program that contains both an HTTP server and encoding/json, so the JSON examples avoid starting a real server.)

Status codes outside the 2xx range often carry a useful error message in the body. Read a limited amount of it (io.ReadAll(io.LimitReader(resp.Body, 4096))) and include it in your error, so a log line says why the server said no, not just that it did.

Closing bodies and reusing connections

Opening a TCP (and TLS) connection is expensive, so the transport keeps finished connections in a pool and reuses them for later requests to the same host. That only works if you read the body to the end and close it. If you close it early, or never, the connection cannot go back to the pool and the next request has to dial again; in the worst case, you leak connections until the process runs out of file descriptors.

net/http/httptrace lets you observe this. The program below makes four requests: the first dials a new connection, the second reuses it, the third reuses it but deliberately skips reading the body before closing it, and the fourth shows the consequence.

Go Watching connection reuse with httptrace

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

The first line says reused=false because nothing was in the pool. The second request finds the idle connection and says true. The third also reuses it, but because we close the body of a 1 MiB response without reading it, the transport drops the connection instead of returning it to the pool, so the fourth request must dial again and reports false. Small unread bodies may be skipped quietly by the transport, which is why the example uses a large one; relying on that is a gamble. The rule is simple: when you do not care about the body, still drain it with io.Copy(io.Discard, resp.Body) before closing.

Two more pool-related facts:

  • Share one client. http.Client and http.Transport are safe for concurrent use and are meant to be created once and reused by many goroutines. Creating a client per request throws the pool away. Package-level or struct-field clients are the norm.
  • http.Client follows redirects (up to 10) by default. Set CheckRedirect to return http.ErrUseLastResponse if you want to see the 3xx response yourself.

Retrying safely

Networks fail briefly and often: a restarting server, a dropped connection, a 503 from an overloaded service. A small, well-behaved retry loop turns many of those into non-events. A good retry policy answers four questions:

  1. Which failures are retryable? Connection errors and 502, 503, 504 (and 429 with a Retry-After header). Not 400, 401, 404: asking again gives the same answer.
  2. Which requests are safe to repeat? GET, PUT and DELETE are idempotent by definition. A POST that creates an order is not, unless the API supports an idempotency key.
  3. How long to wait? Exponential backoff (100 ms, 200 ms, 400 ms, ...), ideally with random jitter so that thousands of clients do not retry in lockstep.
  4. When to give up? A maximum number of attempts, and always the caller's context.
Go A retry loop with exponential backoff

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

The first call retries twice on 503 and then succeeds. The second target answers 404, which the loop treats as permanent and returns immediately without waiting. Notice that the loop uses select on both a timer and ctx.Done(), so a cancelled context stops the waiting at once instead of sleeping through the delay. For real services, add jitter (delay + rand.N(delay/2)) and honor Retry-After.

A request with a body can only be retried if you can produce the body again. bytes.NewReader, bytes.NewBuffer and strings.NewReader set req.GetBody automatically, so the client can replay them on redirects. When you stream from a file or a pipe, you have to reopen it yourself for every attempt.

Other things worth knowing

  • Form posts: client.PostForm(url, values) sends application/x-www-form-urlencoded. For uploads, build a multipart.Writer and pass the buffer as the body.
  • Cookies: set Client.Jar (from net/http/cookiejar) to keep cookies between requests.
  • Proxies and TLS: Transport.Proxy defaults to http.ProxyFromEnvironment; custom certificate pools go into Transport.TLSClientConfig. Never set InsecureSkipVerify: true outside of a throwaway experiment.
  • HTTP/2: the default transport speaks HTTP/2 automatically over TLS. You do not have to enable anything.
  • Large downloads: a Client.Timeout covers reading the body. For a multi-gigabyte download, use a client without Timeout and give the request a context that you cancel yourself, or the download is cut off while it is still making progress.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A server answers a client.Get call with status 500. What does client.Get return?

    An error from Get means the request could not be completed (DNS, connection, timeout). Any answer from the server, including 4xx and 5xx, is a normal response whose StatusCode you must check.
  2. 2. What is wrong with using http.Get for calls to an external service in production code?

    The default client never gives up. A slow or silent server blocks the calling goroutine forever. Create your own http.Client with a Timeout or use a request context.
  3. 3. Why should you drain the response body (for example with io.Copy(io.Discard, resp.Body)) before closing it, even when you ignore the content?

    The transport can only reuse a connection once the body has been read to the end. An abandoned large body makes it close the connection, and the next request has to dial again.
  4. 4. Which of these is the best candidate for an automatic retry?

    503 is usually temporary and GET is idempotent. A 404 or a malformed URL fails the same way every time, and retrying a non-idempotent POST can create duplicate orders.
  5. 5. How should you share an http.Client in a program with many goroutines?

    http.Client and http.Transport are safe for concurrent use. Reusing one client keeps the connection pool alive, while a new client per request discards it.

Summary

  • client.Do(req) is the general call; Get, Post and PostForm are shortcuts. An err means the exchange failed, not that the status was bad: check resp.StatusCode yourself.
  • http.DefaultClient has no timeout. Create a client with Timeout, and attach a context with http.NewRequestWithContext for per-call deadlines and cancellation.
  • Build URLs with url.Values and set headers on the request; use the http.MethodXxx constants.
  • Encode JSON request bodies with json.Marshal and a bytes.Reader, set Content-Type, and decode replies with json.NewDecoder(resp.Body).
  • Always close resp.Body after checking err, and drain it when you ignore the content, so connections return to the pool. Share one client between goroutines.
  • Retry only transient failures of idempotent requests, with exponential backoff, a maximum attempt count and the caller's context.

Next: Building a JSON REST API