Skip to content
Tutorials Center
My learning

Topics

Lesson 37 of 52 ยท Advanced Topics

Calling Other Services: HttpClient and Resilience

Call other services from ASP.NET Core with IHttpClientFactory and typed clients, then add timeouts, retries and a circuit breaker with delegating handlers.

Almost every real web application talks to other services: a payment provider, an inventory system, another team's API. Those calls cross a network, and networks are slow, flaky and occasionally gone. How you create your HttpClient objects and how you react when a call fails decides whether one sick dependency slows your whole application down or is handled quietly.

In this lesson you will use IHttpClientFactory and typed clients, build a small pipeline of delegating handlers, and implement a timeout, a retry with backoff and a circuit breaker by hand so you understand what the ready-made resilience libraries do for you. If you need a refresher on HttpClient and async, see the C# tutorial first, for example Tasks and async/await.

Why not new HttpClient() everywhere

HttpClient looks like something you create, use and dispose. That pattern has two famous problems:

  • Socket exhaustion. Each HttpClient owns a connection pool. Creating one per request and disposing it leaves sockets in the TIME_WAIT state, and under load you run out of them.
  • Stale DNS. The opposite fix, one static HttpClient for the whole process, keeps connections open forever, so a service that moved to a new IP address is never found again.

IHttpClientFactory removes the dilemma. It creates lightweight HttpClient objects that share pooled handlers (the part that owns connections), and it recycles those handlers every two minutes by default. You can create and dispose clients freely.

flowchart LR
    A["Your code<br/>(many HttpClient objects)"] --> B["IHttpClientFactory"]
    B --> C["Pooled handler chain<br/>(recycled every 2 min)"]
    C --> D["Connection pool"]
    D --> E["Remote service"]

Clients are cheap and disposable; the factory keeps the expensive handler chain and its connections.

Registering clients

You register clients on the service collection with AddHttpClient. There are three flavors:

  • Basic: services.AddHttpClient() and then factory.CreateClient().
  • Named: AddHttpClient("inventory", c => ...) and factory.CreateClient("inventory"). The name is a string, so typos only fail at run time.
  • Typed: AddHttpClient<InventoryClient>(c => ...). The class receives an HttpClient in its constructor and exposes methods that mean something to your application. This is the style you should prefer, because the rest of your code never sees URLs or headers.

A typed client is registered as a transient service, so you inject it into endpoints or services like any other dependency (Dependency Injection in ASP.NET Core explains lifetimes). The configuration lambda runs when the client is created, which makes it the right place for the base address and default headers.

The example starts a tiny "inventory service" on one port and uses a typed client to call it. Because it uses only a ServiceCollection, the same code works in a console program, a background worker or a web app.

C# A typed client calling another service

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

Three habits in GetStockAsync are worth copying. The sku is escaped with Uri.EscapeDataString, a 404 is an expected answer and becomes null, and every other failure raises an exception through EnsureSuccessStatusCode. The response is disposed with using, because it holds a connection until you read or dispose it.

In an ASP.NET Core app you normally register the typed client in Program.cs with builder.Services.AddHttpClient<InventoryClient>(...) and inject InventoryClient into your handlers. The ServiceCollection above is the same machinery without the web host.

Handlers: a pipeline around every request

An HttpClient sends each request through a chain of message handlers, exactly like middleware wraps a request on the server (The Middleware Pipeline). The innermost handler, SocketsHttpHandler, talks to the network. Every DelegatingHandler you add sits outside it, can inspect or change the request, call base.SendAsync to continue, and inspect the response on the way back.

You add one with AddHttpMessageHandler on the builder that AddHttpClient returns. The first handler you add is the outermost.

Typical jobs for handlers are adding authentication headers, logging, correlation ids, timeouts and retries. The next sections build the last two.

Timeouts

HttpClient.Timeout defaults to 100 seconds, which is far too long for a service that must answer within a second. Setting it applies to the whole call, including every retry. If you retry three times you usually want a short timeout per attempt and a larger overall limit, so the retry handler gets a timeout handler inside it.

A timeout is implemented with a linked CancellationTokenSource: create one from the caller's token, call CancelAfter, and pass its token on. When it fires, the caller sees an OperationCanceledException, which is confusing because it looks as if the caller cancelled. The usual fix is to translate it into a TimeoutException when the cancellation came from your timer rather than from the caller.

Retries with backoff

Some failures are transient: a network blip, a 503 while a server restarts, a 429 asking you to slow down. Trying again a moment later often works. Others are permanent: a 400 or 404 will never succeed no matter how often you retry. A good retry handler therefore:

  • retries only transient failures (connection errors, timeouts, 408, 429, 5xx),
  • waits longer between attempts (exponential backoff: 100 ms, 200 ms, 400 ms and so on, usually with random jitter so that many clients do not retry in lockstep),
  • gives up after a small number of attempts,
  • retries only operations that are idempotent (safe to repeat). GET, PUT and DELETE are; a POST that creates an order is not, unless it carries an idempotency key (OpenAPI, Documentation and API Versioning covers that convention).

Retries multiply load. If a service is struggling and every client retries three times, it receives four times the traffic exactly when it can least handle it. Keep retry counts low, add jitter in production, and combine retries with a circuit breaker.

The example below has a flaky backend that fails the first two requests with 503 and then succeeds, a slow endpoint, and a pipeline with a timeout handler inside a retry handler. The delays are tiny and jitter is left out so the output is deterministic.

C# A timeout and a retry handler written by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

Read the output carefully. /flaky needed three attempts, /missing returned a 404 after one attempt because 404 is not transient, and /slow was cut off by the timeout handler, retried three more times (each attempt timing out after 400 ms) and finally surfaced as a TimeoutException.

Notice that the retry handler sends the same HttpRequestMessage again. That works for requests without a body. A request with content may have a stream that is already consumed, so production retry code clones the request (or uses a content type that can be replayed, such as StringContent).

A circuit breaker

Retrying a service that is down only wastes time. A circuit breaker notices repeated failures and opens: for a cool-down period every call fails immediately without touching the network. After the cool-down it goes half-open and lets one trial call through. If that succeeds the circuit closes and traffic flows again; if it fails the circuit opens for another cool-down.

stateDiagram-v2
    [*] --> Closed
    Closed --> Open: failures reach the threshold
    Open --> HalfOpen: cool-down elapsed
    HalfOpen --> Closed: trial call succeeds
    HalfOpen --> Open: trial call fails

While open, callers fail fast, which protects both your threads and the struggling service.

The breaker must be shared by all requests to the same destination, so its state lives in a singleton object that the handler receives, not in the handler itself (handlers are created per client lifetime).

C# A circuit breaker handler

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

The output shows the three phases. Calls 1 to 3 reach the network and fail, which opens the circuit. Calls 4 and 5 fail instantly and the network call counter does not move. After the cool-down the trial call goes through, succeeds and closes the circuit.

Use the real libraries in production

Hand-written handlers teach the mechanics, but production code should use a maintained library. The package Microsoft.Extensions.Http.Resilience (built on Polly) adds ready-made pipelines with one line, for example AddStandardResilienceHandler() on an AddHttpClient builder, which combines a rate limiter, total timeout, retry with jitter, circuit breaker and per-attempt timeout with sensible defaults. It is a NuGet package and is not part of the shared framework, so it cannot run in the examples on this site; the code you wrote above is the same idea.

A few more rules for outgoing calls:

  • Always pass a CancellationToken (for example HttpContext.RequestAborted) so abandoned requests stop their outgoing calls too.
  • Set a base address and default headers once, in the registration.
  • Do not log secrets: authorization headers belong in a handler, not in log output.
  • Never ignore status codes. Use EnsureSuccessStatusCode or check them explicitly.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What problem does IHttpClientFactory solve?

    Creating a new HttpClient for every request exhausts sockets, while one static client never refreshes DNS. The factory pools handlers and recycles them periodically, so clients can be created freely.
  2. 2. You add two handlers with AddHttpMessageHandler: first RetryHandler, then TimeoutHandler. Which one is outermost?

    Handlers run in the order they are added, with the first as the outermost. Here the retry handler wraps the timeout handler, so each retry attempt gets its own timeout.
  3. 3. Which response should a retry handler normally NOT retry?

    A 404 is a definite answer that will not change by repeating the request. Connection errors, 5xx, 408 and 429 are transient and worth retrying.
  4. 4. What does an open circuit breaker do?

    While open, the breaker rejects calls without using the network. After the cool-down it goes half-open and lets a trial call decide whether to close or re-open.

Summary

  • Do not create HttpClient per request or keep one forever; register clients with AddHttpClient and let the factory pool handlers.
  • Prefer typed clients: a class with meaningful methods that receives its HttpClient by injection.
  • A DelegatingHandler wraps every request; the first handler added is the outermost.
  • Use a short per-attempt timeout and translate your own timer's cancellation into a TimeoutException.
  • Retry only transient failures and idempotent operations, with exponential backoff, jitter and a low limit.
  • A circuit breaker fails fast when a dependency is down and probes it again after a cool-down.
  • In production use Microsoft.Extensions.Http.Resilience instead of hand-written handlers.

Next: Real-time: WebSockets and Server-Sent Events