Skip to content
Tutorials Center
My learning

Topics

Lesson 147 of 182 ยท ASP.NET Core

Resilience with Polly

Build resilient .NET apps - transient faults, retry with backoff and jitter, timeouts, circuit breakers, hedging and fallbacks with Polly v8, plus runnable hand-written versions of each idea.

Every service you call can fail: the network drops a packet, a server restarts, a database is briefly overloaded, a rate limiter answers 429. Most of these failures are transient, meaning that the same request a moment later succeeds. An application that treats every blip as a fatal error is fragile; one that retries blindly can make an outage worse. Resilience is the discipline of reacting to failure on purpose: retry when it helps, give up when it does not, and protect both your own app and the struggling service.

Polly is the standard .NET library for this. Version 8 organizes everything as a resilience pipeline, an ordered chain of strategies (retry, timeout, circuit breaker, hedging, fallback, rate limiter) wrapped around the code you want to protect. In this lesson you will meet the strategies one at a time. Polly is a NuGet package that our editor does not have, so the Polly code is shown read-only, and next to each strategy you get a small runnable implementation of the same idea, so the behavior is not magic. The HTTP side of this was introduced in Caching and HttpClientFactory, and the retry building block in Exception Filters.

Transient faults and what is safe to retry

Before writing any retry code, answer two questions.

Is the failure transient? Timeouts, connection resets, 502, 503, 504 and 429 often are. A 400, 401, 404 or a validation error will fail every time, so retrying only wastes time and load.

Is the operation safe to repeat? A GET is, because it does not change anything. A POST /payments may have succeeded before the connection dropped; a retry could charge twice. Retry non-idempotent operations only when the server supports an idempotency key that lets it recognize duplicates.

flowchart TD
    A["Call fails"] --> B{"Transient error?<br/>(timeout, 503, 429)"}
    B -- "No (400, 404, bug)" --> X["Fail now"]
    B -- Yes --> C{"Safe to repeat?<br/>(idempotent or has key)"}
    C -- No --> X
    C -- Yes --> D{"Attempts left and<br/>time budget left?"}
    D -- No --> X
    D -- Yes --> E["Wait (backoff + jitter)"] --> F["Try again"]

Retry is a decision tree, not a loop: classify the error, check safety, then check the budget.

Retry with exponential backoff and jitter

Retrying immediately usually hits the same problem. Exponential backoff waits longer after each failure (for example 1 s, 2 s, 4 s) so a recovering service gets room to breathe. Jitter adds randomness so that a thousand clients that failed together do not all retry in the same instant (a "thundering herd").

With Polly the pipeline is built once and reused:

C#
// Read-only: needs the Polly NuGet package (dotnet add package Polly)
using Polly;
using Polly.Retry;

ResiliencePipeline pipeline = new ResiliencePipelineBuilder()
    .AddRetry(new RetryStrategyOptions
    {
        MaxRetryAttempts = 4,
        Delay = TimeSpan.FromSeconds(1),
        BackoffType = DelayBackoffType.Exponential,   // 1s, 2s, 4s, 8s
        UseJitter = true,                             // randomize each delay a little
        ShouldHandle = new PredicateBuilder()
            .Handle<HttpRequestException>()
            .Handle<TimeoutRejectedException>(),
        OnRetry = args =>
        {
            Console.WriteLine($"Retry {args.AttemptNumber + 1} after {args.RetryDelay}");
            return default;
        }
    })
    .Build();

var result = await pipeline.ExecuteAsync(async ct => await client.GetStringAsync(url, ct), cancellationToken);

The same logic is only a few lines when written by hand, which makes it easy to see what each setting does. In this version the delay function is injected, so the example can record the delays instead of really waiting, and the output is the same every time:

C# Retry with exponential backoff, written by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

The when clause is an exception filter: it keeps the exception (and its stack trace) intact when the retry budget is spent. Production code adds jitter (multiply each delay by a random factor between, say, 0.5 and 1.5), a cancellation token so shutdown interrupts the wait, and an overall time limit. If the service sends a Retry-After header, honor that delay instead of your own.

Timeouts

A call without a timeout can hang for as long as the other side wants, holding a thread, a connection and a user's patience. Always set a limit, and set it at two levels: per attempt (so a stuck attempt is abandoned and retried) and overall (so retries cannot add up to minutes).

C#
// Read-only: needs the Polly NuGet package
var pipeline = new ResiliencePipelineBuilder()
    .AddTimeout(TimeSpan.FromSeconds(30))          // outer: whole operation including retries
    .AddRetry(new RetryStrategyOptions { MaxRetryAttempts = 3 })
    .AddTimeout(TimeSpan.FromSeconds(2))           // inner: one attempt
    .Build();

Order matters. Strategies run from the outside in, in the order you add them. Here the 30-second timeout wraps the retry, which wraps the 2-second attempt timeout. Put them the other way round and the overall timeout would restart on every attempt. Timeouts work by cancelling a CancellationToken, so your code must pass that token down to the real I/O call.

Circuit breaker

When a dependency is down, retrying makes things worse: every request waits through several doomed attempts and adds load to a service that is trying to recover. A circuit breaker notices repeated failures and starts failing immediately without calling the dependency, then carefully tests whether it has recovered. It has three states:

stateDiagram-v2
    [*] --> Closed
    Closed --> Open: failures reach the threshold
    Open --> HalfOpen: break duration elapsed
    HalfOpen --> Closed: trial call succeeds
    HalfOpen --> Open: trial call fails

Closed passes calls through, Open rejects them instantly, Half-Open lets one trial call decide whether to close again.

In Polly:

C#
// Read-only: needs the Polly NuGet package
.AddCircuitBreaker(new CircuitBreakerStrategyOptions
{
    FailureRatio = 0.5,                          // open when 50% of calls fail ...
    MinimumThroughput = 10,                      // ... but only after at least 10 calls
    SamplingDuration = TimeSpan.FromSeconds(30), // measured over this window
    BreakDuration = TimeSpan.FromSeconds(15),    // stay open this long
})

When the circuit is open, calls throw BrokenCircuitException. Callers should handle it by using a fallback or a clear "temporarily unavailable" response, not by retrying in a tight loop. Here is a simplified version (consecutive failures instead of a ratio) driven by a fake clock, so you can watch every transition:

C# A circuit breaker with a fake clock

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

Calls 4 and 5 never touch the service: that is the protection. After the break duration, exactly one trial call goes through; the first trial fails, so the breaker re-opens, and the second succeeds, so it closes. Real implementations also need thread safety (the version above is single-threaded for clarity), which is a good reason to use a library.

Hedging

Retries wait for a failure. Hedging fights slowness: if the first call has not answered after a short delay, start a second identical call in parallel, take whichever finishes first, and cancel the other. It trades extra load for a shorter tail latency, and it only makes sense for safe, idempotent calls to a service that has several healthy instances.

C#
// Read-only: needs the Polly NuGet package
.AddHedging(new HedgingStrategyOptions<HttpResponseMessage>
{
    MaxHedgedAttempts = 1,                      // at most one extra call
    Delay = TimeSpan.FromMilliseconds(200),     // start it if no answer by then
    ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
        .Handle<HttpRequestException>()
        .HandleResult(r => !r.IsSuccessStatusCode),
})

The idea in plain C# is a race between tasks, using Task.WhenAny and cancellation:

C# Hedged requests: the second attempt wins

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

Without hedging this request would have taken the full 600 ms. With it, the answer arrives after roughly the hedge delay plus the healthy attempt's time. In a real system you set the hedge delay near the 95th percentile of normal latency, so only the slowest few percent of calls produce a second request.

Fallback

When everything else fails, a fallback supplies a degraded but useful answer: cached data, a default value, or a friendly message. Polly has AddFallback; by hand it is a try/catch that returns a substitute. Use it for non-critical features (recommendations, avatars) and be honest in the UI that the data may be stale. Never use a fallback to hide a failure of something that must be correct, such as a payment.

Resilience for HTTP calls in ASP.NET Core

You rarely build pipelines for HTTP by hand. The package Microsoft.Extensions.Http.Resilience (built on Polly v8) plugs them into IHttpClientFactory:

C#
// Read-only: needs Microsoft.Extensions.Http.Resilience
builder.Services.AddHttpClient<WeatherClient>(client =>
        client.BaseAddress = new Uri("https://weather.example"))
    .AddStandardResilienceHandler(options =>
    {
        options.Retry.MaxRetryAttempts = 3;
        options.AttemptTimeout.Timeout = TimeSpan.FromSeconds(2);
        options.TotalRequestTimeout.Timeout = TimeSpan.FromSeconds(10);
        options.Retry.DisableForUnsafeHttpMethods();      // no automatic POST/PUT/DELETE retries
    });

AddStandardResilienceHandler stacks, from outside in: a rate limiter, a total timeout, retry, a circuit breaker and a per-attempt timeout, with defaults that suit most APIs. AddStandardHedgingHandler is the variant for services with several endpoints. For non-HTTP code, such as a database call or a message-queue publish, register a named pipeline with services.AddResiliencePipeline("db", builder => ...) and inject ResiliencePipelineProvider<string> to fetch it.

To see where the HTTP handler idea comes from, here is a retry as a hand-written DelegatingHandler against a deliberately flaky server (it answers 503 twice, then 200). The handler retries only idempotent requests, so a POST is not repeated:

C# A retrying DelegatingHandler against a flaky server

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

The caller wrote a single GetAsync and got the data, even though the server failed twice; the extra attempts were invisible. The POST got its single failure back because repeating a payment is not safe. A library's handler adds jitter, Retry-After, per-attempt timeouts and telemetry on top of this skeleton.

Resilience in background work

The same strategies apply to code that runs without a user waiting, such as a hosted BackgroundService that polls an API or drains a queue. A few extra rules help there:

  • Never let the worker die silently. An unhandled exception in ExecuteAsync stops the host on current .NET versions. Catch per-iteration errors, log them, back off and continue.
  • Respect the stopping token. Pass stoppingToken to every delay and call so shutdown is prompt; treat OperationCanceledException from that token as a normal exit.
  • Use longer, bounded backoff. A background job can wait a minute between attempts, but cap the delay so the worker still recovers quickly once the dependency returns.
  • Dead-letter poison messages. If one message fails every time, move it aside after N attempts instead of blocking the queue forever.
  • Observe it. Log every retry and every circuit state change and expose counters; a breaker that is open for an hour should page someone.

Limits and good practice

  • Retries multiply load. If three layers each retry three times, one user action can become 27 calls. Retry at one layer, usually the outermost that has full context.
  • Tune from data. Defaults are a starting point; measure real latencies and failure rates before choosing timeouts and thresholds.
  • Test the failure paths. Use a flaky fake like the ones above, and the techniques from Integration Testing ASP.NET Core, to prove that your app returns sensible results while a dependency is down.
  • Resilience is not a substitute for capacity. Backoff and breakers protect a service during a short problem; they do not fix a service that is permanently undersized.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which failure is the best candidate for an automatic retry?

    A 503 is a typical transient fault, and GET is safe to repeat. The other responses will fail again until the request or the credentials change.
  2. 2. What is the purpose of jitter in a retry policy?

    Without randomness, clients that failed together retry together, which can overload a recovering service again.
  3. 3. A circuit breaker is in the Open state. What happens to a new call?

    While open, the breaker fails fast. That protects the service and frees your threads. After the break duration it allows a trial call in Half-Open state.
  4. 4. When is hedging an appropriate strategy?

    Hedging starts a duplicate call when the first is slow, so the operation must be safe to run twice and the service must have capacity.
  5. 5. Why does the order in which you add strategies to a Polly pipeline matter?

    Each strategy wraps the ones added after it. Swapping a total timeout and a per-attempt timeout changes what each one measures.

Summary

  • Transient faults are common; classify errors and only retry the ones that can succeed later, and only operations that are safe to repeat.
  • Use exponential backoff with jitter, a maximum number of attempts and an overall time budget; honor Retry-After.
  • Always set timeouts, per attempt and overall, and pass cancellation tokens to the real I/O.
  • A circuit breaker (Closed, Open, Half-Open) fails fast while a dependency is down and probes for recovery.
  • Hedging cuts tail latency for idempotent calls by racing a second attempt; fallbacks return a degraded answer.
  • In ASP.NET Core use AddStandardResilienceHandler (Polly v8 inside) for HttpClient, and AddResiliencePipeline for other dependencies.
  • Background services must catch per-iteration errors, respect the stopping token and log every retry and breaker change.

Next: Data Access in .NET