The Composed Pipeline

The Composed Pipeline

The patterns above are rarely used alone. In production you layer them, and the order matters. A timeout bounds each individual attempt. A retry wraps that, trying a few times with backoff. A circuit breaker sits across all of it, so a dependency that's truly down trips the breaker instead of grinding through retries forever.

Polly composes these as a single pipeline. This is the listing to copy: retry, breaker, and timeout assembled the way you'd actually run them on the payment gateway call, the most consequential outbound dependency the order service has.

using Polly;
using Polly.CircuitBreaker;
using Polly.Retry;
using Polly.Timeout;

// The pipeline wrapping every charge against the payment gateway.
public static ResiliencePipeline<PaymentResult> BuildPaymentPipeline()
{
    return new ResiliencePipelineBuilder<PaymentResult>()

        // Outermost: trip after sustained failure, fail fast during cooldown.
        .AddCircuitBreaker(new CircuitBreakerStrategyOptions<PaymentResult>
        {
            FailureRatio = 0.5,
            MinimumThroughput = 10,
            SamplingDuration = TimeSpan.FromSeconds(30),
            BreakDuration = TimeSpan.FromSeconds(15),
            ShouldHandle = new PredicateBuilder<PaymentResult>()
                .Handle<PaymentGatewayException>()
                .HandleResult(r => r.IsTransientGatewayError)
        })

        // Middle: retry transient faults with exponential backoff + jitter.
        // Safe only because every charge carries the basket's idempotency key.
        .AddRetry(new RetryStrategyOptions<PaymentResult>
        {
            MaxRetryAttempts = 3,
            BackoffType = DelayBackoffType.Exponential,
            UseJitter = true,
            Delay = TimeSpan.FromMilliseconds(200),
            ShouldHandle = new PredicateBuilder<PaymentResult>()
                .Handle<PaymentGatewayException>()
                .Handle<TimeoutRejectedException>()
                .HandleResult(r => r.IsTransientGatewayError)
        })

        // Innermost: bound each individual attempt.
        .AddTimeout(TimeSpan.FromSeconds(2))

        .Build();
}

Read it from the outside in. The breaker is the gatekeeper; when it's open, nothing below it even runs. Inside, each retry attempt gets its own two-second timeout, and the backoff spaces the attempts out. Register the pipeline once and apply it to every charge against the payment gateway. Other dependencies get their own pipelines: the maps/ETA service from earlier wants a shorter timeout and a fallback rather than three retries, because a slow ETA should degrade instead of blocking. A flaky payment gateway and a slow maps service have different tolerances.

The one rule that ties the room together: the inner timeout must be shorter than the outer ones, or the layers fight each other. A retry budget of three attempts at two seconds each means a worst case near six seconds, so size the breaker and the customer-facing checkout deadline with that in mind.

Steady State

The problem: nothing here failed, but something grew. The couriers stream a GPS ping every few seconds; the live-location cache and the ping table grow without bound until they fill the disk or exhaust the pool at 3am, when no one was watching.

This is the pattern that catches the failures the other six don't. Nygard's Steady State rule (Release It!): for every mechanism that accumulates a resource, there must be a counter-mechanism that reclaims it. Purge GPS pings older than the day's deliveries on a schedule. Cap the courier-location cache with an eviction policy so it holds the latest fix per courier and nothing older. Rotate and expire logs; bound the connection pool with a hard ceiling. A system in steady state can run for months untouched; one without a reclaim path is a time bomb with a slow fuse.

services.AddStackExchangeRedisCache(o =>
    o.Configuration = connectionString);
// Each courier's latest fix expires on its own; stale pings never accumulate.
await _cache.SetAsync($"courier:{courierId}:location", fix,
    new DistributedCacheEntryOptions
    {
        AbsoluteExpirationRelativeToNow = TimeSpan.FromMinutes(5)
    });

What it buys you in production: the quiet kind of uptime, the system that doesn't page you at 3am because the ping table filled the disk. Most of the worst outages aren't dramatic crashes; they're a resource that grew unbounded until it couldn't.

Skip-if: nothing about this one. Every growing resource needs a bound. The skip is only on which mechanism, you might not cap the courier-location cache in Redis this month, but you always need the discipline.

The outage that fells you isn't the one that crashes. It's the one that fills up while you sleep.

Honorable Mentions

Three more that earn a look once the core is in place.

Graceful Degradation turns off non-essential features under load so the core keeps working, hide the "recommended for you" carousel, keep order placement. It's Fallback's strategy applied at the feature level.

Load Shedding drops low-priority requests on purpose when you're saturated, the deliberate twin of Rate Limiting: better to refuse a restaurant's analytics refresh than to fail a customer placing an order.

Failover / Redundancy keeps a standby ready so a dead instance or zone hands off to a live one. It's the infrastructure answer to resilience, and it pairs with the hosting altitude in Ch 9, where redundancy is a deployment property, not application code.

What This Altitude Gives You

Resilience is the altitude where you accept that dependencies fail and design so it doesn't matter. Retry forgives the payment gateway's transient 503; idempotency makes that forgiveness safe, so a retry never double-charges the customer. The breaker fails fast when the maps service is genuinely down. Timeouts bound the wait and a default ETA fills the gap, the bulkhead keeps surge pricing from starving order placement, rate limits cap the public order API, and steady state purges the GPS pings before they fill the disk. Layered through one Polly pipeline on the payment call, they cost you a few dozen lines and buy you the thing that separates a prototype from a product: orders keep going through.

There's a catch, and it's the next chapter's whole job. A resilient system fails quietly. The retry that saved you, the breaker that tripped, the fallback that served stale data, all of it happens silently by design. That's dangerous unless you can see it.

A system that recovers without telling you is a system that's hiding how close it came; next we make it observable.