Retry for External API Calls & Circuit Breakers
The 3-Line Engineering Rule
Never retry blind POST requests without idempotency keys, and always protect external calls with a circuit breaker and backoff jitter.
Blind retries on payment or mutation APIs cause double charges and thundering herds that take down recovering third-party services.
Combine Circuit Breaker (fail-fast on elevated error rates) + Exponential Backoff with Full Jitter + strictly enforce Idempotency-Key headers on mutating POST/PUT requests.
Architectural Blueprint & Sequence
Hover over nodes to inspect state transitionsYour Application
POST /v1/charges
Circuit Breaker
CLOSED (Healthy)
Retry Engine
Jitter + IdempotencyKey
Partner Gateway
Returns 503 Service Unavailable
Degraded Fallback
Queue for async reconciliation
Key Engineering Takeaways
- Only retry transient HTTP codes (408, 429, 502, 503, 504, connect timeouts); never retry 400, 401, 403, 404.
- Always attach an `Idempotency-Key` (UUIDv4) to all retryable mutating requests.
- Use Full Jitter: sleep = random(0, min(max_backoff, base * 2^attempt)).
- Trip the Circuit Breaker to OPEN when error rates cross 50% to prevent connection pool exhaustion.
The Problem & Production Reality
Network requests over the internet will fail. The difference between a resilient system and a fragile one is how retries are orchestrated. A naive retry policy turns a temporary blip into an outage by hammering the downstream dependency with thousands of synchronized requests.
Anti-Pattern: Immediate Unbounded Retries on Mutating Calls
Retrying HTTP POST without an idempotency key can charge a customer's credit card twice if the first request succeeded on the server but timed out returning the response.
// ❌ DANGEROUS: No backoff, no jitter, no idempotency key
for (int i = 0; i < 3; i++) {
try {
return restTemplate.postForObject("https://pay.stripe.com/charges", request, Response.class);
} catch (RestClientException ex) {
// Immediate retry: If first call succeeded on Stripe but timed out, customer charged 3x!
}
}Production Pattern: Resilience4j Circuit Breaker + Jittered Retry
Configure explicit retryable status codes, attach an immutable Idempotency-Key per business transaction, and fail-fast with a circuit breaker when the remote service is down.
// ✅ PRODUCTION PATTERN: Circuit Breaker + Retry with Idempotency Key
@Service
public class ResilientPaymentClient {
private final RestClient restClient;
@CircuitBreaker(name = "paymentGateway", fallbackMethod = "paymentFallback")
@Retry(name = "paymentRetry")
public PaymentResponse chargeCard(PaymentRequest request, String idempotencyKey) {
return restClient.post()
.uri("/v1/charges")
.header("Idempotency-Key", idempotencyKey)
.body(request)
.retrieve()
.onStatus(HttpStatusCode::is5xxServerError, (req, res) -> {
throw new TransientGatewayException("Remote 5xx gateway error: " + res.getStatusCode());
})
.body(PaymentResponse.class);
}
public PaymentResponse paymentFallback(PaymentRequest request, String key, Throwable ex) {
log.warn("Payment gateway circuit OPEN or exhausted retries. Falling back to pending queue. Reason: {}", ex.getMessage());
return PaymentResponse.queuedForReconciliation(request.getOrderId(), key);
}
}Trade-off Analysis & Staff Recommendations
| Aspect | Pros / Strengths | Cons / Hazards | Staff Recommendation |
|---|---|---|---|
| Exponential Backoff vs Jitter | Jitter distributes retry bursts evenly across time. | Slightly less predictable execution windows. | Always use Full Jitter for network calls: t = rand(0, base * 2^attempt). |
| Circuit Breaker Thresholds | Prevents waiting for slow timeouts when a dependency is dead. | Overly sensitive thresholds trip during minor hiccups. | Set minimum calls to 20, failure rate to 50%, and wait duration in OPEN to 15-30s. |