Metrics and Alerting (USE & RED Methods)
The 3-Line Engineering Rule
Structure service metrics using the RED method (Rate, Errors, Duration) and alert on symptoms that impact users (SLO burn rate), not raw CPU spikes.
Alert fatigue causes on-call engineers to ignore alarms. Waking up an engineer because CPU hit 85% on a batch worker is noise; alerting because checkout 5xx rate crossed 1% saves revenue.
Instrument the RED method (req/sec, 5xx error rate, p95/p99 latency) on every API endpoint. Route alerts to PagerDuty only when customer-facing SLOs are breached.
Architectural Blueprint & Sequence
Hover over nodes to inspect state transitionsApplication Traffic
HTTP Endpoints & Workers
RED Telemetry
Rate, 5xx Errors, p99 Latency
Prometheus TSDB
Scrapes /metrics every 15s
SLO Rule Evaluator
99.9% availability error budget
PagerDuty (On-Call)
Actionable alert + Runbook link
Key Engineering Takeaways
- RED method: Rate (requests/sec), Errors (failed requests/sec), Duration (p95/p99 latency).
- USE method for infrastructure: Utilization (% busy), Saturation (queue depth), Errors.
- Never alert on single point-in-time spikes; use 5-minute rolling averages or multi-window burn rates.
- Every PagerDuty alert must have a linked Runbook detailing remediation steps.
The Problem & Production Reality
Observability is not about collecting billions of useless numbers; it is about detecting real customer degradation before users complain on Twitter. The RED method provides a universal mental model for request-driven architectures, while SLO-based alerting prevents on-call burnout.
Anti-Pattern: CPU-Threshold Alerts and Missing Latency Percentiles
Alerting engineers whenever CPU crosses 80%, or measuring only average latency (which masks the worst 5% of customer experiences).
// ❌ ANTI-PATTERN: Alerting on CPU & average latency
# Alert rule:
alert: HighCPUUsage
expr: container_cpu_usage_percent > 80
for: 1m
# Fires every time a harmless batch job runs, waking engineers at 3 AM!
# Meanwhile, p99 latency spikes to 12s on checkout, but average is 200ms -> Zero alerts!Production Pattern: RED Method with Latency Percentiles and SLO Alerts
Track p95 and p99 percentiles using Prometheus histograms and alert on multi-window error budget burn rates.
// ✅ PRODUCTION PATTERN: Micrometer RED Metrics Instrumentation
@Configuration
public class MetricsConfig {
@Bean
public TimedAspect timedAspect(MeterRegistry registry) {
return new TimedAspect(registry);
}
}
@RestController
@RequestMapping("/orders")
public class OrderController {
@PostMapping
@Timed(
value = "http.orders.checkout",
description = "RED metrics for order checkout",
percentiles = {0.50, 0.95, 0.99}, // Measure p50, p95, p99
histogram = true
)
public ResponseEntity<OrderDto> checkout(@RequestBody CheckoutRequest request) {
return ResponseEntity.ok(orderService.process(request));
}
}
# Prometheus Alerting Rule (Multi-window Burn Rate)
- alert: CheckoutErrorBudgetBurning
expr: (
sum(rate(http_orders_checkout_seconds_count{status=~"5.."}[5m]))
/
sum(rate(http_orders_checkout_seconds_count[5m]))
) > 0.01
for: 2m
labels:
severity: page
annotations:
summary: "Checkout 5xx error rate is > 1% (SLO breached)"
runbook: "https://wiki.internal/ops/runbooks/checkout-5xx"Trade-off Analysis & Staff Recommendations
| Aspect | Pros / Strengths | Cons / Hazards | Staff Recommendation |
|---|---|---|---|
| Average Latency vs p99 Latency | Average is easy to compute and understand. | Average hides catastrophic tail latency where high-value enterprise users wait 15 seconds. | Always monitor and alert on p95 and p99 percentiles. |
| Metric Cardinality | Adding tags like `user_id` gives granular diagnostics. | High cardinality tags (millions of unique user IDs) will crash Prometheus with out-of-memory errors. | Never put user IDs, order IDs, or raw emails into Prometheus metric tags. |