Minimal Usage of Logs & Only Insightful Logs
The 3-Line Engineering Rule
Do not log inside tight loops or echo full request/response bodies; log only state transitions, business milestones, and actionable errors.
Excessive logging wastes millions in cloud ingestion costs (Datadog/Splunk), degrades disk I/O, drowns critical alerts in noise, and leaks customer PII.
Use structured JSON with strict log levels (DEBUG for dev, INFO for key lifecycle milestones, WARN for handled anomalies, ERROR for action-required failures). Never log PII.
Architectural Blueprint & Sequence
Hover over nodes to inspect state transitions❌ NOISY: 10,000 logs/sec
log.info('Processing item: ' + id)
✅ INSIGHTFUL: 1 log
log.info('Batch complete', total=10000, failed=2, durationMs=340)
PII Sanitizer
Mask email, phone, card tokens
ERROR = Actionable
If no on-call action needed -> WARN
Key Engineering Takeaways
- Never log inside a loop iterating over thousands of items; emit one summary log after the loop.
- Never log passwords, CVVs, API tokens, or unmasked credit card numbers.
- ERROR level must mean: an engineer needs to be notified or take action.
- Prefer metrics (counters/timers) over logs for high-frequency measurements.
The Problem & Production Reality
Logging everything feels safe during development, but in production, log spew is a silent killer. Disk write saturation, elevated CPU usage during string interpolation, massive cloud bills, and security audits for leaked PII are the direct consequences of unmanaged logging.
Anti-Pattern: Verbose Loop Spew & Leaked Customer Data
Logging inside every iteration of a processing loop floods stdout and introduces severe latency bottlenecks.
// ❌ ANTI-PATTERN: Kills I/O and leaks raw customer PII
for (UserAccount user : userList) { // 100,000 items
log.info("Processing user email: {}, passwordHash: {}, card: {}",
user.getEmail(), user.getPasswordHash(), user.getCardNumber());
// Writes 100,000 lines to disk per minute!
}Production Pattern: Structured Milestone Summary & Metrics
Collect high-cardinality counts into in-memory counters or timers, and emit a single structured summary upon completion.
// ✅ PRODUCTION PATTERN: Batch summary + metrics increment
int processed = 0;
int failed = 0;
long startTime = System.currentTimeMillis();
for (UserAccount user : userList) {
try {
processSingleAccount(user);
processed++;
} catch (Exception ex) {
failed++;
log.warn("Account migration failed: accountId={} reason={}", user.getId(), ex.getMessage());
}
}
// Single insightful summary log
long duration = System.currentTimeMillis() - startTime;
log.info("Account batch processed: total={}, success={}, failed={}, durationMs={}",
userList.size(), processed, failed, duration);
meterRegistry.counter("batch.accounts.processed").increment(processed);
meterRegistry.timer("batch.accounts.duration").record(Duration.ofMillis(duration));Trade-off Analysis & Staff Recommendations
| Aspect | Pros / Strengths | Cons / Hazards | Staff Recommendation |
|---|---|---|---|
| Logs vs Metrics | Metrics (Prometheus counters/gauges) consume near-zero disk and visualize trends instantly. | Metrics lack stack traces and deep contextual payload details. | Use metrics for rate, error count, and durations; use logs for rare anomalies and diagnostic traces. |
| Sampling High-Volume Logs | Prevents log storage exhaustion while maintaining statistical observability. | May miss a specific user's trace during low-probability edge cases. | Sample INFO logs at 1-5% in high-traffic endpoints; log 100% of WARN and ERROR events. |