Observability

Purpose: know what the system is doing in production before a user reports it. Without it, a system is a black box and every incident is a firefight. "It works in my test" is not evidence; production telemetry is.

The three pillars

Metrics tell you something is wrong, traces tell you where, logs tell you why. A senior's system is designed so all three work together: alert on a metric → trace the request → read the log for the detail.

The two frameworks (both matter, differently)

• RED — Rate, Errors, Duration. Per service health. ("Is the notification API healthy?")
• USE — Utilization, Saturation, Errors. Per resource health. ("Is the DB saturated?")

From your calibration, the piece that got added: guarantee metrics — the ones that track your actual promises (delivery rate, p95 latency, queue/DLQ depth). RED/USE keep your infrastructure honest; guarantee metrics keep your product promises honest. A senior monitors both and knows which is which.

SLI / SLO / alerts

• SLI — the measured number (e.g. p95 latency).
• SLO — the target you commit to (p95 < 5s).
• An alert fires when the SLI is about to breach the SLO — before users feel it, which is why you watch queue depth and DLQ growth as early-warning, not just latency after the fact.

Your instinct in the calibration ("queue depth + DLQ depth") was exactly the early-warning layer — the signal that you're about to break a promise, not that you already have.

Where it fits your systems

You run New Relic — the tools are there. The senior habit is the discipline: define the guarantee metrics first (from the SLOs), wire RED/USE under them, then alert on leading indicators, not trailing ones.

Ask yourself

1. If this service breaks at 2 AM, which metric alerts me, and can I trace a sample request from it?
2. Which of my metrics are leading (about to break) vs trailing (already broken)?
3. Can I reproduce "what happened at 9:00 AM on payday" from my logs alone?