0004-failure-testing-lesson

Learner completed Lesson 2 (failure/chaos testing). Demonstrated understanding: load tests prove happy-path capacity, failure tests prove guarantees — they are different suites (correctly restated after grading); the method is guarantees-first → "what failure breaks this guarantee?" → each answer is a required test; fault injection and chaos discipline (hypothesis, blast radius, abort condition); defensive controls (timeouts, retries + backoff + jitter, circuit breaker, queue-as-buffer); reconciliation-as-test as the continuous never-lost check on the notification system.

Graded correct: choosing the <5s p95 guarantee and identifying that provider throttling threatens it; rewriting "I size for capacity and test with load tests" → "I size for capacity, load-test the happy path, and failure-test the guarantees" (one sharpening: sizing is a distinct provisioning activity from load-testing).

Sharpening applied (high value): separation of latency guarantees — ingest latency (user-facing, must hold <5s p95 even under provider outage) vs delivery latency (internal, degrades gracefully to queue + backoff under a delivery SLA, e.g. 100% eventually delivered / 99% within 15 min, page at 10 min). Rule: write the degradation policy alongside every guarantee, or the guarantee cannot be tested or defended.

AZ-loss drill (graded, one misconception corrected): blast radius expectation = surviving AZs absorb traffic, Kafka survives (3× replication), queue absorbs in-flight work, no alert lost. Corrected: the headline assert of an AZ-loss drill is zero loss, not p95; one AZ loss should be invisible to a user if headroom is provisioned (autoscale by queue depth, not CPU). Abort condition corrected: capacity limits are mitigation (scaling continues the drill), the abort is the line past which recovery is impossible — a missing delivered record in reconciliation, or a queue backlog that exceeds the delivery SLA.

Implications: learner learns best through "your-own-system" drills and honest grades; next lessons — (3) estimation reps, then (4) consistency trade-offs.