Purpose: two separate skills that get mixed up — (1) growing capacity as load rises, and (2) shipping code without breaking what's running. Both are senior day-jobs.
| Vertical (bigger box) | Horizontal (more boxes) | |
|---|---|---|
| How | More CPU/RAM on one server | More servers behind a load balancer |
| Good for | Stateful/DB workloads, quick bumps | Stateless services, burst traffic |
| Ceiling | Hardware + you keep a SPOF | Your statelessness and partitioning |
| Cost | Superlinear price spikes | Linear-ish, elastic |
The rule that makes horizontal scaling work: statelessness. A service is horizontally scalable if any instance can serve any request. That's why your notification worker keeps its state in Kafka (the changelog), not in itself — the state store is rebuildable, so the instance is disposable.
Autoscaling grows/shrinks instance count from signals: CPU, request rate, or queue depth (the best signal for pull-based workers — the queue is literally the backlog of work). Autoscaling on CPU for a queue consumer is too slow; scale on the queue.
| Strategy | How | Risk |
|---|---|---|
| Rolling | New version replaces instances gradually | Mixed versions briefly |
| Blue-green | Two full environments, flip traffic | Doubles infra |
| Canary | New version gets 5% traffic, then ramp | Needs good metrics + fast rollback |
All three exist to answer one question: if this deploy is broken, how fast do I recover? That's why every deploy needs a rollback plan and a health check to catch a bad version early — not just a success check on "it started."
Graceful shutdown: when a worker is told to stop (deploy, scale-down), it must stop taking new work but finish current work, then tell the orchestrator it's done. Killing a Kafka consumer mid-processing is exactly how you get redelivery (and why your idempotency key matters).
Your marketing worker "separate deployment so it doesn't hinder trx" was an isolation decision — the blast-radius version of scaling. And your Part 5 answer ("set pod config, confident it's okay") was scaling — the missing half was failure and deploy behavior, which is what Lesson 2 targets.