In your calibration, you hit the wall here — you said it yourself: "I didn't know the strategy for ordering is multiple partitions." This lesson closes that gap. It's the single highest-leverage idea in your whole stack, and you already use it every day: Kafka is a distributed log of partitions.
You needed per-user ordering at 60M users. A user swipes, transfers, pays — the alerts must arrive in order. So the naive answer: process everything in one queue, in order. Ordering ✓. But now one lane handles every user on Earth — a single point, zero parallelism, and one slow message blocks everyone.
Rule 1 — you cannot have both global order and parallelism. They're the same resource. You must give one up.
A partition is a single, strictly-ordered, independently-consumed lane. The trick of distributed systems is that many lanes run in parallel, and each lane is individually ordered.
events (user_id = 42) lanes
| ┌─────────────┐
hash(user_id) ────┼────────────────→ │ partition 0 │ ← serial, in order
42 % 3 = 0 │ └─────────────┘
│ ┌─────────────┐
hash(user_id) ────┼────────────────→ │ partition 1 │ ← serial, in order
7 % 3 = 1 │ └─────────────┘
│ ┌─────────────┐
hash(user_id) ────┼────────────────→ │ partition 2 │ ← serial, in order
99 % 3 = 0 │ └─────────────┘
| Property | What it means | Cost / trade-off |
|---|---|---|
| Ordering | All events with the same key land in the same partition, processed in arrival order. | Order is per partition, never global. |
| Parallelism | Different partitions are processed concurrently — by different workers. | One worker per partition; a hot key caps its own lane's throughput. |
| Locality | Same-key events co-locate: state, retries, and effects can stay per-partition. | Scaling = adding partitions + rebalancing (order preserved per key). |
Your calibration key was user_id: hash it, and every user's alerts go to one partition → per-user order holds, while 60M users spread across hundreds of partitions → the system scales. Partition by what must stay in order. Your banking event stream already does this — that's what your Kafka partitioning key is for.
Within a partition, a failed message creates a fork. Block it → one poison message freezes the user's whole stream (ordering held, availability lost). Skip it and retry later → the stream moves, the failed alert still eventually delivers, possibly out of order (availability held, strict order slipped). In your calibration you chose the second — correctly, because you were designing for guarantees, not strict order.
Rule 2 — the trade-off is always: block-for-order, or skip-and-retry. Decide by guarantee, not by default. Banking: never block a newer alert behind a stuck older one.
Q1. Kafka guarantees order:
Q2. Global order across all users forces:
Q3. A message fails; the user's stream should:
Q4. The same hash key always maps to:
Primary source: Designing Data-Intensive Applications, Martin Kleppmann, Chapter 6 "Partitioning" — the canonical treatment of this exact trade-off. You'll see "hash partitioning, skewed keys, hot spots" and why hash(user_id) beats naive round-robin.
Secondary: your own Kafka — the partitions in your 821k-events/day pipeline already embody everything above.