Every technical term used in this workspace so far, defined tightly. This is the canonical vocabulary — lessons will use these terms and avoid their aliases.
At-least-once: The message is never lost, but may be delivered more than once. The baseline guarantee of queues and Kafka; duplicates are expected and must be handled. Avoid: Reliable delivery (too vague)
Exactly-once: Each message is processed exactly once — no loss, no duplicates. Not free: it requires idempotency keys, and Kafka transactions only cover effects inside Kafka, never external calls (e.g. a push provider). Avoid: Perfect delivery
Idempotency key:
A unique business identifier (e.g. trx_id) carried on the message and checked before acting; applying the operation twice with the same key has the same effect as once.
Avoid: Dedup key (acceptable alias), unique ID
Deduplication (dedup): Skipping an already-processed message by looking up its idempotency key in a "seen" store.
Best-effort: No delivery or latency guarantee; the system may drop the work under load. The load-shedding tier (e.g. marketing pushes). Avoid: Fire and forget (implies loss is fine, weaker)
Outbox pattern: Writing the outbound message in the same database transaction as the business event, then relaying it — so the message and the event can't diverge. An alternative to consuming the event stream directly.
Message queue / broker: The durable buffer between producer and consumer; decouples them, absorbs bursts and downstream slowness. Avoid: MQ, message bus (aliases)
Producer / consumer: The writer to and reader from a topic/queue.
Consumer group: A set of consumers sharing a topic; each partition is assigned to exactly one member of the group at a time. Different groups can each read the same partitions independently.
Topic: A named category of messages. Contains partitions. A topic is not a lane — a lane is a partition.
Backpressure: A slow downstream consumer applying pressure upstream; a queue absorbs it by buffering while keeping the producer unblocked. Avoid: Bottleneck (that's the symptom, not the mechanism)
DLQ (Dead Letter Queue): A holding queue for messages that keep failing (poison messages); quarantine for inspection instead of infinite retry.
Poison message: A message that can never be processed successfully, causing repeat failures until it's DLQ'd.
Retry / exponential backoff: Re-attempting a failed operation; backoff = increasing the delay between attempts.
Partition: A single, strictly-ordered, independently-consumed lane. The unit of ordering, parallelism, and state locality. Ordering is guaranteed within a partition only. Avoid: Shard (means data partition in storage, not stream), segment
Partition key (hash key): The key hashed to route a message to a partition. The same key always lands in the same partition — so it decides what stays in order and what co-locates. Avoid: Routing key, partition id
Ordering guarantee: Order within a partition, never across partitions. Global order forces exactly one partition (and kills parallelism). Avoid: FIFO order (provider-specific term)
Hot key / hot partition / skew:
One partition saturated while others idle, because a single key produces a disproportionate share of messages. The failure mode of hash(user_id) when one user bursts.
Salting: Appending a random suffix to the partition key to spread a hot key across partitions. Breaks ordering for that key — only for order-tolerant work (marketing, not transaction alerts).
State store: Per-partition key-value state held locally by the consumer (e.g. the "seen" set for dedup). Local and fast; backed by a changelog for durability.
Changelog (compacted): A Kafka topic recording state-store changes. On a worker takeover, the new consumer replays it to rebuild state. Compacted = only the latest value per key is kept.
Rebalance / replay-on-takeover: When a consumer dies, its partitions are reassigned; the new owner replays the changelog to restore state before processing.
Offset: The consumer's cursor position within a partition; committing it marks work as done. Commit-before-crash causes redelivery.
Kafka Streams / Flink: Stream-processing frameworks providing the state store + changelog + rebalance machinery automatically — running in your own 821k-events/day pipeline.
Availability Zone (AZ): One isolated failure domain — independent power, network, and physical datacenter. Surviving an AZ outage needs multi-AZ instances + replicated state. Avoid: Region (a region contains many AZs)
SPOF (Single Point of Failure): One component whose failure takes the whole system down. Eliminated by redundancy across failure domains.
Replication: Keeping copies of state across failure domains; the expensive 2–3× part of multi-AZ.
Load shedding: Dropping low-priority work under pressure to protect critical work. "Best-effort" traffic is the shedding tier. Avoid: Throttling (rate-based, not priority-based)
Provider ack ≠ delivery: A push provider accepting your request means "we'll try," not "it's on the phone." Per-message delivery is unverifiable; you monitor aggregate telemetry and use SMS/email as a backstop for critical alerts.
Back-of-envelope estimation: Deriving QPS, storage, and bandwidth from requirements with rough assumptions, before any code. Six steps: demand → daily volume → avg QPS → peak (burst factor) → storage/retention → bandwidth → sanity-check.
Peak vs average QPS / burst factor: Traffic isn't flat; peak = average × burst factor. The burst factor is where uncertainty concentrates — it drives whether you need a queue at all.
Rate vs volume: Volume = total amount over a period; rate = volume ÷ time (QPS). Spreading work over time changes the rate, never the volume. (The most common estimation error.)
Sensitivity analysis / blast radius: Finding which estimate error changes the design most. Always re-check the high-blast-radius, cheap-to-measure assumptions (burst factor, record size) — measure instead of guess.
SLO / percentile latency (p95, p99): Service Level Objective; p95 = 95% of requests under the target. Target p95; monitor p99 (the tail is dominated by factors outside your control).
RED / USE metrics: Rate, Errors, Duration (service health) / Utilization, Saturation, Errors (resource health). Generic infra view — not the same as guarantee metrics (delivery rate, p95, queue/DLQ depth).
Delivery telemetry: Aggregate provider delivery statistics; the only way to "verify" push delivery at scale.
Dead token hygiene: Deactivating device tokens the provider rejects, so retries aren't burned on invalid devices forever.
Reconciliation: Comparing two records of truth to find discrepancies (e.g. your SFTP/API job). Exactly-once comparison becomes a per-partition local problem once the stream is partitioned by the event identity.
Priority queue: A queue with tiers; high-priority items (transaction alerts) are drained first, low-priority (marketing) rate-limited behind them.
Reverse proxy: The server that sits in front of your backends and forwards traffic to them; the usual shape of a load balancer. Not to be confused with a forward proxy (client-side).
L4 / L7 load balancing: L4 balances at the TCP level (transport); L7 at the HTTP level (can inspect URLs, headers, cookies). L7 enables smarter routing (e.g. by path), L4 is faster.
Round-robin / least-connections: Simple balancing policies — next server in turn, or the server with fewest active connections. Picking depends on whether requests have equal cost.
Consistent hashing: Hashing that minimizes remapping when servers are added/removed — only a small slice of keys move. Used for sticky sessions and cache/shard affinity. Same hot-key skew as stream partitions.
Health check / connection draining: Health check = probing a backend so the LB stops sending to dead ones; draining = letting in-flight requests finish before a server is removed. What makes rolling deploys safe.
Sticky session: Pinning a client to one backend (via hash or cookie) so its session state stays local. Works great until the pinned backend is hot or dies.
Cache-aside / write-through / write-back: Three write strategies — app loads on miss (lazy), every write also writes cache (fresh), or writes go to cache and flush later (fast, risky). See the caching fundamentals page.
TTL (time-to-live): How long a cache entry lives; the safety valve that bounds staleness even if invalidation fails.
CDN (Content Delivery Network): A global network of edge caches for public/static content, serving users from the closest edge. The "cache closest to the user" layer.
Cache stampede / thundering herd: An entry expires and thousands of requests all miss and hit the source at once. Fixed by singleflight (one loader, others wait) or stale-while-revalidate.
Invalidation: Removing/updating a cache entry when the source changes. The hard part of caching; TTL is the fallback when you can't invalidate reliably.
SQL vs NoSQL: Relational (fixed schema, joins, strong transactions) vs non-relational (flexible docs, horizontal scaling). Choice driven by the guarantees the data needs, not fashion.
Index: A query-shaped shortcut in a database; speeds reads, slows writes. Every schema should be read backward from its query patterns.
Leader-follower (read replicas): One node accepts writes, followers serve reads; adds read capacity and failover. Cost: replica lag — followers can be a moment behind.
Replica lag: The delay between a write to the leader and it appearing on a follower. Breaks read-your-writes if not handled.
Sharding (shard key): Splitting rows across nodes by a shard key. The key is permanent and must distribute evenly — the storage twin of stream partitions. Range vs hash sharding trade scans for balance.
CAP theorem: Consistency, Availability, Partition tolerance — pick two. The honest version: partitions happen, so you choose stale but serving (availability) or fail rather than wrong (consistency).
ACID / BASE: The two ends of the consistency spectrum. ACID = strict (transactions, strong consistency). BASE = basically available, eventually consistent (the NoSQL end).
Strong / eventual consistency: Every read sees the latest write, vs reads that may be stale and converge later. Money is strong; feeds/notifications are eventual.
Read-your-writes: The middle-ground guarantee that a user always sees their own writes, even on a replicated system.
2PC (two-phase commit): A coordinator atomically commits a write across multiple systems. Strong but fragile — a coordinator failure blocks everything; avoid where a saga works.
Saga: A distributed transaction as steps + compensating actions (book flight, cancel if hotel fails). Eventually consistent, resilient, the microservices standard. Reconciliation work is a saga in disguise.
Vertical / horizontal scaling: Bigger machine vs more machines. Horizontal requires statelessness; vertical keeps a SPOF and hits a ceiling.
Statelessness: Any instance can serve any request; state lives elsewhere (DB, queue, changelog). The precondition for horizontal scaling and killing instances freely.
Autoscaling: Growing/shrinking instance count from signals — CPU, rate, or queue depth (the right signal for queue workers).
Blue-green / canary / rolling: Deploy strategies: two environments with a traffic flip, a small-percentage ramp, or gradual replacement. All answer: how fast do I recover from a bad deploy?
Graceful shutdown: A worker stops taking new work, finishes in-flight work, then exits — preventing mid-processing kills that cause redelivery.
Metrics / logs / traces: The three pillars. Metrics = numbers/trends (something is wrong), traces = one request's path (where), logs = discrete events (why).
SLI / SLO: SLI = the measured number (p95 latency); SLO = the target you commit to (p95 < 5s). Alerts fire when the SLI is about to breach the SLO.
Guarantee metrics: The metrics that track your actual product promises (delivery rate, latency percentile, queue/DLQ depth) — distinct from RED/USE, which track infrastructure health.
Alert: A notification that fires when a metric crosses a threshold. Leading alerts (queue depth growing) warn before users feel it; trailing alerts (latency already bad) report after.