Purpose: know exactly what a distributed system can and cannot guarantee — because "it worked on my machine" becomes "which of these promises silently break under a network failure?" This is the mental model under your whole calibration.
CAP: pick two (but really, it's about partitions)
A distributed system stores copies of data across nodes. CAP says you can't simultaneously have all of:
Consistency — every read sees the latest write
Availability — every request gets a response (not necessarily fresh)
Partition tolerance — the system keeps working when the network splits (a partition = nodes can't talk to each other)
The honest reading: partitions happen (network failures are not optional). So on a partition, you must choose: serve stale data (availability) or fail rather than serve wrong data (consistency). That's the whole theorem.
Consistency models in practice
Strong consistency — every read is the latest. Expensive, slower, needs quorum. For money and state you can't get wrong.
Eventual consistency — reads may be stale for a moment, converge later. Cheap, fast, scales. For feeds, likes, notifications, tokens.
Read-your-writes — a middle ground: a user always sees their own writes.
ACID (SQL) vs BASE (NoSQL, eventually-consistent) are just the two ends of this spectrum: strict correctness vs availability-first.
Distributed transactions: 2PC vs Saga
When one operation spans multiple services, you need a distributed transaction:
2PC (two-phase commit): a coordinator asks everyone to prepare, then commits everyone. Strong, but a coordinator failure blocks everything — avoid unless you must.
Saga: break into steps, each with a compensating action (book flight, then cancel if hotel fails). Eventual consistency, resilient, the standard for microservices.
Your reconciliation work is a saga in disguise — compensating, comparing, converging to consistency eventually.
Where idempotency fits
Every "exactly-once" claim in distributed systems is really at-least-once + idempotency (your trx_id dedup). You can't make the network deliver once; you make re-delivery harmless. That single sentence is why your calibration design was correct.
Ask yourself
If this read is stale by 5 seconds, who gets hurt?
If this node can't reach the others, should it fail or serve old data?
Am I trusting "exactly once" without an idempotency key?