Skip to content

10. CAP Theorem Reality — Not a personality test for databases

~18 min read. CAP matters only when the network breaks, and that is exactly when hand-wavy answers become dangerous.

Built on the ELI5 in 00-eli5.md. The restaurant now has multiple kitchens, and one order ticket may reach some counters but not others — so the house rules must say whether to wait, reject, or serve with uncertainty.

1) What CAP actually talks about

See. CAP is about three things. Consistency. Availability. Partition tolerance. But not in the vague way people say it. Consistency here means every read sees the latest successful write, or a clear failure instead of stale success. Availability means every request to a non-failed node gets a response. That response may be stale. Partition tolerance means the system keeps operating despite network splits. Look at the picture. 

client A ──→ node 1 ──x── node 2 ←── client B
                     partition
Now what is the problem? During the partition, node 1 and node 2 cannot coordinate. If client A writes to node 1, what should client B read from node 2? Latest truth? Old truth? An error? That is the CAP moment. Not normal operation. Not benchmark day. Partition day. Simple, no? So CAP does not say, "pick any two forever." It says that when a partition happens, you must choose whether to preserve consistency by refusing some requests, or preserve availability by answering despite incomplete coordination. That is the real statement.

2) Why CA is not a real distributed option

People love saying, "We choose CA." Look. If the system is truly distributed, packets can be delayed or lost. That means partitions are possible. Once partitions are possible, you are already in the P world. So there is no magical "ignore P" choice. 

single machine:      CA is possible
multiple nodes:      partition can happen
partition happens:   must choose C or A for that operation
CA exists only if we pretend the network never breaks, or if the whole system is really one machine. That is why serious answers say: "For this operation, under partition, we prefer C." or "For this operation, under partition, we prefer A." See the difference? We stopped labelling the whole restaurant. We started labelling a path. A payment confirmation may prefer C. A feed refresh may prefer A. A profile avatar read may prefer A. A seat reservation write may prefer C. So what to do in interviews? Name the operation. Name the partition event. Name the sacrificed side. That is much stronger than saying "this database is AP." Because real systems expose multiple APIs with multiple choices.

3) Real systems make different choices

Cassandra is the classic AP-leaning example. If replicas are partitioned, you can keep accepting writes on reachable replicas, then reconcile later. That preserves availability more often. You accept temporary divergence. Spanner is a CP-leaning example. It spends serious engineering effort on coordination and time bounds, so transactions preserve strong global ordering. During some failures, it would rather refuse or delay work than lie. DynamoDB is the useful practical example. It gives tunable read choices. You can ask for eventual reads on one endpoint, and stronger reads on another endpoint. Look at the comparison. 

┌────────────┬─────────────────────────────┬──────────────────────────┐
│ system     │ during partition leans      │ trade you feel           │
├────────────┼─────────────────────────────┼──────────────────────────┤
│ Cassandra  │ availability                │ stale/conflicting reads  │
│ Spanner    │ consistency                 │ higher latency / refusal │
│ DynamoDB   │ per-operation tuning        │ must choose path by path │
└────────────┴─────────────────────────────┴──────────────────────────┘
Now what is the problem? Students hear "AP database" and think, "Great, consistency solved by background magic." No. Conflict resolution is part of the application story too. You must know what happens to counters, carts, likes, or reservations. Likewise, hearing "CP database" and thinking, "Great, no tradeoff." Also no. You are paying in latency, complexity, or rejected operations. The database label is the start of design. Not the end.

4) Worked example: one system, two operations, two choices

Suppose we run a global shopping service in Mumbai, Frankfurt, and Virginia. Local region read latency is 12 ms. Cross-region round trip for coordination is 90 ms. We have two operations. Operation one is product-detail read. Operation two is inventory reservation at checkout. Look at normal conditions first. If product reads use local eventual replicas, latency is about 12 ms. If inventory reservation waits for cross-region quorum, latency includes coordination. Assume one local write step is 15 ms. Total strong reservation latency is: 15 + 90 = 105 ms. Good. Now a partition isolates Mumbai from Frankfurt and Virginia. What should product-detail read do? Usually answer locally. Maybe the stock badge is briefly stale. That is acceptable for many browse pages. So this operation leans A during partition. What should inventory reservation do? If Mumbai cannot reach quorum, it should reject or retry rather than oversell. So this operation leans C during partition. Simple, no? Now see PACELC. PACELC says: if there is a Partition, trade Availability vs Consistency. Else, when there is no partition, trade Latency vs Consistency. Our example shows both halves. Under partition: - product read chose A over C - inventory reserve chose C over A Else, with healthy network: - product read stayed at 12 ms with weaker consistency - inventory reserve paid 105 ms for stronger consistency That is why PACELC is useful. CAP only talks about failure time. PACELC reminds you that even healthy systems keep paying tradeoff rent.

5) Practical rules for CAP answers

Rule one. Never say "we choose AP" without naming the operation. Rule two. Never say "CA database" for a real distributed deployment. Rule three. Describe the user-visible failure. Does the user get stale data? A timeout? A retry suggestion? A hard rejection? Rule four. Tie the choice to business damage. Wrong likes are annoying. Wrong money movement is expensive. Wrong inventory count causes oversell. Rule five. Remember replicas are not free truth. A replica is another place where disagreement can appear. Look. The best CAP answer sounds like this. "For browse reads, we allow stale results during partition. For checkout writes, we reject when quorum is unavailable. In normal operation, we still pay extra latency for the strong path." That is senior-level. That is concrete. That respects the house rules.

Where this lives in the wild

  • Cassandra-backed social inbox or feed — backend engineer accepts temporary divergence during partitions because dropping every write would be worse for the product.
  • Google Ads budget or config state on Spanner — distributed systems engineer pays coordination cost to keep global updates strongly ordered.
  • Amazon shopping cart or session paths on DynamoDB — retail backend engineer can choose eventual reads for fast browsing while tightening guarantees for sensitive operations.
  • YouTube like counters — serving engineer can tolerate temporary replica disagreement better than a checkout system could.
  • Airline seat inventory or ticketing hold — booking platform engineer prefers rejecting some writes over silently overselling the same seat twice.

Pause and recall

  1. Why is CAP mainly about partition-time behavior rather than normal healthy operation?
  2. Why is "choose CA" not a serious answer for a distributed system?
  3. In the worked example, why did product reads and inventory reservations choose differently during partition?
  4. What extra idea does PACELC add beyond CAP?

Interview Q&A

Q: Why choose C or A during partition instead of claiming CA for the whole system? A: Because once multiple nodes communicate over a network, partitions are possible. During a real partition, you must decide whether to reject work or answer without full coordination.

Common wrong answer to avoid: "Because databases always replicate" — replication is common, but the real reason is network uncertainty, not a product feature checklist.

Q: Why describe CAP tradeoffs per operation instead of labelling the whole system once? A: Different operations have different damage profiles. Browsing, payments, reservations, and counters should not all make the same sacrifice during failure.

Common wrong answer to avoid: "Because interviewers like nuance" — nuance helps, but the deeper point is that real APIs expose different consistency needs.

Q: Why treat Spanner as CP-leaning with cost instead of as free CA? A: Spanner buys strong guarantees with coordination, latency, and the possibility of delaying or refusing work during certain failures. The tradeoff is engineered, not erased.

Common wrong answer to avoid: "Because global clocks are impossible" — the issue is not impossibility talk, but the continuing cost of maintaining strong cross-node agreement.

Q: Why use PACELC in design discussion instead of stopping at CAP? A: Because most of the time the network is healthy, and you still must choose whether lower latency is worth weaker consistency on each path.

Common wrong answer to avoid: "Because CAP is outdated" — CAP is still correct. PACELC simply adds the healthy-network tradeoff that teams feel every day.

Apply now (5 min)

Exercise: Pick one product with both browse reads and critical writes. Write what should happen during a partition for each path. Then write the latency you would pay in healthy conditions for the strong path. Sketch from memory: Draw two replicas and a broken link. Mark one operation as choose A. Mark one operation as choose C. Write one user-visible consequence beside each choice.

Bridge. We understand the tradeoffs. But tradeoffs assume things degrade gracefully. In reality, systems don't slow down — they break. Hard. We need to design for failure. → 11-failure-modes-and-resilience.md