Skip to content

00. Design a Payment System

~20 min read · Level: advanced · Part 1 of 4 (Overview → HLD → LLD → Q&A)

Problem

A payment system takes money from a buyer's card and moves it, minus fees, toward a merchant's bank account, while keeping a record precise enough to survive an audit years later. This is the product behind Stripe, Adyen, and Braintree — the layer a merchant integrates so they never touch the card networks directly. A checkout page calls POST /v1/payment_intents with an amount and a card, and the platform orchestrates authorization, capture, fee accounting, and eventual payout, exposing a clean state machine while hiding the mess of acquiring banks, card networks, and settlement files underneath.

The problem looks like "call a bank API and store the result." What makes it a genuine system-design question is that money is involved, which changes the rules. A duplicate is not a cosmetic bug — it is a customer charged twice and a chargeback. A lost write is not a stale cache — it is money that moved in the world but not in your books, which you will discover weeks later during reconciliation. The network between you and the card networks is unreliable in exactly the way that hurts most: it can fail after the charge succeeded but before you learned about it. Getting three things right — never charging twice, never losing a movement, and always being able to prove your books are correct — is most of the battle.

To keep the reasoning concrete, thread one scenario through the whole design: a customer's $50 checkout during a Black Friday sale, where the client's request times out and the client retries it three times. All four requests carry the same amount and the same idempotency key; exactly one $50 charge must reach the card, and exactly one balanced set of ledger entries must land in the books — no matter which of the four requests the platform actually saw first, or whether the card network already charged the card before a timeout hid that fact. And it must hold this guarantee while the platform is processing 10,000 payments/second at the Black Friday peak. That single charge, and that single tension between correctness and throughput, will test every decision below.

Functional requirements

  • Create a payment: given an amount, currency, and payment method, authorize and (usually) capture funds, returning a payment object with a well-defined status.
  • Idempotent retries: a client may safely resend the same request; the system charges once and returns the original result.
  • Capture and cancel: support authorizing now and capturing later (a two-step flow for orders that ship days after purchase), and cancelling an uncaptured authorization.
  • Refund: return some or all of a captured payment, itself idempotent.
  • Ledger and balance: maintain a double-entry ledger that is the source of truth for every cent, and expose merchant balances derived from it.
  • Reconciliation: match internal records against the payment processor's settlement reports and surface any discrepancy.
  • Webhooks: notify merchants of state changes (payment.succeeded, refund.updated) with at-least-once delivery.

De-scoped for this round, and worth naming so the interviewer knows it is a choice: fraud scoring and risk models, PCI card-vaulting internals (assume a tokenized payment_method already exists), multi-currency FX conversion, and the merchant onboarding/KYC flow. Each is a large system in its own right, and none of them changes the shape of the money-movement core, which is what this study is about.

Non-functional requirements

The single dominant constraint is correctness under concurrency and partial failure — exactly-once money movement. Everything else bends to it.

  • Consistency: strong, per payment. The idempotency check and the ledger write must be atomic with respect to a given payment; there is no "eventually we won't double-charge." This is the opposite of the URL shortener's eventual-consistency read path.
  • Durability: absolute, and audit-grade. A committed money movement must survive any single-node or single-AZ loss, and the ledger is retained for years (typically 7) for financial audit. Losing a ledger entry is unrecoverable in a way a lost cache entry never is.
  • Availability: high on the write path (four nines), but correctness wins ties — the system prefers to reject or defer a payment over risking a double charge. "Fail closed toward not-charging-twice" is the rule.
  • Latency: the merchant-facing API should respond in a few hundred milliseconds, but the actual card-network call can take seconds, so the design must decouple our response from the processor's latency rather than block on it.
  • Throughput: 10,000 payments/second at peak, each fanning out into several ledger writes — a genuinely write-heavy, contention-heavy load, unlike the read-heavy systems in this series.

Scale estimation

Anchor on the scenario. Baseline traffic is roughly 1,000 payments/second on an ordinary day, peaking at 10,000 payments/second on Black Friday — a 10× seasonal spike. At an average ticket of $50, baseline throughput is 1,000 × $50 = $50,000/second of processed volume, and over a year 1,000 × 86,400 × 365 ≈ 31.5 billion payments, or about $1.5 trillion in annual processed volume — the right order of magnitude for a large processor.

The write amplification is the number that shapes the design. A single payment is not one write. It creates a payment-intent record, an idempotency-key record, and a set of double-entry ledger rows — at minimum a debit and a credit, and in practice more once you split the gross amount into the merchant's share, the processing fee, and the platform's cut. Call it ~8 durable row-writes per payment. At the 10,000 payments/second peak that is 10,000 × 8 = 80,000 row-writes/second, all of which must be transactional and durable. This, not read volume, is why a payment ledger is a hard storage problem: the hot path is writes that cannot be lost, batched, or eventually-consistent.

Idempotency lookups add their own load. Every mutating request does a keyed read-then-conditional-write against the idempotency store. On a clean day that is one lookup per payment, but the scenario's retries matter: our $50 charge generates four requests for one payment (the original plus three retries). If retry storms during a processor slowdown push the average to ~2 attempts per payment at peak, that is 10,000 × 2 = 20,000 idempotency operations/second, each of which must be strongly consistent — a retry that races the original and both slip through is precisely the double charge we are paid to prevent.

Storage grows and never shrinks. At ~8 ledger-adjacent rows plus the payment and idempotency records, budget ~3 KB of durable data per payment. Over a year: 31.5B × 3 KB ≈ 95 TB/year, retained for seven years, so the ledger alone trends toward half a petabyte — append-only, immutable, and audited. Bandwidth, by contrast, is trivial: a payment payload is ~2 KB, so 10,000 × 2 KB = 20 MB/s at peak, well within any network. Payments are not a bandwidth problem or a read problem; they are a write-correctness-at-volume problem, and the estimates all point at the same place.

API sketch

POST /v1/payment_intents
  Idempotency-Key: 3f9c-...            # client-generated, required on all mutations
  body: { "amount": 5000, "currency": "usd",   # amount in minor units: 5000 = $50.00
          "payment_method": "pm_abc",
          "capture_method": "automatic" }       # or "manual" for auth-now-capture-later
  200: { "id": "pi_123", "status": "succeeded", "amount": 5000, "amount_captured": 5000 }
       # replayed retries with the same key return this same body, byte-for-byte

POST /v1/payment_intents/{id}/capture           # for manual capture
  200: { "id": "pi_123", "status": "succeeded", "amount_captured": 5000 }

POST /v1/refunds
  Idempotency-Key: 8a1d-...
  body: { "payment_intent": "pi_123", "amount": 5000 }
  200: { "id": "re_456", "status": "succeeded", "amount": 5000 }

GET  /v1/payment_intents/{id}
  200: { "id": "pi_123", "status": "succeeded", "charges": [...], "amount": 5000 }

GET  /v1/balance                                # merchant balance, derived from the ledger
  200: { "available": [{ "amount": 4855, "currency": "usd" }], "pending": [...] }

Two contract choices are load-bearing. Amounts are integers in minor units — 5000 means $50.00 — because floating-point cents are a rounding-error lawsuit waiting to happen. And the Idempotency-Key is a required client-supplied header on every mutation, not an optional nicety, because it is the single mechanism that makes the scenario's three retries safe.

Solutioning

Start from the dominant constraint — exactly-once money movement under concurrency — and the first two decisions follow directly. The first is an idempotency layer in front of every mutation. The client generates a key per logical operation and resends it on retry; the server stores (key → result) atomically the first time it commits, and on any subsequent request with that key returns the stored result without re-executing. This is what makes the scenario safe: the original $50 request and its three retries share one key, so at most one of them performs the charge and the other three replay its answer. The reframing to carry into the room is that a double charge is not a database concurrency bug to be patched; it is a missing-idempotency-key problem to be designed away — you do not try to make the charge operation faster or the lock tighter, you make retries return a remembered result.

The second decision is to make a double-entry ledger the source of truth, not a balance column you increment. Every money movement is recorded as balanced debits and credits against accounts, appended and never updated, so the ledger is an immutable log whose invariant — debits equal credits for every transaction — is checkable at any instant. A merchant's balance is derived by summing their account's entries, not stored and mutated. This costs more writes (our ~8 rows per payment) but buys the one property money demands: you can always prove where every cent is, and a bug can never silently corrupt a running total because there is no running total to corrupt. Money is not a number you edit; it is an append-only sequence of balanced movements you sum.

The third tension is synchronous versus asynchronous capture, and it is where throughput collides with the external world. The honest reason you cannot just "call the card network and wait" is latency and coupling: a processor call routinely takes 300 ms to several seconds, and at 10,000 payments/second a synchronous design would hold tens of thousands of request threads open waiting on someone else's bank. So a payment is modeled as a durable state machine — requires_confirmation → processing → succeeded/failed — persisted before any external call, with a worker driving the transitions and calling the processor asynchronously. The merchant's API call can return processing quickly and learn the terminal state via webhook or poll. A slow processor then degrades latency, not correctness, and never ties up the front door.

The fourth decision confronts the uncomfortable truth underneath all of this: you cannot achieve true exactly-once over an unreliable network. The processor call can succeed at the bank and then fail on the way back to you, so you genuinely do not know whether the card was charged. The resolution is a layered one — forward the idempotency key to the processor too (so your retry is also safe at their end), treat delivery as at-least-once with idempotent effects, and then run reconciliation as the backstop: every day, match your ledger against the processor's settlement file, and any movement that exists in one but not the other is flagged for repair. Exactly-once is not a guarantee you buy from the network; it is idempotent effects plus reconciliation that you build. The following files take these four decisions down to components (HLD), then to schemas, the idempotency and ledger internals, the state machine, and the reconciliation algorithm (LLD).