01. Payment System — High-Level Design¶

~16 min read · Part 2 of 4 (Overview → HLD → LLD → Q&A)

This file turns the solutioning narrative into concrete boxes and the flows between them. Read the architecture top to bottom, then follow a payment through it, then look at what happens when the pieces — and the card network beyond them — fail.

Architecture¶

                          ┌──────────────┐
        merchant ───────▶ │  API Gateway  │  (auth, rate-limit, idempotency-key check)
        checkout          └──────┬───────┘
                                 │
                                 ▼
                         ┌──────────────────┐        ┌──────────────────┐
                         │ Payment Service  │◀──────▶│ Idempotency Store │
                         │ (state machine)  │        │  (key → result)   │
                         └───┬─────────┬────┘        └──────────────────┘
                             │         │
              write intent   │         │ enqueue "drive this payment"
                             ▼         ▼
                    ┌──────────────┐  ┌──────────────┐
                    │ Ledger (DB,  │  │ Payment Queue │
                    │ double-entry,│  │  (durable)    │
                    │ ACID, sharded)│  └──────┬───────┘
                    └──────────────┘         │
                             ▲               ▼
                             │        ┌──────────────┐        ┌──────────────┐
                             │        │ PSP Worker   │───────▶│ PSP / Acquirer│──▶ card
                             └────────┤ (calls bank, │◀───────│ (Stripe-core, │   networks
                                      │  writes led.) │  webhk │  Adyen, bank) │
                                      └──────┬───────┘        └──────────────┘
                                             │
                                             ▼
                    ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐
                    │ Webhook      │  │ Reconciliation│  │ Settlement files  │
                    │ Dispatcher   │  │  Engine       │◀─┤ (daily, from PSP) │
                    │ (to merchant)│  └──────────────┘  └──────────────────┘
                    └──────────────┘

Read it top to bottom. A merchant's checkout hits the API Gateway, which authenticates, rate-limits, and — critically — enforces the idempotency contract by consulting the Idempotency Store before any work happens. The Payment Service owns the payment as a state machine: it writes the intent and an opening ledger entry inside one transaction against the Ledger, then enqueues a durable job onto the Payment Queue and returns to the merchant. A PSP Worker pulls that job, calls the external PSP/acquirer (which talks to the card networks), and on the result writes the settling ledger entries and advances the state machine. Terminal states fan out two ways: the Webhook Dispatcher notifies the merchant, and, on a separate daily cadence, the Reconciliation Engine pulls the PSP's settlement files and matches them against the ledger to catch anything the real-time path got wrong. The left edge — gateway, idempotency, ledger — is the strongly-consistent correctness core; the right edge — queue, worker, reconciliation — is the asynchronous machinery that absorbs the external world's unreliability.

Components¶

API Gateway. Terminates TLS, authenticates the merchant's API key, applies per-merchant rate limits, and performs the first idempotency check. It is the single choke point where the Idempotency-Key header is validated as present on every mutation, so no request reaches the payment logic without one.

Payment Service. The brain, and the owner of the payment state machine. It persists a payment intent before any external call, records the opening ledger movement, and hands off the risky external work to the queue. It holds no long-lived per-request state across the PSP call — that state lives durably in the payment record — so any instance can pick up any payment.

Idempotency Store. A strongly-consistent key-value store mapping idempotency-key → (status, response). It is the mechanism that collapses the scenario's four requests into one charge. It must offer an atomic "insert-if-absent" so two concurrent retries cannot both claim the key. Records carry a TTL (typically 24 hours) since a client will not retry a day-old request.

Ledger. The durable, ACID, double-entry system of record. Every money movement is one transaction of balanced debit/credit rows, append-only, never updated in place. It is sharded — by merchant account for balance queries — but each individual payment's entries commit within a single transaction so the debits-equal-credits invariant is never momentarily violated.

Payment Queue. A durable, ordered queue (Kafka or a transactional outbox drained to it) that decouples the fast merchant response from the slow PSP call. It guarantees at-least-once delivery, which is exactly why the worker downstream must be idempotent.

PSP Worker. Consumes payment jobs and makes the actual authorize/capture call to the external processor, forwarding the idempotency key so the processor also deduplicates. On the response it writes the settling ledger entries and advances the state machine. On timeout or ambiguous failure, it does not guess — it leaves the payment in processing for reconciliation to resolve.

Webhook Dispatcher. Delivers state-change events to merchants with at-least-once semantics, retries with backoff, and signs payloads so merchants can verify authenticity. Merchants are expected to handle duplicates, mirroring our own internal contract.

Reconciliation Engine. The backstop. It ingests the PSP's daily settlement file — the processor's authoritative record of what actually moved — and matches each line against the ledger, classifying every row as matched, missing-internally, or missing-externally, and routing the mismatches to repair. It is the only component that can catch a charge that succeeded at the bank but never made it into our books.

Primary write path (create a $50 payment)¶

POST /v1/payment_intents with Idempotency-Key: 3f9c reaches the API Gateway, which authenticates and checks the key.
The gateway / Payment Service does an atomic insert-if-absent on the idempotency key. If the key is new, it wins the slot and proceeds. If the key already exists in a terminal state, it returns the stored response immediately — this is the branch the scenario's three retries take.
On the winning path, the Payment Service opens a transaction against the Ledger: it writes the payment intent (status = processing) and an opening entry moving $50 from a "customer receivable / pending" account, then commits. The idempotency record is updated to processing in the same logical step.
It enqueues a durable "drive pi_123" job on the Payment Queue and returns { status: "processing" } (or waits briefly and returns succeeded for fast processors) to the merchant. The front-door latency is now independent of the card network.
The PSP Worker dequeues the job, calls the processor's authorize+capture with the same idempotency key forwarded downstream, and on success opens a second Ledger transaction: debit the pending account, credit the merchant's balance ($48.55), credit the platform fee account ($1.45), advance the payment to succeeded, and finalize the idempotency record with the full response body.
Terminal state triggers the Webhook Dispatcher to notify the merchant. The payment is done; every cent is accounted for in balanced entries.

Primary read path (fetch a payment / balance)¶

GET /v1/payment_intents/{id} is a point lookup by primary key against the payment store — cheap and cacheable, since a terminal payment is immutable.
GET /v1/balance is the interesting read: a balance is derived, not stored, so it is the sum of a merchant's ledger entries. Summing millions of rows per request would be far too slow, so the system maintains materialized balance snapshots — a periodically-checkpointed running total plus the entries since the last checkpoint — turning a balance read into "snapshot + a small tail sum" rather than a full scan.
Reads never block writes and are served from replicas; because a committed payment is immutable, a slightly stale replica read is acceptable for display, while any operation that moves money re-reads authoritatively from the ledger primary.

Storage choices¶

Ledger: ACID relational store, sharded by account. The ledger's whole value is the transactional guarantee that debits and credits commit together; that is a job for a database with real transactions (PostgreSQL, Spanner, CockroachDB), not an eventually-consistent KV store. Sharding by merchant account keeps a merchant's entries co-located for balance derivation, and each payment's entries stay within one transaction. Append-only means no update contention on a shared balance row.
Idempotency Store: strongly-consistent KV with atomic put-if-absent. The access pattern is a keyed lookup and a conditional insert, and the one non-negotiable is linearizable compare-and-set so concurrent retries cannot both win. DynamoDB with conditional writes, or a Redis/Postgres row with a unique constraint, fits. TTL support prunes old keys automatically.
Payment records: same relational store as the ledger (or co-partitioned). A payment is queried by id and lives beside its ledger entries, so co-locating them lets the intent write and the opening ledger entry share one transaction.
Payment Queue: durable log (Kafka) or transactional outbox. Must not lose a job — a dropped job is an authorized-but-never-captured payment. The outbox pattern (write the job to a table in the same transaction as the intent, then relay it) closes the gap where a crash between "commit intent" and "enqueue job" would otherwise strand a payment.
Settlement / reconciliation data: object storage + columnar analytics. Settlement files are large flat files dropped daily; land them in object storage and match with a columnar engine that can join millions of ledger rows against millions of settlement lines efficiently.

Scaling¶

Write path. The load is 80,000 ledger row-writes/second at the 10,000-payment peak, and the key insight is that payments are independent: different payments touch different accounts and different idempotency keys, so the write path shards cleanly. Partition the ledger and idempotency store by a hash of the payment/account id and the 80,000 writes/second spread across shards with no cross-shard coordination on the common path. Adding shards adds write throughput linearly. The one place contention concentrates is a single hot merchant account taking thousands of payments/second on Black Friday — solved not by locking that balance row but by the append-only ledger design, where thousands of credits to one account are thousands of independent inserts, and the balance is derived by summation rather than by serialized increments.

Idempotency throughput. Retries multiply lookups: the 10,000-payment peak becomes ~20,000 idempotency operations/second under retry pressure. This store is partitioned by key, so it scales horizontally too, and because keys are uniformly distributed there is no hot partition. The atomic put-if-absent is per-key, so it never serializes across keys.

PSP call fan-out. The external processor is the slow, rate-limited dependency. At 10,000/second with 500 ms average call latency, that is 10,000 × 0.5 = 5,000 concurrent in-flight PSP calls to hold — which is exactly why those calls live in async workers behind a queue, not in request threads. Worker count scales with queue depth; if the PSP rate-limits us, the queue absorbs the backlog and drains as capacity returns, trading latency for not dropping payments.

Read path. Balance snapshots turn an O(entries) sum into O(1) + small tail, and payment lookups are point reads served from replicas. Reads are a rounding error next to the write load and scale with read replicas.

Operational signals¶

The healthy signal is the authorization success rate holding steady around its baseline (say 92–95% — some cards legitimately decline) with p99 end-to-end payment latency flat; a Black Friday launch that keeps both steady while volume climbs 10× is the system working as designed. The first metric to degrade under trouble is payment-queue depth / worker lag: when the PSP slows or rate-limits, jobs pile up here first, long before the ledger or gateway feel anything, so it is the early-warning gauge. The misleading metric is API-gateway latency — it stays flat and fast because the front door returns processing without waiting on the PSP, so a merchant staring at gateway latency sees green while payments quietly stall in the queue behind it; watch queue lag and terminal-state latency, not front-door latency. The graph an experienced operator opens first during a money incident is the reconciliation break count — the number of ledger-versus-settlement mismatches per day — because it is the only number that measures actual correctness rather than performance, and a nonzero, climbing break count means money is moving in the world that your books disagree about.

Failure modes and resilience¶

Client timeout and retry (the scenario). The customer's $50 request times out and the client retries three times. The idempotency key makes the first committed attempt authoritative and the other three replay its result — so at most one charge reaches the card. The dangerous sub-case is a retry arriving while the original is still processing: the atomic put-if-absent lets only one request own the key, and a retry that finds the key in processing returns a "still processing" response rather than starting a second charge. Exactly-once is preserved even though the network delivered the request four times.
PSP timeout with unknown outcome. The worker calls the processor and the connection drops before a response — the card may or may not have been charged. The worker must not retry blindly (that risks a real double charge) and must not mark the payment failed (that risks losing a real charge). It leaves the payment processing, retries with the same forwarded idempotency key (safe at the PSP end), and if still ambiguous, defers to reconciliation, which will see the truth in the settlement file.
Ledger primary loss. Writes fail closed — better to reject new payments briefly than to record money incorrectly. A synchronous replica is promoted; because entries are append-only and each payment commits atomically, there are no partial-balance states to repair, only the last few in-flight transactions to retry.
Queue backlog under peak. If PSP throughput can't keep up at 10,000/second, jobs queue rather than drop. Merchant-facing latency rises (payments sit in processing longer) but no payment is lost and none is double-charged; the backlog drains as PSP capacity recovers. This is the deliberate latency-for-durability trade.
Webhook delivery failure. A merchant endpoint is down. The dispatcher retries with exponential backoff and the event persists, so delivery is at-least-once and merchants dedupe. A payment's correctness never depends on webhook delivery — the ledger is already right.
Reconciliation break. The settlement file shows a charge the ledger is missing (or vice versa). This is caught by design rather than prevented: the break is flagged, an automated or manual repair posts the corrective ledger entry, and the invariant is restored. Reconciliation is not a cleanup nicety; it is the mechanism that converts at-least-once-with-gaps into eventually-exactly-once.

Where this shows up in production¶

Stripe — models every payment as a PaymentIntent state machine and requires a client-supplied Idempotency-Key on all mutations, so a merchant's retry after a timeout returns the original charge instead of creating a second.
Stripe's ledger — an internal double-entry, immutable ledger is the source of truth for balances; the public balance is derived from summing entries, never stored as a mutable counter.
Adyen — decouples authorization from capture and reconciles against acquirer settlement reports, the same real-time-plus-daily-reconciliation two-track design shown here.
PayPal / Braintree — treat the payment as an async workflow behind a durable queue so a slow card network degrades latency, not correctness.
Square — forwards idempotency keys down to the acquiring processor so deduplication holds end-to-end, not just at their own edge.
Uber's ledger (LedgerStore / bank-grade double-entry) — built an append-only, immutable ledger precisely to make every rider/driver money movement auditable and non-corruptible by a stray update.
Airbnb / Uber reconciliation pipelines — daily batch jobs that match internal ledgers against processor settlement files and open "breaks" for any mismatch, exactly the backstop role described above.
Kafka transactional outbox in payments — writing the payment job to an outbox table in the same transaction as the intent, then relaying it, so a crash can never authorize a payment without also enqueuing its follow-through.