03. Payment System — Interview Q&A

~16 min read · Part 4 of 4 (Overview → HLD → LLD → Q&A)

These are the questions an interviewer actually asks once the diagram is on the board. Each answer is the strong version, followed by the wrong answer that quietly sinks candidates.

Q1. A client's $50 charge times out and it retries three times. How do you guarantee exactly one charge? Require a client-supplied idempotency key on every mutation and make the first phase an atomic claim: INSERT ... ON CONFLICT DO NOTHING on the key. Exactly one of the four requests wins the slot and does the work; the others find the key present and either return "still processing" (if the winner hasn't finished) or replay the stored response byte-for-byte (once it has). The card is charged once because only the slot-winner ever calls the processor, and the key is forwarded to the processor too so even an internal retry is deduplicated at their end. Four requests collapse to one claim, one charge, one balanced ledger transaction, four identical responses. Common wrong answer to avoid: "Check if a payment with these details already exists, then insert if not." That check-then-act is a race — two retries both read "not found" and both charge. The atomic claim, not a prior lookup, is what makes it safe.

Q2. Why a double-entry ledger instead of a balance column you increment? Because money demands provability and a mutable counter can't provide it. Every movement is recorded as balanced debit/credit rows in an append-only ledger, and a balance is derived by summing an account's entries, never stored and updated. This means the debits-equal-credits invariant is checkable at any instant, a bug can never silently corrupt a running total because there is no running total, and thousands of credits to one hot merchant on Black Friday are thousands of independent inserts rather than serialized increments on one contended row. It costs more writes — about 8 rows per payment — but that is the price of books you can audit. Common wrong answer to avoid: "Store balance and UPDATE balance = balance + amount per payment." That row becomes a contention hotspot under load, and a single bad update or lost transaction corrupts the total with no way to reconstruct the truth.

Q3. The processor call times out and you don't know if the card was charged. What do you do? Treat it as unknown, not failed and not succeeded — the two guesses that lose or duplicate money. Leave the payment in processing, retry with the same idempotency key forwarded to the processor (safe because they dedupe on it), and if the outcome is still ambiguous, defer to reconciliation. The next day's settlement file is the processor's authoritative record; if it shows the charge, reconciliation posts the settling ledger entries the real-time path missed. You cannot get exactly-once over an unreliable network, so you get at-least-once with idempotent effects plus reconciliation as the backstop. Common wrong answer to avoid: "Retry the charge until it succeeds" (risks a real double charge if the first one actually went through) or "mark it failed" (loses a charge that really happened). A timeout is ambiguous; resolving it by guessing is how money leaks.

Q4. How does the system handle 10,000 payments/second on Black Friday? Recognize that payments are independent, so the correctness guarantee is per-key, not global. Partition the idempotency store and ledger by payment/account id and the ~80,000 ledger row-writes/second at peak spread across shards with no cross-shard coordination — the exactly-once claim is per-key and therefore embarrassingly parallel. The slow, rate-limited part is the processor call at ~500 ms each, which at 10,000/second would mean ~5,000 concurrent in-flight calls; that lives in async workers behind a durable queue, not in request threads, so a slow processor grows queue depth (recoverable latency) rather than dropping or double-charging payments. The front door returns processing fast and the state machine finishes the work. Common wrong answer to avoid: "Add a global lock or transaction around payment creation to prevent double charges." A global lock serializes all 10,000 payments/second through one point and collapses throughput; correctness is per-key and must stay that way.

Q5. Synchronous or asynchronous capture — why not just call the bank and wait? Because the processor call routinely takes 300 ms to several seconds, and at 10,000 payments/second a synchronous design holds tens of thousands of request threads hostage to someone else's bank. Model the payment as a durable state machine (requires_confirmation → processing → succeeded/failed), persist it before any external call, and drive the processor call from an async worker. The merchant's request returns processing quickly and learns the terminal state via webhook. A slow processor then degrades latency, not correctness, and never ties up the front door or risks a timeout-induced double submit. Common wrong answer to avoid: "Call the processor synchronously inside the request and return the final status." It couples your availability and latency to the card network's, and under peak load the thread exhaustion takes down the whole API.

Q6. Where do you store amounts, and why does it matter? As integers in minor units — $50.00 is the integer 5000 — never as floats or decimals in application code. Floating-point can't represent many decimal cents exactly, so 0.1 + 0.2 drifts, and across billions of payments those fractions of a cent become real, auditable discrepancies. Integer minor units make money arithmetic exact and the ledger's balance-to-zero invariant a clean integer check. Common wrong answer to avoid: "Use a float/double for amounts." Rounding error is not hypothetical at scale; it produces reconciliation breaks and, eventually, a regulator asking where the missing cents went.

Q7. What is reconciliation and why is it not optional? Reconciliation is the daily batch that matches your ledger against the processor's settlement file — their authoritative record of what actually moved — and opens a "break" for every mismatch: a charge they settled that you never booked (MISSING_INTERNALLY), one you booked that they didn't settle (MISSING_EXTERNALLY), or an amount that differs. It is not optional because exactly-once over the network is impossible: the timeout case in Q3 leaves gaps by design, and reconciliation is the only mechanism that catches a charge that succeeded at the bank but never reached your books. It converts at-least-once-with-gaps into eventually-correct books. Common wrong answer to avoid: "If the real-time path is correct we don't need reconciliation." The real-time path cannot be exactly-once across an unreliable network; a system without reconciliation is one that silently loses money and finds out from an angry customer instead of a report.

Q8. What's the single most important operational metric during a payment incident? The reconciliation break count — mismatches between the ledger and settlement per day — because it is the only number that measures actual money correctness rather than performance. A fast, green API with a climbing break count means money is moving in the world that your books disagree about, which is the real emergency. For early warning of trouble, watch payment-queue depth / worker lag, since a slowing processor backs up there first. The trap is watching API-gateway latency, which stays flat and fast because the front door returns processing without waiting on the processor — it looks healthy while payments stall in the queue behind it. Common wrong answer to avoid: "Monitor API latency and error rate." Those miss the failure that matters most — a payment can be perfectly fast and still be silently un-booked; latency dashboards are green during the worst money incidents.

Q9. A retry arrives while the original request is still in flight. What happens? The atomic claim already ran for the original, so the key exists with status = in_progress. The retry's INSERT ... ON CONFLICT finds it and returns a safe "still processing" response — it does not start a second charge and does not overwrite the in-flight work. When the original completes and finalizes the key, later retries replay the stored result. The one loose end is a crash between claiming and finalizing, which leaves the key stuck in_progress; a sweeper reconciles orphaned keys against the payment record — finalizing if the intent committed, releasing if not — so the guarantee is never lost, only occasionally delayed. Common wrong answer to avoid: "The retry waits on a lock until the first finishes, then returns its result." Holding a request thread on a lock across a multi-second processor call is how you exhaust threads under peak load; return "in progress" and let the client poll or receive the webhook.

Q10. How do you make refunds safe against double submission? Give each refund its own idempotency key so a resent refund request refunds once, and additionally guard at the ledger level: cumulative refunds for a charge can never exceed the captured amount, enforced by checking the sum of prior refund entries before posting a new one. The refund is itself a balanced ledger transaction (debit the merchant, credit the customer-payable account), so it obeys the same append-only, provable rules as the original charge. Common wrong answer to avoid: "Set a refunded = true flag on the payment." A boolean can't represent partial refunds, races with concurrent requests, and doesn't stop the total refunded from exceeding what was captured.

Q11. How do you deliver webhooks reliably without coupling them to payment correctness? Deliver at-least-once with persisted events, exponential-backoff retries, and signed payloads so merchants can verify authenticity and dedupe on the event id. Crucially, a payment's correctness never depends on webhook delivery — by the time an event is dispatched the ledger is already right, so a merchant endpoint being down delays notification, not the money movement. Merchants are expected to handle duplicate deliveries, mirroring the idempotency contract we ask of them on the way in. Common wrong answer to avoid: "Fire the webhook synchronously and mark the payment succeeded only after the merchant returns 200." That couples your payment state to the merchant's uptime — their down endpoint would strand real, completed charges.

Q12. How do you keep a hot merchant account from becoming a write bottleneck on Black Friday? The append-only ledger already solves it. Thousands of payments/second to one merchant are thousands of independent credit inserts, not competing updates to a shared balance row, so there is no contention point to serialize on. The balance is derived by summation, kept cheap with a periodic snapshot plus a small tail sum, so even a merchant with millions of entries answers a balance query in near-constant time. Contention only appears if you store a mutable balance — which is exactly why we don't. Common wrong answer to avoid: "Shard that merchant's balance across sub-accounts and reconcile them." That's a workaround for the mutable-balance mistake; with a derived balance over an append-only ledger, the hotspot never exists in the first place.

Deeper follow-ups

• How would you handle multi-currency, where a customer pays in EUR and the merchant settles in USD, without introducing floating-point FX drift into the ledger?
• The auth succeeds but the capture (days later) fails — how does the state machine and ledger represent an expired or voided authorization?
• How would you detect and defend against a compromised merchant API key replaying old idempotency keys to probe your system?
• If a reconciliation break is MISSING_INTERNALLY for a charge from 30 days ago, how do you post the corrective entry without breaking the historical immutability of the ledger?
• How would you shard the ledger for a merchant so large their single account's entries no longer fit or sum on one node?
• What changes if you must support exactly-once payouts to merchant bank accounts, where the downstream (ACH/wire) has no idempotency key and settlement takes days?

How this round is scored

Interviewers use the payment system to see whether you treat correctness as the primary constraint and everything else as negotiable around it. The strong signal is reaching for idempotency keys and a double-entry ledger early and unprompted, and being able to say precisely why — a double charge is a missing-idempotency-key problem, a lost movement is a mutable-balance problem — rather than bolting locks onto a naive design. Seniority shows up most in the timeout discussion: candidates who understand that exactly-once is impossible over the network, and who reach for at-least-once-plus-reconciliation instead of trying to make the network reliable, are the ones who have run money systems rather than only drawn them. The math matters too — knowing that 10,000 payments/second is ~80,000 ledger writes/second and ~5,000 concurrent processor calls, and using those numbers to justify partitioning and async workers, is what separates a plausible answer from a senior one. The candidates who lose points are the ones who polish the happy path and have no answer for the retry, the timeout, or the settlement break — because in payments, the failure modes are the design.