01. Digital Wallet — High-Level Design¶

~15 min read · Part 2 of 4 (Overview → HLD → LLD → Q&A)

This file turns the solutioning narrative into concrete boxes and the flows between them. Read the architecture top to bottom, follow a transfer through the write path and a balance through the read path, then look at what breaks when a piece fails — including Priya's two $70 transfers.

Architecture¶

                          ┌──────────────┐
        client ─────────▶ │ API Gateway  │  (auth, rate-limit, idempotency-key intake)
                          └──────┬───────┘
                                 │
                                 ▼
                          ┌──────────────┐
                          │ Wallet        │  stateless; orchestrates a transfer
                          │ Service       │
                          └──┬───────┬─────┘
                    sync gate│       │ money movement
                             ▼       ▼
                    ┌──────────────┐ ┌───────────────────────────────┐
                    │ Risk / Fraud │ │        Ledger Store            │
                    │ Gate (<20ms) │ │  sharded by account_id         │
                    └──────────────┘ │ ┌───────────┐  ┌────────────┐ │
                                     │ │ balance    │  │ ledger_    │ │
                                     │ │ (1 row/acct)│  │ entry      │ │
                                     │ │  hot,       │  │ append-only│ │
                                     │ │  serialized │  │ immutable  │ │
                                     │ └───────────┘  └────────────┘ │
                                     └───────────────┬───────────────┘
                                                     │ commit → emit
                                                     ▼
                                              ┌──────────────┐
                                              │ Event Bus    │  (Kafka; ordered per account)
                                              └──┬────────┬──┘
                                                 │        │
                        ┌────────────────────────┘        └──────────────┐
                        ▼                                                 ▼
                ┌──────────────┐                                 ┌──────────────┐
                │ Async Fraud  │                                 │ Reconciler + │
                │ Scorer (ML)  │  can freeze / reverse           │ Analytics /  │
                └──────────────┘                                 │ History store│
                                                                 └──────────────┘

The client hits the API Gateway, which authenticates, rate-limits, and captures the idempotency key that makes retries safe. The Wallet Service orchestrates each transfer: it calls the Risk Gate synchronously for an allow/deny, then executes the money movement against the Ledger Store, which is sharded by account and holds two tables — the tiny, hot, strongly-serialized balance table and the large, append-only ledger_entry table. On commit it emits an ordered event to the Event Bus, from which an async fraud scorer and a reconciler / analytics pipeline consume independently. The critical path is deliberately short: gateway → wallet service → one ledger transaction. Everything below the event bus is off the hot path and cannot slow or break a payment.

Components¶

API Gateway. Authenticates the caller, enforces per-user and per-endpoint rate limits, and — critically for a payments system — records the Idempotency-Key so a network retry of the same transfer returns the original result instead of moving money twice. It is the first line against both abuse and accidental double-submission.

Wallet Service. Stateless orchestrator of a transfer. It validates the request, invokes the synchronous risk gate, opens the ledger transaction, and returns the result. Holding no per-request state means any instance can handle any request and the tier scales horizontally; the correctness lives in the store, not here.

Risk / Fraud Gate (synchronous). A low-latency service (target p99 under 20 ms) that runs cheap, deterministic checks — velocity limits (this account has moved N transfers in the last minute), blocklists, device and geo reputation, amount thresholds — and returns allow / deny / challenge before the debit commits. It exists to keep obvious fraud off the money path without adding meaningful latency.

Ledger Store (sharded by account_id). The system of record, and the heart of the design. The ledger_entry table is append-only and immutable — money is moved only by inserting balanced debit/credit rows, never by mutating past ones. The balance table holds one materialized row per account (available, held, version) updated inside the same transaction as the entries, so the fast balance read is always consistent with the ledger. Sharding by account keeps most transfers single-shard and keeps a single account's writes serialized on one node.

Event Bus. An ordered, durable log (Kafka or similar), partitioned so that all events for one account arrive in commit order. It decouples the committed ledger from every downstream consumer, so async fraud scoring, analytics, notifications, and history projection can lag or restart without touching the write path. Ordering per account matters: a reversal must never be processed before the transfer it reverses.

Async Fraud Scorer. Consumes the ledger stream and runs the expensive models — graph analysis, behavioral ML — that are too slow for the inline gate. When it flags a committed transfer, it acts through the same reversible ledger: freeze the account (status change that the sync gate then honors) and/or post a compensating reversal entry. It trades latency for depth, and relies on the ledger being append-only and reversible.

Reconciler + History/Analytics store. The reconciler periodically re-sums each account's ledger entries and compares the total against the materialized balance row; any drift raises an alarm, because in a wallet a silent mismatch is money unaccounted for. The same pipeline projects the ledger into a read-optimized transaction-history store for the "list my transactions" screen, keeping heavy history queries off the hot balance table.

Primary write path (a transfer)¶

POST /api/v1/transfers with an idempotency key reaches the Wallet Service through the gateway.
The service checks the idempotency record: if this key already produced a result, it returns that result verbatim and stops — no second movement.
It calls the Risk Gate synchronously. A deny returns 403 and nothing is written; a challenge triggers step-up auth; an allow proceeds.
It opens a single ledger transaction. When both accounts are on the same shard, this transaction does all four writes atomically: guarded debit of the sender's balance, credit of the receiver's balance, and two ledger_entry inserts. The guarded debit (WHERE available >= amount) is where an overdraft is refused.
If the guarded debit changes zero rows, the transaction rolls back and the service returns 409 insufficient_funds. Otherwise it commits.
On commit it writes the idempotency result, emits the ordered event, and returns 201 with the new balance. Cross-shard transfers replace step 4 with the hold/settle flow described under Scaling.

Primary read path (a balance)¶

GET /accounts/{id}/balance reaches the Wallet Service.
The service reads the materialized balance row for the account — a single point lookup by primary key on its shard, sub-millisecond. This is authoritative because it was updated in the same transaction as every entry.
A read-through cache can front this for display, but with a short TTL and, crucially, the cache is never consulted for a debit decision — the guarded update inside the transaction reads the row directly under lock. The cache accelerates the "show my balance" screen, not the "can I pay" decision.
Transaction history reads go to the separate history store, not the hot ledger, so scrolling a year of transactions never contends with live payments.

Storage choices¶

Ledger + balance: a strongly-consistent, transactional store, sharded by account_id. The access pattern is point lookups and small atomic multi-row transactions with hard invariants — exactly what a relational engine (PostgreSQL/MySQL, or a distributed SQL store like CockroachDB/Spanner/TiDB) does well. A CHECK (available >= 0) constraint and real transactions are worth more here than raw throughput. Sharding by account co-locates an account's balance and entries and serializes its writes on one node.
Idempotency keys: a fast store with a unique constraint. Either a dedicated table with a UNIQUE(idempotency_key) in the same transactional store (so dedup and commit are atomic) or Redis with a TTL for the recent window. Correctness favors the transactional table; Redis is an optimization in front of it.
Event bus: an ordered, durable log (Kafka). Partitioned by account so per-account order is preserved. This is the decoupling seam between the correct-and-slow write path and every eventually-consistent consumer.
History / analytics: an append-optimized store (columnar or time-series). Queried by account over time ranges and by aggregation, never for a debit decision, so it can be eventually consistent and denormalized for cheap reads.

Scaling¶

Write path. Global throughput (4,600 entry writes/second at peak) scales by adding shards, since transfers spread across accounts. The real limit is per-account serialization: one account's balance row can only be updated one transaction at a time. For ordinary users that ceiling is far above their real rate. For a hot merchant taking thousands of payments a second, you relieve the single-row contention by splitting that merchant's balance into N sub-accounts (a sharded counter) — payments land on a random sub-account, and the true balance is the sum across them, reconciled continuously. This trades an instantaneous exact balance for throughput on exactly the accounts that need it.

Cross-shard transfers. When sender and receiver are on different shards, there is no cheap single transaction, so the transfer becomes: hold on the sender's shard (atomic available → held, which already enforces the no-overdraft guard), credit on the receiver's shard, then capture the hold (convert to a committed debit). A coordinator drives this as a saga; any failure releases the hold and the sender is made whole. The cost is two round trips and a briefly "held" amount instead of one local commit — paid only on genuinely cross-shard transfers, which you minimize by co-locating related accounts.

Read path. Balance reads (58,000/second peak) are point lookups that scale with shards and an optional short-TTL display cache. History reads are offloaded entirely to the projection store. Neither read path is allowed to influence a debit decision, so scaling reads never risks correctness.

Operational signals¶

The healthy signal is the reconciliation delta — the sum of every account's ledger entries minus its materialized balance — which should be flat zero. It is the one metric that proves money is neither leaking nor being invented, and an expert watches it first. The first metric to degrade under trouble is transfer p99 latency, which climbs when the risk gate slows or when hot-account row-lock contention queues transactions behind each other. The misleading metric is aggregate transfer throughput: it can look perfectly healthy at 2,300/second while a single hot merchant's account is serializing and timing out — the average hides per-account contention, so you must watch the p99 and per-account lock-wait time, not the total. During an incident the graph to open first is rejected/failed transfer rate broken down by reason: a spike in insufficient_funds may be normal payday behavior, but a spike in lock_timeout or risk_gate_timeout means the write path itself is choking.

Failure modes and resilience¶

Two concurrent debits on one account (the threaded scenario). Priya's two $70 transfers arrive in the same instant. Both cannot be allowed. The balance row's lock serializes them: the first acquires the row, applies the guarded debit 100 → 30, commits; the second waits, then finds 30 < 70, changes zero rows, and is rejected with 409. The balance never goes negative because the check and the deduction are the same atomic statement — there is no window between reading 100 and writing 30 for the second transfer to sneak through. This is the design working as intended, not a failure, but it is the exact case an interviewer probes.
Partial cross-shard transfer. The sender is debited (hold captured) but the receiver's credit fails. Because the flow is hold-then-credit-then-capture, the coordinator never captures until the credit confirms; on credit failure it releases the hold and the sender's available is restored. No entry is ever mutated — a failed leg leaves a released hold, fully auditable.
Duplicate submission / client retry. A flaky network makes the client resend the same transfer. The idempotency key's unique constraint means the second attempt returns the first result and moves no money. Without it, a retry is a silent double-charge — the most common real-world wallet bug.
Risk gate outage. If the synchronous gate is down, the policy is a deliberate choice, not an accident: fail-closed for high-risk movements (large amounts, new devices) and fail-open with tighter velocity caps for low-risk ones, so a fraud-service outage degrades safety gracefully rather than either halting all payments or opening the doors.
Balance/ledger drift. A bug updates the balance without a matching entry, or vice versa. The reconciler catches it within its cycle, freezes the affected account, and the ledger — being the immutable source of truth — is used to recompute the correct balance. This is why the ledger, not the balance row, is authoritative.
Event bus lag. Async fraud scoring, history, and analytics fall behind, but payments keep committing because those consumers are off the hot path. Events are durable and replayed on recovery; the only visible effect is that fraud clawbacks and history updates arrive a little later.

Where this shows up in production¶

Venmo / PayPal — model balances as a double-entry ledger with immutable postings, so a "payment" is two balanced entries and disputes are resolved by reading the trail, never by editing a balance.
Stripe — exposes the authorization/capture (hold/settle) split directly in its API and runs a synchronous risk gate (Radar) inline on the charge path, the exact two-speed fraud model here.
Square / Cash App — shard wallet balances by account and use guarded, serialized updates so a single account cannot be overdrawn under concurrency.
PhonePe / Paytm — operate at festival-sale peaks where a single popular merchant is the hot account, and split high-volume merchant balances across sub-accounts (sharded counters) to escape single-row contention.
Uber's money/payments platform — built an internal double-entry ledger service precisely so every fare, tip, and refund is an auditable posting rather than a mutable field.
Adyen — runs the reserve-then-capture flow across systems for cross-institution movement, releasing holds on failure exactly as the cross-shard saga does here.
Banking core ledgers (e.g. Thought Machine, TigerBeetle) — TigerBeetle is a purpose-built double-entry accounting database whose whole reason to exist is high-throughput, strictly-serialized, never-negative balance transfers — this problem, productized.
Kafka in payments pipelines — used as the per-account-ordered event log so a reversal is never processed before the transfer it reverses, the ordering guarantee this design leans on.