01. Hotel / Stay Booking — High-Level Design¶

~15 min read · Part 2 of 4 (Overview → HLD → LLD → Q&A)

This file turns the two-halves narrative from the overview into boxes and the flows between them. Read the architecture top to bottom, follow a search and a reservation through it, then watch what happens when Guest A's payment stalls mid-checkout.

Architecture¶

                          ┌──────────────┐
        client ─────────▶ │  Edge / CDN  │  (photos, static, search cache)
                          └──────┬───────┘
                                 │
                          ┌──────────────┐
                          │  API Gateway  │  (auth, rate-limit, routing)
                          └──────┬───────┘
              search reads       │       reserve / confirm writes
        ┌────────────────────────┴────────────────────────┐
        ▼                                                  ▼
 ┌──────────────┐                                   ┌──────────────┐
 │   Search      │                                   │   Booking     │
 │   service     │                                   │   service     │
 └──────┬───────┘                                   └──────┬───────┘
        │                                                  │
        ▼                                    ┌─────────────┼──────────────┐
 ┌──────────────┐                            ▼             ▼              ▼
 │ Elasticsearch│                     ┌────────────┐ ┌──────────┐ ┌──────────────┐
 │ (geo+facets, │                     │ Inventory /│ │ Pricing  │ │  Payment      │
 │  coarse avail│                     │ Calendar DB│ │ service  │ │  gateway      │
 └──────▲───────┘                     │ (SoT,      │ │ (quotes) │ │ (external)    │
        │ async reindex               │  sharded)  │ └──────────┘ └──────────────┘
        │                             └─────┬──────┘
 ┌──────┴───────┐                           │ booking events
 │  Indexer     │◀──────────────────────────┤
 │  (consumer)  │        ┌──────────────┐   ▼
 └──────────────┘◀───────│  Event Bus   │◀──┴─── (holds, confirms, cancels)
                         │  (Kafka)     │
                         └──────┬───────┘
                                ▼
                         ┌──────────────┐
                         │  Reaper +    │  (expire holds, release nights)
                         │  workers     │
                         └──────────────┘

Read it as two lanes joined at the gateway. On the left, a search request flows to the search service and is answered from Elasticsearch, which holds a denormalized, geo-indexed copy of every listing plus a coarse availability flag — deliberately allowed to lag the truth. On the right, a reservation flows to the booking service, which is the only component that writes to the Inventory/Calendar DB, the single source of truth for who owns which room-night. The booking service consults the pricing service for a quote and the external payment gateway for money, and emits every state change (hold, confirm, cancel) onto an event bus. Two consumers hang off that bus: an indexer that asynchronously refreshes availability in Elasticsearch, closing the loop between a booking and future search results, and a reaper that expires stale holds and returns their nights to inventory.

Components¶

Edge / CDN. Serves listing photos and static assets close to users and caches anonymous search responses for popular queries (a "Goa, this weekend" search is asked thousands of times). It absorbs a large share of read volume before it reaches origin.

API gateway. Authenticates, rate-limits per account (a guest hammering reserve, a scraper crawling search), and routes reads to the search lane and writes to the booking lane. It is where per-user reservation rate limits live so one client cannot spam holds.

Search service. Translates a user's location + dates + filters into an Elasticsearch query, applies ranking, and paginates. It reads only the index, never the calendar, so browsing at 12k QPS never touches the transactional store. It over-returns candidates and marks availability as "likely," leaving exact confirmation to the detail/reserve step.

Elasticsearch (search index). A denormalized document per listing — location, amenities, price band, capacity, and a coarse availability summary — tuned for geo-bounded, faceted queries in memory. It is a derived, rebuildable view, not a source of truth; a stale or lost index costs relevance, never a booking.

Booking service. The heart of the write path and the only writer to the calendar. It runs the two-phase reserve (hold → confirm), enforces inventory consistency through the database's uniqueness constraint, freezes the price quote into the hold, calls the payment gateway on confirm, and emits events. It is stateless; all durable state lives in the calendar DB.

Inventory / Calendar DB. The system of record for availability and bookings, sharded by listing_id so all of one listing's nights and their contention live together on one shard. A relational store (PostgreSQL/MySQL) or any store offering a per-key transaction with a unique constraint fits, because the arbitration is a single-partition transactional insert, not a distributed one.

Pricing service. Computes a nightly price from a base rate plus demand signals (occupancy, lead time, length-of-stay, local events) and returns a quote_id with a short TTL. Quotes are cached; the booking service stores the quoted total on the hold so mid-checkout price moves never surprise the guest.

Payment gateway (external). Charges the card on confirm. Treated as a slow, fallible external dependency — the design specifically avoids holding any database lock across this call.

Event bus + reaper/workers. Kafka decouples booking state changes from their downstream effects. The reaper consumes hold-created events (or scans by expiry) and releases nights whose holds lapsed; other workers drive the indexer, notifications, and host calendars.

Primary read path (search for a stay)¶

GET /api/v1/search?... hits the CDN; a popular anonymous query may be answered from the edge cache outright.
On miss, the API gateway routes to the search service, which builds a geo-bounded, filtered query — bounding box from lat/lng/radius, predicates for price band, capacity ≥ guests, required amenities, and the coarse availability flag for the requested month.
Elasticsearch returns ranked candidates in ~50–150 ms; the search service paginates with a cursor and returns tens to a few hundred results marked avail: "likely".
When the user opens a listing, GET /listings/{id}/availability and /quote hit the booking and pricing services for the exact free dates and current price — the authoritative check that search only approximated.
Nothing on this path writes to the calendar or takes a lock, so search scales purely by adding search-service and Elasticsearch capacity.

Primary write path (reserve and confirm)¶

POST /api/v1/bookings reaches the booking service with the listing, date range, and the quote_id from the detail page.
The service opens a single-shard transaction on the calendar DB and attempts to insert one night-row per requested night in state HELD, guarded by the unique constraint on (listing_id, date).
If every insert succeeds, the transaction commits: the guest owns those nights for the hold window (e.g. 10 minutes), the frozen price total is recorded, and the service returns 201 HELD. If any night collides, the constraint aborts the whole transaction and the service returns 409 with the conflicting nights — no partial hold, ever.
The client collects payment and calls POST /bookings/{id}/confirm with an idempotency key; the booking service charges the payment gateway, and on success flips the held night-rows to CONFIRMED and clears the expiry.
It emits booking.confirmed to the event bus; the indexer updates the listing's availability in Elasticsearch, and notifications fire to guest and host.
If the guest abandons checkout, the reaper releases the expired HELD nights and emits an availability update so search and future holds see them free again.

Storage choices¶

Availability & bookings: sharded relational (or per-key transactional) store, sharded by listing_id. The arbitration is a single-partition transaction with a unique constraint — all contention for a listing lives on one shard, so no distributed transaction is needed. Relational gives the constraint, the transaction, and secondary indexes for "my bookings" for free. Replicate each shard for durability and read availability.
Search: Elasticsearch. Geo queries, faceted filters, and relevance ranking over 5M small documents, served from memory. Chosen precisely because it is a derived store — rebuildable from the calendar and listing DBs — so we can trade its freshness for cost without risking correctness.
Price quotes: Redis (short TTL). Quotes are computed, cached for their TTL (~10 min), and referenced by quote_id; a cache miss just recomputes. Redis also caches hot anonymous search results and listing metadata.
Events: Kafka. Durable, replayable log so a lagging indexer or reaper catches up rather than losing state changes.

Scaling¶

Read path. Search scales horizontally and independently: add CDN pops for geographic and repeat-query load, add Elasticsearch nodes and shards to grow the index and its QPS, and add stateless search-service instances behind the gateway. At 12k searches/second, most repeat and anonymous queries are absorbed at the edge and Redis, so Elasticsearch sees a fraction of that. Growing from 5M to 15M listings triples the index to ~30 GB — still comfortably in cluster memory across a handful of nodes.

Write path. Global write volume is trivial (~250 reservations/second at holiday peak), so the store is never throughput-bound. Sharding by listing_id is about isolation, not throughput: it keeps one hot listing's contention on one shard so it cannot slow bookings for the rest of inventory. Adding shards spreads listings across more nodes and shrinks the blast radius of a hot shard.

Hot inventory (the real scaling problem). During the New Year peak, LST-42 might draw 40 reservation attempts on its three nights within a few seconds. This is not solved by more machines — 40 QPS is nothing — but by making the losing case cheap and fast: the unique-constraint insert either succeeds or fails in a single-shard transaction in a few milliseconds, so 39 losers get a clean 409 almost instantly rather than queueing behind a held lock. The design goal is that contention resolves at the speed of one row insert, so a stampede on the hottest listing costs 40 fast transactions, not 40 serialized payment waits. Search-side, that same hot listing is cached at the edge, so the browsing storm never reaches the calendar at all.

Operational signals¶

The healthy signal is a steady reservation success rate (holds that convert to confirms) alongside flat search p95 latency — the system is matching supply to demand without friction. The first metric to degrade under trouble is the hold-conflict rate (409s): a climbing 409 rate on specific listings means contention is spiking on scarce inventory, which is expected during a peak but pathological if it appears system-wide (a sign the reaper is failing to release holds, so nights look falsely taken). The misleading metric is aggregate booking QPS — it stays low and calm even while LST-42 is melting under 40-way contention, because global throughput hides per-listing pain; watch per-listing conflict rate, not the sum. The graph an experienced operator opens first during a booking incident is hold-table size and reaper lag: a growing population of expired-but-not-released HELD rows is the classic silent failure, phantom-unavailable inventory that starves both search and new holds while every dashboard says throughput is fine.

Failure modes and resilience¶

Payment gateway slow or down (the threaded case). Guest A holds the nights of the 25th and 26th, then A's card processor hangs for 90 seconds. Because the hold is a committed row with a short expiry, not a lock, nothing else on the shard is blocked — but those two nights stay HELD and unavailable to others. Mitigation: keep the hold TTL short (10 min), and if A's confirm never lands, the reaper releases the nights and they return to inventory; A's client gets a clean "hold expired, please retry." No double-charge, no double-book, no frozen listing.
Reaper failure. Expired holds never release, so inventory silently shrinks — phantom unavailability. Mitigation: run the reaper redundantly, alert on hold-age and hold-table growth, and make release idempotent so a backlog drains safely on recovery.
Search index stale or down. Booking correctness is untouched because the calendar is the source of truth; the cost is relevance — results may omit some free listings or include some just-booked ones. Mitigation: fail open on search (serve broader/cached results, degrade ranking) since a slightly-off search is far better than no search, and rebuild the index from the calendar if it is lost.
Calendar shard down. Listings on that shard cannot be booked. Mitigation: fail closed — reject reserves for that shard rather than risk an unguarded write — and fail over to a replica; serve those listings as browse-only until the primary returns.
Pricing service down. Quotes cannot be computed. Mitigation: fall back to the last-cached quote or the listing's base rate, honoring whatever total is already frozen on existing holds so in-flight checkouts still complete.
Cross-channel oversell. The same room sold on another OTA whose sync lagged. Mitigation: make this calendar the single system of record that all channels reserve against via ARI push, and where true multi-channel independence exists, monitor and absorb rare oversells through relocation/compensation rather than pretending sync is instant.

Where this shows up in production¶

Airbnb — treats the host calendar as the source of truth and indexes only approximate availability for search, confirming exact free dates at the listing and reserve step, exactly the fresh-enough-index split here.
Booking.com — runs a channel manager that pushes availability/rate/inventory (ARI) to keep the same room consistent across many distribution channels, the "one system of record, push don't sync-and-hope" pattern.
Ticketmaster — holds seats in a cart with a countdown TTL during checkout and releases them if payment lapses, the same committed-hold-plus-reaper mechanism under extreme contention.
OpenTable / Resy — reserve a finite, dated unit (a table at a time slot) with a short hold while the diner confirms, the identical uniqueness-per-slot arbitration.
Amazon — chooses when to reserve inventory (add-to-cart vs checkout) as a deliberate oversell-vs-conversion tradeoff, the same hold-timing decision this design makes explicit with hold TTL.
Elasticsearch — powers the geo + faceted search with denormalized documents reindexed asynchronously, deliberately decoupled from the transactional store of record.
Stripe — the external, slow payment call you must never hold a database lock across, and whose idempotency keys let a retried confirm charge exactly once.
Sabre / Amadeus (GDS) — central inventory sources of truth that distribute availability and rates to thousands of downstream sellers, the large-scale version of "one calendar, many channels."