01. Vector Search Engine — High-Level Design¶

~15 min read · Part 2 of 4 (Overview → HLD → LLD → Q&A)

This file turns the solutioning narrative into concrete boxes and the flows between them. Read the architecture top to bottom, then follow a query and an upsert through it, then look at what happens when pieces fail — including what the 10k-QPS, 95%-recall scenario does to the fan-out.

Architecture¶

                          ┌──────────────┐
        query vector ───▶ │  API Gateway │
                          └──────┬───────┘
                                 ▼
                          ┌──────────────┐      ┌──────────────┐
                          │ Query        │◀────▶│ Metadata /   │  filters, payloads
                          │ Coordinator  │      │ payload store│
                          └──────┬───────┘      └──────────────┘
             scatter (fan-out to ALL shards)          ▲
        ┌────────────┬──────────┼──────────┬──────────┘ (hybrid: keyword hits)
        ▼            ▼          ▼          ▼          ┌──────────────┐
  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌────────┐  │ Keyword /    │
  │ Shard 1  │ │ Shard 2  │ │ Shard 3  │ │ … 20   │  │ BM25 index   │
  │ HNSW/IVF │ │ HNSW/IVF │ │ HNSW/IVF │ │ shards │  └──────────────┘
  │ ×5 repl. │ │ ×5 repl. │ │ ×5 repl. │ │        │
  └────┬─────┘ └────┬─────┘ └────┬─────┘ └───┬────┘
       │            │            │           │  gather + merge top-k
       └────────────┴─────┬──────┴───────────┘  (back at Coordinator)
                          │
                 ┌────────┴────────┐        ┌──────────────┐
                 │ segment load    │◀───────│ Index Builder│  (offline)
                 │ (from object    │        │ HNSW build / │
                 │  storage)       │        │ IVF training │
                 └─────────────────┘        └──────┬───────┘
                          ▲                         │
                 ┌────────┴────────┐        ┌───────▼──────┐
                 │ Object storage  │◀───────│ Raw vector   │  system of record
                 │ (immutable      │        │ store (durable)│
                 │  segments)      │        └──────────────┘
                 └─────────────────┘

Read it top to bottom. A query enters at the gateway and lands on a coordinator, which is the brain of the read path: it resolves filters against the metadata store, decides the recall/latency knob, and — because a query's neighbors can live on any shard — scatters the query to all 20 shards in parallel. Each shard holds one twentieth of the billion vectors as an in-memory ANN index (HNSW or IVF) behind ~5 replicas, does a local approximate top-k, and returns its candidates. The coordinator gathers the 20 partial results, merges them into a single global top-k, optionally fuses in keyword hits from the BM25 index for hybrid queries, hydrates payloads from the metadata store, and returns. Off to the side and off the hot path, an index builder consumes raw vectors from the durable store, builds immutable index segments into object storage over hours, and shards load those segments — the slow, batch, correctness path that the fast serving path never blocks on.

Components¶

API Gateway. Terminates TLS, authenticates, rate-limits per tenant, and validates the query shape (vector dimension must match the index, k within bounds). It exists to keep malformed or abusive traffic away from the coordinator, which is doing expensive work per request.

Query Coordinator. The scatter-gather engine and the only stateful-per-request component. It fans a query out to all shards, sets the per-shard efSearch/nprobe from the recall knob, merges partial top-k lists, runs hybrid fusion, and enforces the deadline (a shard that misses its slice of the latency budget is dropped or hedged). It holds no vector data itself, so it scales horizontally behind the gateway.

Shards (ANN index nodes). Each owns a disjoint slice of the corpus — 50M vectors — as an in-memory HNSW graph or IVF-PQ index. A shard answers exactly one question: "given this vector and this recall setting, what are your local top-k?" Replicas of a shard are identical copies; they exist to serve QPS and to give the coordinator a second target to hedge against a slow replica.

Keyword / BM25 index. A conventional inverted index over the same documents' text, queried in parallel with the vector shards for hybrid search. It exists because dense vectors are blind to exact-term matches — a part number, a rare proper noun, an exact quote — that a sparse keyword index nails.

Metadata / payload store. A key-value or document store holding each vector's structured attributes (for filtering) and its display payload (for hydration). Kept separate from the index because filters and payloads change independently of the vectors and because the index should stay a lean numbers-and-edges structure.

Index Builder. The offline pipeline that turns raw vectors into servable segments: it trains IVF centroids or builds the HNSW graph, applies quantization, and writes immutable segments to object storage. It runs on batch compute, decoupled entirely from serving, because a billion-vector build takes hours and must never contend with query traffic.

Raw vector store + object storage. The durable system of record. Raw full-precision vectors live in a durable store; built index segments live as immutable objects. If every serving shard died, the corpus and the ability to rebuild every index survive here — the in-memory shards are derived, replaceable state.

Primary write path (upsert a vector)¶

POST /upsert reaches the coordinator through the gateway.
The coordinator writes the full-precision vector and its payload to the durable raw store first — this is the acknowledgeable, never-lose-it step, and it is what a future rebuild reads from.
It routes the vector to its owning shard (by a hash of the vector id, so the same id always lands on the same shard) and appends it to that shard's incremental in-memory index — for HNSW, that means inserting the node and wiring its edges live; the vector is searchable within seconds.
The vector's structured attributes go to the metadata store and its text to the keyword index, so filters and hybrid search see it too.
The coordinator acks. Deletes follow the same route but mark the vector with a tombstone in the shard rather than surgically removing it from the graph (removing a node from an HNSW graph cleanly is expensive; tombstones are filtered out of results and reclaimed at the next rebuild).

Primary read path (a k-NN query)¶

POST /query with a 768-dim vector reaches a coordinator.
If the query carries a filter, the coordinator decides the strategy: for a loose filter it lets shards search then drops non-matching results; for a very selective filter it may pre-fetch the matching id set from the metadata store and pass it as an allow-list so shards don't waste the candidate budget on vectors that will be filtered out.
The coordinator scatters the query to all 20 shards in parallel, stamping each with the efSearch that the recall monitor says currently yields 95%, and a deadline.
Each shard runs its local ANN search — an HNSW greedy walk or an IVF probe of the nearest nprobe clusters — and returns its top-k candidates with scores. Slow shards are hedged: if a replica hasn't answered by the deadline's soft edge, the coordinator fires the same sub-query at a second replica and takes whichever returns first.
The coordinator gathers all partial lists and merges them into one global top-k by score. For a hybrid query it also queried the BM25 index and now fuses the dense and sparse rankings with Reciprocal Rank Fusion.
It hydrates the surviving ids with payloads from the metadata store and returns the results. The whole thing is a parallel fan-out plus a merge, so end-to-end latency is roughly the slowest un-hedged shard plus a small merge cost.

Storage choices¶

In-memory ANN index (per shard): HNSW graph or IVF-PQ, in RAM. Chosen because the latency SLO demands the index be walked in memory — disk-resident search cannot hit tens of milliseconds at a billion vectors. HNSW when recall and incremental inserts matter more than RAM; IVF-PQ when RAM is the binding constraint and a re-rank pass can recover recall.
Durable raw vectors: object/blob store or a columnar store. Full-precision float32, write-once, read only by the index builder. Cheap, durable, never on the hot path — its only job is to make rebuilds possible and losses recoverable.
Built segments: immutable objects in object storage. Versioned and content-addressed so a shard can atomically swap from segment v41 to v42 and roll back if v42 regresses recall.
Metadata / payloads: key-value or document store. Point-lookup by vector id for hydration, plus secondary indexes for filter attributes. Kept off the vector index so payload churn never touches the graph.
Keyword index: inverted index (Lucene-family). The right structure for BM25 term scoring, and a separate system so its query load and the vector load scale independently.

Scaling¶

Read path. Two dimensions move independently. To hold recall as the corpus grows, you add shards — going from 1B to 2B vectors means going from 20 to ~40 shards so each still fits ~165 GB in RAM. To serve more QPS at fixed corpus, you add replicas — the scenario's 10k QPS needs ~5 replicas per shard (200k internal shard-queries/second ÷ 2k per replica); doubling to 20k QPS means going to ~10 replicas per shard, a pure horizontal add. The coordinator tier scales by adding stateless coordinators behind the gateway. Note the fan-out cost that scaling shards imposes: every extra shard adds one more sub-query to every query, so the internal query rate is QPS × shard_count — at 40 shards and 10k QPS that is 400k internal queries/second — which is exactly why you shard only as far as RAM forces you to, not further.

Write path. Upserts are modest and append to per-shard incremental indexes, so throughput scales with shard count. The expensive write is the full rebuild, which is deliberately offline: it runs on separate batch compute reading from the durable store, produces new segments, and shards hot-swap them, so ingestion pressure never touches serving latency.

The recall knob under load. Recall and QPS trade against each other on fixed hardware. Dropping efSearch from 100 to 60 cuts per-shard work ~40% (recall falls from ~95% to ~92%), letting the same replicas absorb a QPS surge without adding machines — a lever the coordinator can pull automatically under overload, shedding recall instead of dropping requests. Pushing efSearch to 200 buys ~98% recall but roughly halves the QPS each replica sustains, so it doubles the replica count for the same throughput. Naming that number movement — efSearch 100→60 frees ~40% capacity at a 3-point recall cost — is the difference between "we tune recall" and a senior answer.

Operational signals¶

The healthy signal is measured recall@10 against a fixed ground-truth probe set, sampled continuously and sitting steady at ~95%; a system whose recall holds while QPS climbs is working as designed. The first metric to degrade under trouble is p99 query latency, and specifically the gap between p50 and p99 — because a scatter-gather query is only as fast as its slowest shard, a single slow replica widens the tail long before it moves the median. The misleading metric is mean latency: it stays flat because most shards are fast, hiding one replica that has gone from 3 ms to 80 ms and is silently dragging p99 on every query that touches it; watch the tail and the per-shard latency distribution, not the average. The graph an experienced operator opens first during an incident is per-shard p99 latency, broken out by replica — it immediately shows whether the tail is one sick replica (hedge or evict it), one hot shard (its data is drawing disproportionate queries), or a system-wide efSearch that crept up after a bad recall-tuning push.

Failure modes and resilience¶

A single slow shard replica. Because every query fans out to all shards, one slow replica poisons the tail of every query. Mitigation: request hedging — the coordinator sends the sub-query to a second replica when the first misses a soft deadline and takes the faster response — plus ejecting a replica from rotation when its p99 crosses a threshold.
The 10k-QPS fan-out overwhelms shards. The scenario's steady state is 200k internal shard-queries/second across 100 replicas. If a wave of traffic or a replica outage pushes per-replica load past its 2k/second capacity, latency spikes fleet-wide. Mitigation: the coordinator sheds recall before it sheds requests — dropping efSearch from 100 to 60 reclaims ~40% of per-replica capacity, absorbing a ~1.6× surge while recall dips to ~92% and a warning fires, which is a far better failure than timing out queries. Autoscaling replicas is the slower, durable fix.
A shard's index doesn't fit RAM after growth. Silent recall or latency death as the OS starts paging the index to disk. Mitigation: alarm on index-size-vs-RAM headroom per shard and split the shard (re-shard) before it crosses the line; this is why RAM headroom is a monitored SLO, not a static assumption.
A bad index build regresses recall. A rebuild with mis-trained IVF centroids or a too-low efConstruction can silently drop recall from 95% to 85%. Mitigation: segments are immutable and versioned; the recall probe runs against a new segment in shadow before it takes traffic, and a regression triggers an atomic rollback to the previous segment.
Metadata / keyword store outage. Filters and hybrid fusion degrade, but pure vector search still works. Mitigation: fail open to vector-only results with a flag rather than failing the whole query, so a keyword-index blip doesn't take down semantic search.
Total shard-tier loss. In-memory indexes are derived state; a region losing its shards loses no data because the durable raw store and immutable segments survive. Recovery is reloading segments from object storage — minutes to tens of minutes for a billion vectors, which is why segments live pre-built and not rebuilt-on-recovery.

Where this shows up in production¶

Pinecone — separates the query coordinator from stateless "pods" that each hold an index shard, and sizes pod replicas for QPS while sizing pod count for corpus size, the same shard-for-RAM / replica-for-QPS split used here.
Meta FAISS — the library most large in-house engines are built on; its IVF-PQ and HNSW implementations are the reference for the memory-vs-recall quantization tradeoff, and its GPU indexes push the per-shard throughput ceiling.
Milvus / Zilliz — makes the offline "build immutable segments, load and hot-swap them" lifecycle explicit, with a separate build tier so ingestion never contends with search.
Weaviate — ships HNSW with live incremental inserts and tombstone-based deletes, exactly the "insert live, reclaim at rebuild" freshness path, and native hybrid search with RRF fusion.
Elasticsearch / OpenSearch kNN — bolts HNSW onto a keyword engine, so hybrid search is dense-plus-BM25 fusion in one system, showing why the keyword index sits beside the vector shards rather than being replaced by them.
Google ScaNN — the anisotropic-quantization line of research behind Google's embedding retrieval, tuning the quantizer to preserve inner-product ranking, the sharpest version of "compress the vector but keep the ranking."
pgvector — brings ANN into Postgres with IVFFlat and HNSW, the case where the metadata store and the vector index are deliberately the same system, trading peak scale for one-store simplicity.
Spotify Annoy / Vespa — Annoy's mmap'd tree forests popularized loading immutable index files at read time; Vespa runs vector search and structured filtering together at serving time, the model for evaluating filters alongside the vector search rather than after it.