Skip to content

01. URL Shortener — High-Level Design

~15 min read · Part 2 of 4 (Overview → HLD → LLD → Q&A)

This file turns the solutioning narrative from the overview into concrete boxes and the flows between them. Read the architecture top to bottom, then follow a write and a read through it, then look at what happens when pieces fail.

Architecture

                         ┌──────────────┐
        client ────────▶ │  Edge / CDN  │  (caches 301 redirects near users)
                         └──────┬───────┘
                                │ miss / writes
                                ▼
                         ┌──────────────┐
                         │ Load Balancer │
                         └──────┬───────┘
                                │
                 ┌──────────────┼──────────────┐
                 ▼              ▼              ▼
           ┌──────────┐  ┌──────────┐  ┌──────────┐
           │  App     │  │  App     │  │  App     │   stateless app tier
           │  server  │  │  server  │  │  server  │   (mint codes locally)
           └────┬─────┘  └────┬─────┘  └────┬─────┘
                │             │             │
       ┌────────┼─────────────┼─────────────┼────────┐
       ▼        ▼             ▼             ▼        ▼
 ┌──────────┐  ┌──────────────┐   ┌──────────────┐  ┌──────────────┐
 │  Redis   │  │  ID Allocator │   │  URL Store    │  │  Event Bus   │
 │  cache   │  │ (range vendor)│   │ (KV, sharded, │  │ (Kafka etc.) │
 │ code→URL │  │               │   │  replicated)  │  └──────┬───────┘
 └──────────┘  └──────────────┘   └──────────────┘         │
                                                             ▼
                                                     ┌──────────────┐
                                                     │  Analytics    │
                                                     │  pipeline +   │
                                                     │  store        │
                                                     └──────────────┘

Components

Edge / CDN. The outermost cache. Because most redirects are permanent and identical for every user, the CDN can cache the redirect response itself keyed by path (/aZ8kQ2). A large fraction of global redirect traffic never reaches our servers at all, which is the cheapest possible way to absorb a viral link. It also terminates TLS close to users, shaving latency.

Load balancer. Spreads requests across a stateless application tier and health-checks servers out of rotation. Nothing about a request is pinned to a particular server, which is what lets us add capacity by adding boxes.

Application servers. The brains, and deliberately stateless. On a write they mint a code and persist the mapping; on a read they resolve a code via cache then store. Because they hold no per-request state, any server can handle any request and the tier scales horizontally. Each server does hold one small piece of soft state — a locally cached block of IDs from the allocator (see below) — but losing it costs only an unused block, not correctness.

ID Allocator (range vendor). A small, strongly-consistent service that hands out contiguous ranges of integers, e.g. "server A owns 4,000,000–4,003,999." Servers base62-encode integers from their current block to form codes. The allocator is consulted once per block (every few thousand writes), not once per write, so it is nowhere near the hot path. This is the mechanism that makes code generation collision-free without a per-write bottleneck; the LLD details it.

Redis cache. An in-memory code → long_url map fronting the store. This is where the read-heavy load actually lands. It also holds a short-lived negative cache for codes that resolved to 404, so a storm of requests for a bogus code cannot hammer the store.

URL Store. The durable system of record: a sharded, replicated key-value store (or a partitioned relational table) holding the mapping and its metadata. Keyed by short code, its only frequent query is a point lookup by primary key — exactly what a KV store does best.

Event Bus + Analytics. A durable log (Kafka or similar) that decouples click events from redirect serving. The app fires an event after deciding the redirect and moves on; a downstream pipeline consumes the stream to update counts and dashboards. If the pipeline lags or fails, redirects are unaffected and events replay on recovery.

Primary write path (shorten a URL)

  1. POST /api/v1/urls reaches an application server through the load balancer.
  2. If a custom alias was requested, the server does a synchronous existence check against the store; if taken, it returns 409 immediately.
  3. Otherwise the server takes the next integer from its local ID block (fetching a fresh block from the allocator first if the block is exhausted) and base62-encodes it into a short code.
  4. It writes the {code, long_url, owner, created_at, expires_at} row to the URL Store, using a conditional insert so a code is never overwritten.
  5. It optionally warms the cache with the new mapping (a small write, and it means the creator's first test-click is a guaranteed hit).
  6. It returns 201 with the short URL. The whole path is one local operation plus one durable insert.

Primary read path (resolve a redirect)

  1. GET /{short_code} may be served entirely by the CDN if the redirect was cached there — the common case for popular links, and it never touches our origin.
  2. On CDN miss, the request hits an application server, which looks up the code in Redis.
  3. Cache hit: return the redirect. This is the overwhelming majority of origin traffic and costs one in-memory lookup.
  4. Cache miss: read from the URL Store, then backfill Redis with a TTL so subsequent reads hit cache. If the store also has no such code (or it is expired), record a short negative-cache entry and return 404.
  5. Regardless of hit or miss, the server fires an analytics event onto the event bus asynchronously — fire-and-forget, never blocking the response.
  6. The server returns the HTTP redirect. Whether to use 301 vs 302 is a real choice: 301 (permanent) lets browsers and the CDN cache aggressively, maximizing performance, but means a later delete or analytics change may not be seen because clients stop asking; 302 (found/temporary) keeps every click coming back to us — good for accurate analytics and revocable links, at the cost of more origin traffic. A common answer is 302 by default (so analytics and deletes work) with 301 as an opt-in for links the owner marks permanent.

Storage choices

  • Mapping (code → URL): sharded, replicated key-value store. The access pattern is a point lookup by primary key and an occasional insert — no joins, no range scans. A KV store (DynamoDB, Cassandra, or a partitioned relational table) fits perfectly and shards cleanly on the code. Replication (multi-AZ, and ideally multi-region for the read path) gives the durability and availability the mapping demands.
  • Hot mappings: Redis, in-memory. Chosen for microsecond reads and TTL support. It is a cache, not a source of truth — a cold Redis simply means more store reads until it warms, never data loss.
  • Analytics: append-optimized store. Click events are write-heavy, append-only, and queried by aggregation, not by key. A columnar or time-series store (ClickHouse, BigQuery, Druid) behind the event bus suits this far better than the mapping store, and keeping it separate protects redirect latency from analytics load.

Scaling

Read path. Three tiers absorb reads in order: CDN, then Redis, then the store. The CDN handles geographic scale and viral spikes; Redis handles the warm working set; the store sees only cold misses. To scale reads further you add CDN pops, grow the Redis cluster, and add read replicas of the store — all independent, horizontal moves. The short code is a natural, uniform partition key: base62 codes distribute evenly across shards, so no shard becomes a hotspot from key skew (traffic skew is handled by cache, not by partitioning).

Write path. Writes are modest (~400/s peak). The only thing that could bottleneck is code generation, and the per-server range allocation removes it: coordination happens once per block, so write throughput scales with the number of app servers. The store's insert load is well within a single sharded cluster's capacity.

Hot keys. A single viral link is the classic failure, and sho.rt/bf26 is exactly it: 30,000 reads/second concentrated on one code. Defenses stack. The CDN caches it at the edge, so with a modest edge hit ratio the origin sees maybe 1,000/s of those 30,000. Redis holds it as a hot key and serves that remainder from memory. And request coalescing ensures that at the one dangerous moment — the instant the entry is cold or its TTL expires under peak load — only one request reads the store while the other ~30,000 in that window wait for its result, rather than 30,000 simultaneous store reads stampeding a single shard. The design goal is counterintuitive but firm: the hottest link in the system should be the cheapest to serve, because it lives entirely in edge and memory.

Operational signals

The healthy signal is cache hit ratio, which should sit around 99% and barely move when a campaign launches — a launch that raises the hit ratio (one link, requested endlessly) is the system working as designed. The first metric to degrade under trouble is p99 redirect latency: when it climbs while hit ratio holds, the misses are getting expensive (a slow store shard or a saturated Redis node), not more frequent. The misleading metric is average latency — it stays flat because 99% of reads are fast cache hits, hiding a cold-key path that has quietly gone from 5 ms to 200 ms; watch p99, not the mean. The graph an experienced operator opens first during an incident is store read QPS: it should be a flat few-hundred/s even at peak, so any spike toward the tens-of-thousands means the cache is failing to absorb a hot key and a stampede is forming.

Failure modes and resilience

  • Redis outage. Reads fall through to the store, latency rises, and store load spikes. Mitigations: run Redis clustered with replicas so a single node loss is survivable; rely on the CDN to keep absorbing the hottest links; and apply request coalescing to prevent a miss-storm from overwhelming the store during the degraded window.
  • Store shard outage. Links on that shard become unresolvable. Mitigations: replicate each shard (serve reads from a replica if the primary is down) and, for the read path, keep serving from cache/CDN for a grace period even if the backing shard is briefly unreachable, since mappings are immutable.
  • ID Allocator outage. New-link creation stalls once servers exhaust their current blocks — but redirects, the critical path, are entirely unaffected. Mitigation: hand out generous block sizes so servers can keep minting for a long time through an allocator outage, and make the allocator itself a small replicated service.
  • Event bus lag or outage. Analytics falls behind or pauses; redirects keep working because analytics is off the critical path. Events are durably queued and replayed on recovery, so counts catch up rather than being lost.
  • Bad cache invalidation. A deleted link can keep resolving from cache until its entry is evicted. Mitigation: on delete/expiry, actively invalidate the cache entry rather than waiting for TTL, and keep a fast admin kill-switch that can purge a malicious link from cache and CDN immediately.
  • Abuse / malicious links. A shortener is an attractive cloak for phishing. Mitigations (mostly async, per the de-scope): rate-limit creation per account, screen against threat-intel blocklists, and support rapid takedown via the kill-switch above.

Where this shows up in production

  • Bitly — serves redirects from an edge cache with the origin behind it, so viral links are absorbed near users rather than at the database.
  • Twitter t.co — wraps every posted link so the redirect hop can apply safety checks and click measurement, the textbook case for defaulting to a non-permanent redirect to keep analytics flowing.
  • TinyURL — one of the original counter-plus-encoding shorteners, showing how far a simple sequential-ID scheme scales before sequence-hiding matters.
  • YouTube video IDs — 11-character base64 codes are the same idea as our base62 codes: a compact, opaque, collision-free identifier minted from an internal counter.
  • Instagram media IDs (Snowflake) — distributed ID generation that hands each server a slice of the ID space, the same "vend ranges, mint locally" pattern as our allocator, so no server coordinates on the write path.
  • Amazon DynamoDB / Cassandra — the point-lookup-by-key, cleanly-shardable store profile that a mapping table wants; both are common choices behind large shorteners.
  • Redis at CDN/edge providers — used as the in-memory redirect map exactly as here, with TTLs tuned so hot links never fall out of cache.
  • URL-safe hashing in Git — content-addressed short IDs (abbreviated SHA prefixes) are the hash-based alternative to counters, and Git's occasional prefix collisions are the same collision tax discussed above.