Skip to content

03. Web Crawler — Interview Q&A

~15 min read · Part 4 of 4 (Overview → HLD → LLD → Q&A)

These are the questions an interviewer actually asks once the diagram is on the board. Each answer is the strong version, followed by the wrong answer that quietly sinks candidates.

Q1. What is the actual bottleneck in crawling 10 billion pages — bandwidth, CPU, or something else? Politeness scheduling, not raw resources. You may only fetch each domain slowly (default ~1 request/second), so throughput comes from having many domains in flight at once, not from fetching any one faster. To sustain 10,000 pages/second at 1 fetch/host/second you need ~10,000 distinct hosts with URLs ready to go at every instant. That is why the frontier is organized by host with a per-host politeness gate, and why the design's center of gravity is the scheduler. A web crawler is not a fetching problem; it's a scheduling problem. Common wrong answer to avoid: "Add more fetcher machines / more bandwidth." Fetcher CPU and bandwidth sit low while the crawl is behind, because the ceiling is host-breadth under the politeness cap, not fetcher horsepower.

Q2. How is the URL frontier structured to satisfy priority and politeness at the same time? Two layers. Front queues encode priority — a re-crawled news homepage lands in the top band, ordinary discovered links in a middle band, trap-suspected URLs at the bottom. Back queues encode politeness — exactly one queue per active host, released through a heap keyed by each host's next_fetch_at, so a fetcher only ever receives a URL from a host whose delay has elapsed. Separating "what next" (front) from "when allowed" (back) is what lets both hold simultaneously; a single priority queue would either fetch a hot host too fast or block on its delay. Common wrong answer to avoid: "One priority queue ordered by importance." It cannot enforce per-host rate limits — the top of the queue is often many URLs from the same popular host, which then gets hammered.

Q3. How do you deduplicate URLs across tens of billions of discovered links, and what does it cost? Track a 64-bit fingerprint of each canonicalized URL. Storing ~100 billion exactly is 800 GB; a Bloom filter at a 0.1% false-positive rate is ~180 GB and fits in the fleet's RAM. Use both in tiers: the Bloom filter is the in-memory fast reject (no false negatives, so a SEEN answer needs zero disk), and only its rare MAYBE_NEW answers touch an exact fingerprint store on SSD to distinguish a true new URL from a false positive. The common "already seen this link" case — which is most of the thousands of links on every page — stays in nanoseconds of RAM. Common wrong answer to avoid: "Keep a hash set of all URLs in memory." 100B full URLs is ~8 TB, and even 8-byte fingerprints are 800 GB — you either blow the RAM budget or add a disk hop to the hottest check in the system.

Q4. A false positive in the Bloom filter drops a real URL. Why is that acceptable? Because the cost is bounded and self-healing. At 0.1% false positives on a 10B-page crawl, ~10 million genuinely-new URLs get wrongly judged "seen" and dropped. But any page worth indexing is linked from many other pages, so it is very likely rediscovered through a different link whose fingerprint doesn't collide. The exact-tier confirmation on MAYBE_NEW answers also catches most would-be drops. You are trading a sliver of coverage for a seen-set that fits in RAM — and coverage of the web is inherently probabilistic anyway. Common wrong answer to avoid: "Bloom filters are unreliable, so use an exact set." That quadruples the memory for a guarantee the crawl doesn't need, since web coverage is already best-effort and links are redundant.

Q5. How do you keep 10,000 news domains fresh hourly while still crawling 10 billion pages and never over-fetching a domain? Freshness comes from fetching smarter, not harder, because harder is forbidden. Re-crawled news pages enter the frontier's top priority band so they preempt ordinary discovery. Each domain still drains at its polite rate: 100 due pages at 1/second sweep in 100 seconds, inside the hour. When a domain floods — say 5,000 pages come due in a breaking-news surge — a full sweep would need ~83 minutes at 1/second, past the deadline, so you refresh a prioritized subset (homepage, section pages, newly-linked articles — the top ~1,000, sweeping in ~17 minutes) and cap how many top-band slots one domain can hold so it doesn't starve the other 9,999. If a site's robots.txt permits a 0.3s crawl-delay, that domain can safely sweep 5,000 pages in ~25 minutes. The news tier is only ~280 pages/second — 3% of the 10,000/second budget — so discovery fills the slots it leaves idle. Common wrong answer to avoid: "Crawl the news sites faster / bump their rate to hit the deadline." Over-crawling earns 429s and eventually an IP ban, which loses that domain entirely — a far worse freshness outcome than a slightly late sweep.

Q6. What is the freshness-versus-politeness tradeoff, in numbers? They pull directly against each other. A news domain with 5,000 pages at the polite 1 fetch/second takes ~83 minutes to sweep — you miss the hourly deadline. Push to 3 fetches/second and it's ~28 minutes — you make the deadline, but you may exceed what a mid-size server tolerates and trigger rate-limiting. The resolution isn't a single global speed knob; it's per-page adaptive scheduling (spend fetches where content actually changes) plus prioritized subset refresh plus honoring each site's declared crawl-delay. Freshness is bought by targeting the fetch budget, not by raising the per-domain rate. Common wrong answer to avoid: "Set a short global re-crawl interval for everyone." That wastes most of the budget re-fetching static pages that never change, and still can't beat the per-domain politeness cap for a large site.

Q7. How do you detect that two different URLs serve the same content? URL dedup and content dedup are separate problems. Even after URL dedup, mirror sites, print-view pages, and session-id or tracking-parameter variants deliver identical or near-identical bodies. So fingerprint the content too: an exact hash of the normalized body collapses byte-identical duplicates, and a SimHash (locality-sensitive) with a small Hamming-distance threshold catches near-duplicates that differ only in ads or timestamps. Aggressive URL canonicalization before fingerprinting (lowercase host, strip default ports and tracking params, resolve ./..) removes most of these before a fetch even happens. Common wrong answer to avoid: "URL dedup handles it." Distinct URLs routinely serve the same page; without content fingerprinting you index the same article many times and waste crawl budget on mirrors.

Q8. Two fetchers pull work at the same instant and both target the same popular host. How do you prevent a politeness violation? It's prevented structurally, not by a check that can be skipped under load. get_next reschedules the host to now + crawl_delay in the host heap before it returns the lease, and the heap only ever exposes a host whose next_fetch_at <= now. So the moment one fetcher is cleared for a host, that host disappears from the "due" set until its delay elapses; a second fetcher polling in the same instant finds it not due and gets a different host. Politeness is an invariant of the data structure. Common wrong answer to avoid: "Each fetcher checks a last-fetched timestamp before fetching." That read-then-act has a race: both fetchers read the old timestamp, both decide it's fine, both fetch.

Q9. A fetcher crashes mid-download. Do you lose that URL or crawl it twice? Neither, by design. URLs are handed out under a lease with a visibility timeout. If the fetcher acks, the URL is done; if it crashes and never acks, the lease expires and the frontier re-queues the URL, so it's retried rather than lost. The one residual case is a slow-but-alive fetch whose lease expires and gets re-leased, causing a double fetch — made harmless by keying the content store on URL fingerprint + version, so a second store is idempotent and the late ack is ignored. Common wrong answer to avoid: "Delete the URL from the queue when you hand it out." Then a crash silently loses the URL, and it's only re-crawled if some other page happens to link it again.

Q10. What are crawler traps and how does the design survive them? Traps are URL spaces that are effectively infinite: a dynamic calendar with a "next month" link forever, or session-id / tracking parameters that mint a distinct URL on every request. They balloon the frontier and can consume the crawler. Defenses stack: canonicalization strips the parameters that generate most variants; per-host URL budgets and a max crawl depth cap how much any one space can enqueue; low-value URL patterns are deprioritized to the bottom band; and content-dedup collapses the identical bodies these traps usually serve, so even if URLs slip through, they don't get stored or re-crawled. Common wrong answer to avoid: "Just set a global max depth." Depth alone doesn't stop session-id explosions (which stay shallow) or distinguish a legitimately deep site from a trap.

Q11. Why is DNS a concern, and how do you keep it off the critical path? At 10,000 fetches/second against distinct hosts, naive per-fetch DNS lookups generate enormous query volume and make an external resolver a latency and throttling dependency on the hottest path. Cache aggressively with long TTLs (host→IP rarely changes within a crawl window), run a dedicated resolver fleet, and pre-resolve hosts before their URLs reach the fetchers, so a fetch almost always hits a warm cache entry rather than waiting on a network round-trip you don't control. Common wrong answer to avoid: "DNS is instant, ignore it." Uncached DNS at this rate becomes a bottleneck and a way to get throttled by your upstream resolver.

Q12. Where do you store 300 TB of page content, and why not a database? A distributed blob/object store, with bodies gzip-compressed from ~100 KB to ~30 KB, giving ~300 TB per full 10B-page snapshot and low petabytes with a few historical versions. The access pattern is write-once, read-by-key, keep-versions — no joins, no ad-hoc queries — which is exactly a blob store's sweet spot and far cheaper per terabyte than a database. Metadata that is queried by key (last-crawled time, content hash, next-crawl time) lives in a separate KV store keyed by URL fingerprint. Common wrong answer to avoid: "Put the pages in a big relational/NoSQL database." You pay database overhead and cost for what is fundamentally large-object storage read by key, and you strain it with multi-KB blobs it isn't built for.

Deeper follow-ups

  • How would you crawl JavaScript-rendered pages (headless-browser fetches are ~10× more expensive) without letting them starve the plain-HTML crawl?
  • How do you partition the frontier and seen-set across a cluster while keeping each host's politeness state on a single node?
  • How would you make the whole crawl distributed across data centers — who owns which hosts, and how do you avoid two data centers crawling the same host simultaneously?
  • How would you estimate a page's change rate more rigorously than interval-halving (e.g. a Poisson change model over observed history)?
  • How do you prioritize the discovery crawl itself — what makes an undiscovered URL "high value" before you've seen its content?
  • How would you detect and gracefully handle a site that starts returning 429s, and feed that back into that host's scheduled rate?

How this round is scored

Interviewers use the web crawler to see whether you recognize that the constraint is politeness, and therefore the problem is scheduling. The strong signal is reaching for a host-partitioned frontier with a per-host politeness gate early, and explaining throughput as host-breadth rather than fetcher count — candidates who "add more machines" are reading the misleading metric. Seniority shows in the tradeoff discussions: Bloom-versus-exact for the seen-set with the 180 GB / 800 GB numbers, freshness-versus-politeness with the 83-minute / 17-minute sweep math, and content-dedup as a problem separate from URL-dedup. The failure-mode thinking — crawler traps, lease expiry and double-fetch idempotency, the politeness race between two fetchers, getting banned for over-crawling — separates people who have operated a crawler from those who have only drawn one. Doing the back-of-envelope math out loud (10,000 hosts in flight, 300 TB compressed, ~180 GB of Bloom bits) and using it to justify the structure, rather than as decoration, is what pushes an answer from correct to senior.