Search Engine Architecture

A search engine is really three systems joined together: a crawler that discovers documents, an indexer that turns those documents into a searchable structure, and a serving layer that answers queries in milliseconds. Each part has different constraints. Crawling is network-bound and politeness-bound. Indexing is storage-bound and CPU-bound. Serving is latency-bound.

Crawling starts with a frontier, which is a queue of URLs waiting to be fetched. The crawler downloads a page, normalises the URL, extracts links, and adds newly discovered targets back into the frontier. It also needs rules. robots.txt and crawl-delay policies limit what should be fetched. Per-host rate limiting prevents the crawler from behaving like a denial-of-service client. Canonical tags, redirects, and duplicate content detection stop the engine from wasting resources indexing many URLs that represent the same page.

Modern crawling is harder than just fetching HTML. Some pages render important content through JavaScript, so the engine may need a separate rendering stage that executes scripts in a controlled environment. That is expensive, which is why engines usually try the cheap path first: parse the raw HTML, extract links and text, and only render when signals suggest the page needs it. This creates a practical tradeoff for site owners. Content hidden behind client-side rendering may still be indexed, but it is slower and less reliable than content available in the initial response.

Indexing turns fetched pages into structures that are efficient for retrieval. The core data structure is the inverted index. Instead of storing each document and scanning them all for every query, the engine stores, for each term, a posting list of documents containing that term. Posting lists also keep metadata such as term frequency, positions, and sometimes field information like whether the term appeared in the title, anchor text, or body. During indexing, the engine tokenises text, handles normalisation such as lowercasing or stemming, detects language, extracts entities, and stores auxiliary signals like freshness, page quality, and link data.

Ranking begins after query analysis. The engine parses the query, corrects spelling when needed, expands synonyms carefully, and tries to infer intent. A query for "python" could mean a programming language or an animal, so historical behaviour and nearby terms matter. Retrieval then gathers a candidate set of documents from the index. Ranking scores those candidates using many signals: textual relevance, link authority, freshness, location, page quality, loading performance, and user context. The engine is not searching the whole web at query time. It is scoring a precomputed subset quickly enough to stay interactive.

Two operational constraints shape the whole design. First, the web changes constantly, so the index is always slightly stale. Engines must decide which pages deserve frequent recrawls and which can wait. Second, quality control matters as much as relevance. If ranking rewarded keyword repetition alone, spam would win. Search systems therefore invest heavily in deduplication, abuse detection, trust signals, and demotion of manipulative patterns.

The serving path is also more layered than it looks. Frontend servers accept the query, dispatch it to index shards, merge partial results, apply final ranking adjustments, and build a results page with snippets, cached metadata, and possibly special result types such as maps or direct answers. Tail latency is the enemy here. One slow shard can delay the whole response, so production search systems use aggressive time budgets, replication, and partial-result strategies.

So the core idea is simple: discover pages, index terms, rank answers. The hard part is doing all three at internet scale while pages change, queries are ambiguous, and latency expectations stay close to instant.