URL: https://webscraping.spaceBest viewed at 1024×768

Scaling Published Jul 13, 2026 · Updated Jul 13, 2026 · 23 min read · 5,167 words

Building search infrastructure for 1M+ queries a month: the stuff nobody tells you

A deep, no-bullshit technical dive into building search infrastructure that handles over 1 million queries per month. Inverted index internals, posting list compression, WAND/MaxScore scoring, Go hot-path code, five-layer cache hierarchies, mmap and SSD reality, tiered segment merging, tail latency, and the non-obvious stuff that actually kills you at scale.

One million queries a month sounds like a lot. It is not. Let me show you why, and then let me show you what actually hurts.

The math that kills your assumptions

1,000,000 queries / 30 days / 24 hours / 3600 seconds = 0.385 queries per second.

That's nothing. A single Go process on a $5 VPS can handle 0.385 qps in its sleep. A Raspberry Pi could do it. So why are you reading this?

Because average is a lie.

Real traffic is not flat. Real traffic is a diurnal curve. It dips to near zero at 4 AM and spikes to 5x the average at 8 PM. People search in bursts. One user opens your site and fires 6 queries in 12 seconds, then disappears for 40 minutes. Another user pastes a 14-word query, waits 200ms, refines it, waits 200ms, refines again.

So 0.385 qps average becomes:

Peak sustained: 2 to 5 q/s (5x average)
Burst spikes: 10 to 20 q/s (a user session firing rapid queries)
Worst case you actually care about: 50 q/s for 30 seconds when a newsletter goes out or someone tweets about you

And here is the part that ruins people. Your p99 latency target is not your average-case latency target. At 1M queries/month:

p50: 2.1ms. Fine. Nobody cares.
p90: 5.1ms. Still fine.
p99: 12.4ms. You're proud of this. Don't be.
p99.9: 28.7ms. This is the one users feel.
p99.99: 94.3ms. This is the one that makes them leave.

A user session fires 4 to 8 queries in a few seconds. The probability that at least one of those 8 queries hits the p99 is 1 - (0.99)^8 = 7.7%. One in thirteen sessions hits your p99. That's 77,000 sessions a month seeing your worst case. p99 is a vanity metric at this scale. Measure p99.9. Optimize p99.9.

What you are actually building

You are building a system that takes a string from a user, turns it into a list of matching documents, ranks them, extracts snippets, and returns JSON in under 8 milliseconds at p99.9. That's the whole job. Everything else is implementation.

The path is:

CDN terminates TLS and does edge caching for repeated queries
Gateway receives the HTTP request, does auth and rate limiting
Coordinator parses the query, rewrites it, checks the result cache
If cache miss, fan out to N shards in parallel
Each shard reads posting lists, scores documents with WAND, returns its local top-k
Coordinator merges the per-shard top-k into a global top-k
Snippet builder fetches document fields and generates highlighted snippets
Response is serialized and sent back

The latency budget for all of this:

Total budget: 8ms. The two big eaters are posting list reads (2.8ms) and WAND scoring (1.9ms). Everything else is noise. If you cannot get those two under control, nothing else matters.

The inverted index: your entire system in 200 lines

Everything in search is a posting list. A posting list is a sorted list of document IDs that contain a given term. When the user searches for "golang channels", you:

Look up the posting list for "golang"
Look up the posting list for "channels"
Intersect them
Score the surviving documents
Return the top-k

That's it. That's search. The entire complexity is in doing steps 1 through 4 fast enough.

Posting list compression: VByte and beyond

A posting list for a common term like "the" in a 10M document corpus has 10,000,000 entries. If you store each doc ID as a 4-byte integer, that's 40 MB for one term. You have millions of terms. You are dead.

The fix is delta encoding plus variable-byte encoding. Instead of storing absolute doc IDs, store the gap between consecutive IDs. Then encode each gap with the minimum number of bytes.

// VByte encode: a doc ID delta into 1-5 bytes
func vbyteEncode(delta uint32, buf []byte) int {
    n := 0
    for delta >= 0x80 {
        buf[n] = byte(delta) | 0x80
        delta >>= 7
        n++
    }
    buf[n] = byte(delta)
    return n + 1
}

// VByte decode: read one delta from the buffer
func vbyteDecode(buf []byte) (uint32, int) {
    var x uint32
    var s uint = 0
    for i := 0; i < len(buf); i++ {
        b := buf[i]
        if b < 0x80 {
            return x | (uint32(b) << s), i + 1
        }
        x |= (uint32(b) & 0x7f) << s
        s += 7
    }
    return 0, 0
}

VByte gets you 3 to 4x compression on typical posting lists. For 10M documents, "the" goes from 40 MB to about 11 MB. Better. Still not enough.

The next level is PFOR-Delta (Patched Frame of Reference). Instead of variable-length bytes, you pack deltas into fixed-width bit fields. Most deltas are small (1 to 64), so they fit in 6 bits. The exceptions (rare large gaps) get patched out into a separate exception list. PFOR gets you 8 to 12x compression and, critically, it is SIMD-friendly. You can decode 128 deltas in a single AVX2 instruction. This matters because posting list decoding is often 30 to 40% of your query CPU.

I have seen teams spend two weeks tuning their Go garbage collector and zero weeks on posting list compression. The compression fix is worth 10x more. Start there.

Block-based skipping: the real performance win

Compression saves memory. Block skipping saves CPU. Here is the idea.

You divide each posting list into blocks of 128 or 256 entries. For each block, you store:

The block's maximum doc ID (the "block max")
The block's maximum possible score contribution (the "block max score")
A pointer to the block's start in the compressed data

When you are doing a multi-term intersection and you know the current top-k threshold is 15.3, and the block-max-score for the current block of the "golang" posting list is 3.2, you can skip the entire block. The documents in this block cannot possibly make the top-k. You jump to the next block in one pointer dereference.

This is the foundation of WAND and Block-Max WAND. More on that below. But the key insight is: skipping is worth more than scoring. A query for "the best golang tutorial" matches 4 million documents. Your top-k is 20. You need to score maybe 300 of those 4 million. The other 3,999,700 are skipped entirely.

WAND and MaxScore: how to not score 4 million documents

The naive approach to top-k retrieval: score every matching document, sort, take the top-k. For "the best golang tutorial" with 4M matches, that's 4M scoring operations. At 100ns per scoring op (optimistic), that's 400ms. You're 50x over budget.

WAND (Weak AND) fixes this. Here is the algorithm in plain English:

Maintain a current top-k heap with a threshold score T (the k-th best score so far)
For each term in the query, position a cursor at the start of its posting list
Find the pivot document: the document that, if scored, has the best chance of beating T
If the sum of all term block-max-scores at the pivot is less than T, skip forward. No document at or before this position can beat T.
If the sum exceeds T, score the pivot document. If its actual score beats T, insert it into the heap.
Advance the cursor with the smallest doc ID. Repeat.

type WandScorer struct {
    cursors  []*PostingCursor  // one per query term
    heap     *MinHeap           // size k, tracks current top-k
    k        int
}

func (w *WandScorer) Evaluate() []ScoredDoc {
    for {
        pivot, ok := w.findPivot()
        if !ok {
            break
        }
        // sum of block-max-scores at pivot position
        upperBound := w.blockMaxSum(pivot)
        if upperBound <= w.heap.Threshold() {
            // cannot beat current top-k, skip to next pivot block
            w.advanceAll(pivot)
            continue
        }
        // might beat top-k, actually score the document
        score := w.scoreDoc(pivot)
        if score > w.heap.Threshold() {
            w.heap.Push(ScoredDoc{ID: pivot, Score: score})
        }
        w.advanceSmallest(pivot)
    }
    return w.heap.Sorted()
}

On typical queries, WAND evaluates 2 to 8% of matching documents. That 4M-document query now scores 80,000 documents. At 100ns each, that's 8ms. Close. Block-Max WAND (BMW) pushes it further by using block-level max scores to skip entire blocks without even looking at individual documents. BMW evaluates 0.5 to 2% of matches. Now you're scoring 8,000 documents in 0.8ms. You have budget left.

The DAAT vs TAAT debate (it is not really a debate)

DAAT (Document-At-A-Time) scores one document fully before moving to the next. TAAT (Term-At-A-Time) processes one posting list fully before moving to the next. WAND is DAAT. MaxScore is TAAT-ish.

DAAT wins for top-k queries because early termination works naturally. TAAT wins for complex queries with many terms because you can skip terms whose contribution is below the threshold. In practice, DAAT with block-max skipping (BMW) wins 90% of the time. Use DAAT. Move on.

BM25: the ranking function that refuses to die

BM25 has been around since 1994. It is still the default in Lucene, Elasticsearch, and most search engines. Not because it's the best. Because it's robust, parameter-free, and good enough.

score(D, Q) = sum over terms t in Q of:
    IDF(t) * (f(t, D) * (k1 + 1)) / (f(t, D) + k1 * (1 - b + b * |D| / avgdl))

Where:

f(t, D) = term frequency of t in document D
|D| = document length
avgdl = average document length
k1 = 1.2 (term frequency saturation)
b = 0.75 (length normalization)
IDF(t) = log((N - df + 0.5) / (df + 0.5) + 1)

BM25 works because it captures three things that matter:

Rare terms matter more (IDF component)
More occurrences matter more, but with diminishing returns (the saturation curve from k1)
Longer documents are penalized (the b parameter normalizes by document length)

The parameters are not sacred. For short queries against long documents, drop b to 0.5. For product search where titles are short and dense, push k1 to 2.0. For full-text search of articles, the defaults are fine.

Beyond BM25: what to add when BM25 is not enough

BM25 scores text relevance. It does not know about freshness, authority, personalization, or business rules. You layer those on top:

func finalScore(bm25 float64, doc Document, query Query) float64 {
    s := bm25
    // freshness decay: documents lose 30% of their score per 90 days
    ageDays := time.Since(doc.UpdatedAt).Hours() / 24
    s *= math.Exp(-ageDays / 300)  // exponential decay, tau=300 days
    // authority boost: verified sources get 1.2x
    if doc.Authority > 0.7 {
        s *= 1.2
    }
    // exact title match: big boost
    if strings.Contains(strings.ToLower(doc.Title), query.Original) {
        s *= 2.5
    }
    // business rules: promoted content
    if doc.Promoted {
        s *= 1.5
    }
    return s
}

At 1M queries/month, you do not need learning-to-rank. You need BM25 plus 4 to 6 hand-tuned signals. LTR becomes useful at 10M+ queries/month where you have enough click data to train a model. Below that, hand-tuned heuristics beat ML because you can debug them.

Sharding: how to split your index

You have 10M documents. Each shard holds 1M. You have 10 shards. The coordinator fans out every query to all 10 shards, each returns its local top-100, and the coordinator merges into a global top-20.

The wall-clock time of a fan-out query is the maximum of the per-shard times, not the average. Shard 2 took 3.8ms. Your query took 3.8ms plus merge overhead. The slowest shard is your latency. This has implications:

The cold shard problem

Some shards are hot. They hold documents for trending topics. They get hit harder because their posting lists are longer and their block-max scores are higher. Other shards are cold. They hold documents nobody searches for. They return in 0.5ms.

The cold shards are not your problem. The hot shards are. If shard 2 consistently takes 3.8ms while everything else takes 1.5ms, you have two options:

Split the hot shard. Move half its documents to a new shard. Now you have 11 shards. The hot one is half the size and twice as fast.
Replicate the hot shard. Put 2 replicas of shard 2 behind a load balancer. Each replica handles half the traffic. The posting list read time doesn't change, but you double your throughput for that shard.

I prefer option 1. Smaller indexes are faster indexes. The relationship between index size and query latency is roughly O(log n) for the index lookup plus O(matches) for scoring. Both shrink when you split.

Document-based vs term-based partitioning

Document-based partitioning: each shard holds a random subset of documents. Every query hits every shard. Simple. Fan-out is N. This is what Elasticsearch does by default.

Term-based partitioning: each shard holds all documents for a subset of terms. A query for "golang channels" hits only the shards that hold "golang" and "channels". Fan-out is the number of query terms, not the number of shards. For 2-term queries against 100 shards, you hit 2 shards instead of 100.

Term-based partitioning sounds great until you realize that common terms ("the", "is", "of") live on one shard that gets 100x the traffic of every other shard. You have recreated the hot shard problem, but worse, because now it's structural instead of statistical.

Use document-based partitioning. Accept the fan-out. The coordinator is cheap.

The cache hierarchy: five layers you need

Five cache layers. Each one catches what the one above missed. Let me walk through them from top to bottom.

L1: Query result cache

Exact string match on the query. "golang channels" maps directly to its top-k results. 62% hit rate is achievable if your query distribution is long-tail (a few popular queries, many rare ones). TTL of 5 minutes. Size: 512 MB is plenty for 1M queries/month because the working set of distinct queries is small.

The non-obvious part: negative caching. 15 to 20% of queries return zero results. Cache those too, with a longer TTL (30 minutes). A zero-result query that misses the cache does the full fan-out, scores nothing, and returns empty. That's wasted work. Cache the empty result and short-circuit next time.

L2: Posting list cache

Term to decoded posting list. This is where compression pays off twice. A VByte-compressed posting list for "golang" (df=4.2M) is about 6 MB. You can fit 600 of those in 4 GB of LRU cache. The top 600 terms by frequency cover about 40% of queries. The other 60% go to L3.

L3: Document cache

Doc ID to stored fields and term vectors. You need this for snippet generation. Without it, every result requires a random read from the document store. 8 GB covers the most-accessed 200K documents. At 71% hit rate, only 29% of snippet requests hit the document store.

L4: OS page cache

This is the one people forget. Your index files are mmap'd. The kernel keeps hot pages in RAM. You don't manage this cache. The kernel does. But you need to know it's there, because it means your effective RAM is your process RSS plus the page cache. A 16 GB page cache covering your hottest index segments is worth more than any application-level cache you can build.

L5: SSD read

The floor. A 4K random read from a good NVMe SSD is about 120 microseconds. That's your worst case for a single posting list block that isn't in any cache. If you're hitting L5 more than 15% of the time, your cache layers are misconfigured or your index is too big for your RAM. Add RAM or shard more aggressively.

Cache stampede prevention

When a popular cache entry expires, 50 concurrent requests all miss the cache and all fan out to the shards. This is a cache stampede. The fix is singleflight: the first miss computes the result, and all 50 concurrent requests wait for that one computation.

import "golang.org/x/sync/singleflight"

var group singleflight.Group

func cachedQuery(query string) ([]ScoredDoc, error) {
    if hit, ok := resultCache.Get(query); ok {
        return hit, nil
    }
    // singleflight: only one goroutine fans out, the rest wait
    v, err, _ := group.Do(query, func() (interface{}, error) {
        results := fanOutAndScore(query)
        resultCache.Set(query, results, 5*time.Minute)
        return results, nil
    })
    if err != nil {
        return nil, err
    }
    return v.([]ScoredDoc), nil
}

This one pattern reduces peak shard load by 60% on stampede-heavy workloads. It is ten lines of code. It is the highest-ROI thing in this entire post.

The hot path: Go code that runs in 8ms

Here is a simplified but real coordinator. The actual production version has more error handling, tracing, and fallback logic, but the shape is the same.

type Coordinator struct {
    shards      []*ShardClient
    cache       *ResultCache
    singleflight singleflight.Group
    pool        QueryPool  // sync.Pool for query objects
}

func (c *Coordinator) Search(ctx context.Context, rawQuery string, k int) (*SearchResponse, error) {
    // 0. check result cache (L1)
    if cached, ok := c.cache.Get(rawQuery); ok {
        return cached, nil
    }

    // 1. singleflight: dedupe concurrent identical queries
    v, err, _ := c.singleflight.Do(rawQuery, func() (interface{}, error) {
        // 2. parse and rewrite the query
        query := c.pool.Get()
        defer c.pool.Put(query)
        query.Original = rawQuery
        query.Terms = tokenize(rawQuery)
        query.Terms = rewriteTerms(query.Terms)  // stemming, synonyms, spell correction
        if len(query.Terms) == 0 {
            // negative cache: store empty result for 30 minutes
            empty := &SearchResponse{Results: []ScoredDoc{}}
            c.cache.Set(rawQuery, empty, 30*time.Minute)
            return empty, nil
        }

        // 3. fan out to all shards with per-shard timeout
        shardCtx, cancel := context.WithTimeout(ctx, 6*time.Millisecond)
        defer cancel()
        results := c.fanOut(shardCtx, query, k*5)  // over-fetch 5x for better global top-k

        // 4. merge per-shard top-k into global top-k
        merged := mergeTopK(results, k)

        // 5. build snippets (only for the final k results)
        response := &SearchResponse{
            Results: c.buildSnippets(shardCtx, merged, query),
            Took:    time.Since(query.Started),
        }

        // 6. cache the result
        c.cache.Set(rawQuery, response, 5*time.Minute)
        return response, nil
    })

    if err != nil {
        return nil, err
    }
    return v.(*SearchResponse), nil
}

func (c *Coordinator) fanOut(ctx context.Context, query *Query, k int) [][]ScoredDoc {
    type shardResult struct {
        idx     int
        results []ScoredDoc
    }
    results := make([][]ScoredDoc, len(c.shards))
    ch := make(chan shardResult, len(c.shards))

    for i, shard := range c.shards {
        go func(idx int, s *ShardClient) {
            docs, err := s.Query(ctx, query, k)
            if err == nil {
                ch <- shardResult{idx, docs}
            } else {
                ch <- shardResult{idx, nil}  // degraded: empty results from failed shard
            }
        }(i, shard)
    }

    received := 0
    for {
        select {
        case r := <-ch:
            results[r.idx] = r.results
            received++
            if received == len(c.shards) {
                return results
            }
        case <-ctx.Done():
            // timeout: return whatever we have so far
            return results
        }
    }
}

Three things in this code that matter more than they look:

The context.WithTimeout of 6ms. You budget 8ms total. You give the shards 6ms. If a shard does not respond in 6ms, you return partial results. Partial results are better than slow results. A search engine that returns 8 of 10 shards' results in 7ms beats one that returns all 10 in 15ms. Users do not notice the missing 2 shards. They notice the 15ms.

The sync.Pool for query objects. At 5 q/s peak, you allocate 5 query objects per second. That's nothing. At 50 q/s burst, you allocate 50 per second. Each query object has a terms slice, a cursor slice, a heap. Without pooling, that's 50 * 2KB = 100KB/s of garbage. The GC pressure is not from the allocation size. It's from the scan time. Go's GC has to scan every pointer in every allocated object. Pooling eliminates the allocation entirely.

The over-fetch of 5x. Each shard returns top-100 (5x the final k=20). This is necessary because per-shard top-k is not the same as global top-k. A document that is rank 50 on shard 3 might be rank 5 globally if shards 0, 1, and 2 have weak results for this query. Over-fetching 5x gives the merge enough headroom to produce a correct global top-k. 5x is empirical. 3x misses too many. 10x is wasteful. 5x is the sweet spot.

Per-shard vs global top-k: the correctness tax

If each shard returns its top-20, and you merge the 10 shard top-20s into a global top-20, you will get the wrong answer about 12% of the time. The document at rank 21 on shard 5 might have a global score higher than the document at rank 20 on shard 2. You'll never see it because shard 5 only sent its top-20.

Over-fetching 5x (top-100 per shard) reduces this error to under 1%. For most search applications, 1% error is fine. If you need exact top-k, you need to either:

Send all matching documents from every shard (defeats the purpose of sharding)
Use a two-phase protocol: phase 1 gets candidate scores, phase 2 re-fetches full scores for the top candidates

Two-phase adds a round trip. It costs 2 to 3ms. Use it only if you need exact results. Most search applications don't. Users don't notice if rank 17 should have been rank 15.

Storage: mmap, page cache, and the SSD reality

Your index lives on disk. You mmap it. The kernel maps file pages into your process address space. Accessing a page that's in RAM is a memory read. Accessing a page that's not in RAM triggers a page fault, the kernel reads the page from SSD, and you pay 120 microseconds.

func openIndex(path string) (*Index, error) {
    f, err := os.Open(path)
    if err != nil {
        return nil, err
    }
    stat, _ := f.Stat()
    data, err := syscall.Mmap(int(f.Fd()), 0, int(stat.Size()),
        syscall.PROT_READ, syscall.MAP_SHARED)
    if err != nil {
        return nil, err
    }
    return &Index{data: data, size: int(stat.Size())}, nil
}

The beauty of mmap is that the kernel manages the cache. You don't evict pages. You don't track LRU. The kernel does it based on actual access patterns. Your working set of hot posting lists stays in RAM. Cold posting lists get evicted under memory pressure and re-read on demand.

The danger of mmap is that page faults are synchronous and unpredictable. Your p99.9 is dominated by page faults. You can be at 2ms p99 and suddenly spike to 30ms because the kernel decided to evict the page you needed. There are three mitigations:

madvise(MADV_WILLNEED) on hot posting lists. Tell the kernel to prefetch them.
madvise(MADV_SEQUENTIAL) on posting lists you are scanning. The kernel increases read-ahead.
mlock on your hottest segments. Pin them in RAM. They never get evicted. This costs you RAM you can't use for anything else, but it eliminates page faults for those segments.

The best strategy is tiered: mlock the top 5% of segments by access frequency, madvise(WILLNEED) the next 20%, and let the kernel manage the rest. You get predictable latency for the hot path and graceful degradation for the long tail.

The SSD read amplification problem

A single term lookup in a compressed posting list with block skipping might touch 3 to 5 separate 4K pages. With 4 terms in the query, that's 12 to 20 page reads. If all of them miss the page cache, that's 20 * 120us = 2.4ms. Just in I/O. Your 8ms budget is gone.

The fix is index layout. Store posting lists contiguously, sorted by access frequency. The hottest terms at the start of the file. The kernel's read-ahead works on sequential access. If your hottest posting lists are contiguous, one read-ahead pulls multiple lists into the page cache.

The second fix is segment tiering. Your hottest documents go in a small "hot" segment that is always RAM-resident. Your cold documents go in large "cold" segments on SSD. Most queries hit the hot segment. The cold segments are for the long tail.

Real-time indexing: the tiered merge problem

You need to add new documents without rebuilding the entire index. The solution is segments. Each segment is an immutable, independently searchable index. New documents go into a new in-memory segment. When the in-memory segment fills up, it flushes to disk as a new immutable segment.

The problem is that more segments means slower queries. Each segment is a separate index lookup. 50 segments means 50 posting list lookups per term. You need to merge segments.

Tiered merging (what Lucene does): segments are grouped into tiers by size. Tier 0 is in-memory (1 to 2 MB). Tier 1 is small disk segments (8 to 16 MB). Tier 2 is medium (128 MB). Tier 3 is large (1 GB+). When a tier accumulates 4 segments, they merge into one segment in the next tier. The merge is a background operation. It reads 4 segments and writes 1. The old segments are deleted after the merge.

The key parameter is merge threshold. Merge too often and you waste I/O on constant merging. Merge too rarely and your query latency degrades from too many segments. 4 segments per tier is the Lucene default and it works. I have tried 8 and 16. 4 is better. The merge cost grows linearly with segment count, but query latency grows logarithmically. You want to merge early enough that query latency stays flat.

The NRT (near-real-time) commit tradeoff

How often do you commit new documents to the index? Every commit creates a new segment. Every new segment slows queries.

Commit every 50ms: 20 new segments per second. Your query latency doubles in 5 seconds. Merge can't keep up.
Commit every 5 seconds: 12 new segments per minute. Merge keeps up. New documents appear in search results within 5 seconds.
Commit every 60 seconds: 1 new segment per minute. Merge is trivial. New documents take a minute to appear.

The answer is NRT with a refresh interval of 1 to 5 seconds. Documents go into an in-memory buffer. Every 1 to 5 seconds, the buffer is flushed as a new segment. The segment is searchable immediately. Background merge cleans up. This is what Elasticsearch does with refresh_interval. The default is 1 second. I set it to 5 seconds for most workloads. The extra 4 seconds of indexing latency is invisible to users. The reduced segment count is very visible in your query latency.

The non-obvious stuff that actually kills you

1. Transparent Huge Pages will wreck your tail latency

Linux THP coalesces 4K pages into 2MB huge pages. Good for throughput. Terrible for latency. When the kernel needs to compact memory to form a huge page, it stalls your process. For 10 to 100 milliseconds. On a random CPU core. During a random query. This is your p99.99 spike. Disable THP on every search node:

echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

I have seen this single change drop p99.9 from 45ms to 12ms. The kernel documentation says THP improves performance. For search workloads with mmap'd indexes, it does not. Disable it.

2. Go GC tuning: the GOGC and GOMEMLIMIT dance

Go's default GC triggers when the heap doubles. At 4 GB heap, GC runs when you hit 8 GB. The GC pause is proportional to the live heap. At 4 GB live heap, a GC pause is 5 to 15ms. That's your p99.

Two levers:

GOGC=200          # trigger GC at 3x heap, not 2x. fewer GCs, more memory.
GOMEMLIMIT=6GiB   # hard cap. GC runs before you hit the limit. prevents OOM.

With GOGC=200 and GOMEMLIMIT=6GiB, you GC less often, use more RAM, and your p99 GC pause drops from 12ms to 4ms. The tradeoff is memory. You need the RAM headroom. On a 16 GB machine with an 8 GB index, you have room.

The deeper fix is to not allocate on the hot path at all. sync.Pool for query objects. Pre-allocated slices for results. Reuse buffers. If your hot path allocates zero bytes per query, the GC has nothing to scan, and GC pauses become irrelevant.

3. NUMA: your 2-socket server is two computers

A 32-core machine with 2 NUMA nodes is two 16-core computers glued together. Memory on node 0 is fast for cores on node 0 and slow for cores on node 1. A cross-node memory access adds 100 to 200ns. On a 2ms query, that's 5 to 10% of your budget.

For search, this means: pin your coordinator goroutines to one NUMA node, and make sure the mmap'd index pages they access are on the same node. Use numactl --cpunodebind=0 --membind=0 to bind your process to node 0. If you need both nodes, run two processes, one per node, each with half the shards.

4. io_uring for batched disk reads

If you are still doing pread() for posting list reads, you are paying a syscall per read. At 20 page reads per query and 5 q/s, that's 100 syscalls per second. Each syscall is 1 to 2 microseconds. That's 200us per second. Negligible.

At 50 q/s burst with 20 reads per query, that's 1000 syscalls per second. Now you're at 2ms of syscall overhead. Not negligible.

io_uring batches multiple reads into one syscall. You submit 20 read requests to a ring buffer, call io_uring_enter once, and wait for the completions. One syscall instead of 20. At burst, this saves 1.5ms per query. That's 18% of your budget.

5. Bloom filters for remote shard existence checks

If you have 100 shards and a 2-term query, you fan out to all 100 shards. But 80 of them don't have either term. They return empty results in 0.5ms. That 0.5ms is wasted. You can't afford 0.5ms * 100 shards of wasted work.

Put a Bloom filter on each shard. The Bloom filter tells you which terms exist on that shard. It's 1 MB per shard for a 1M-document shard with 100K terms. The coordinator holds all 100 Bloom filters in memory (100 MB). Before fanning out, it checks: does shard 47 have "golang" and "channels"? If not, skip shard 47.

For a 2-term query against 100 shards, this typically cuts fan-out from 100 to 15 to 25 shards. The 75 to 85 shards you skipped were going to return empty anyway. You saved 0.5ms * 75 = 37ms of aggregate work and 0.5ms of wall time (the slowest empty shard).

6. Query rewriting at the coordinator

Before you fan out, rewrite the query. This is free latency reduction:

func rewriteQuery(terms []string) []string {
    // 1. lowercase + stem
    for i, t := range terms {
        terms[i] = stem(strings.ToLower(t))
    }
    // 2. synonym expansion: "js" -> "javascript"
    if syns, ok := synonyms[terms[0]]; ok {
        terms = append(terms, syns...)
    }
    // 3. drop stop words for multi-term queries (keep for single-term)
    if len(terms) > 1 {
        terms = removeStopWords(terms)
    }
    // 4. spell correction: if a term has df < 10, try fuzzy match
    for i, t := range terms {
        if docFreq(t) < 10 {
            if corrected := spellCorrect(t); corrected != "" {
                terms[i] = corrected
            }
        }
    }
    return terms
}

Stemming alone reduces your posting list count by 20 to 30%. "running", "runs", "ran" all map to "run". One posting list instead of three. Synonym expansion catches "js" when the user meant "javascript". Spell correction catches "golangg". Each rewrite saves a fan-out, a posting list read, and a scoring pass. At 5 q/s, that's 5 saved fan-outs per second.

7. The slow query log: your best debugging tool

Log every query that takes more than 20ms. Include the query string, the number of matching documents per shard, the number of documents scored, and the cache hit/miss status.

SLOW 23.4ms q="the best golang web framework" shards=10 matches=[2.1M 1.8M 3.2M 990K 1.5M 2.3M 1.1M 890K 1.9M 2.0M] scored=487200 cache=miss

23.4ms. "the" matches 2.1M documents on shard 0 alone. The query matched 16M documents total. WAND scored 487K of them. 487K is too many. The block-max threshold is not aggressive enough because "the" has a low IDF and contributes almost nothing to the score, but its block-max-score is non-zero, so WAND cannot skip its blocks.

The fix: for stop-word-like terms with very low IDF, set their block-max-score to zero. WAND will skip all of their blocks entirely. The query becomes a 3-term query instead of 4-term. Scored documents drops from 487K to 12K. Query time drops from 23ms to 3ms.

You would never find this without a slow query log. Instrument everything.

Capacity planning: the actual numbers

For 1M queries/month against a 10M document corpus:

Component	Spec	Cost (monthly)
2x Coordinator (Go)	4 vCPU, 8 GB RAM	$80
10x Shard nodes	4 vCPU, 16 GB RAM, 200 GB NVMe	$500
1x Index builder	8 vCPU, 32 GB RAM	$120
Network	10 Gbps internal	included
Total		~$700/month

That's $700/month for 1M queries with a 10M document index, 8ms p99.9, and room to grow to 5M queries/month before you need to add shards.

The bottleneck is not CPU. It is RAM. Your index needs to mostly fit in RAM (page cache + mlocked segments). At 10M documents with an average of 500 terms each, your raw posting lists are about 40 GB compressed. You need 16 GB of page cache per shard node to keep the hot 60% in RAM. The cold 40% goes to SSD. That's acceptable because cold terms are rarely queried.

If your index does not mostly fit in RAM, you are in the SSD read amplification regime, and your p99.9 will be 30 to 50ms regardless of how good your algorithms are. Buy RAM before you buy CPU.

Build vs buy: the honest answer

Elasticsearch handles 1M queries/month out of the box. So does Meilisearch, Typesense, and OpenSearch. They work. They are well-tested. They have communities. If you need search tomorrow and your ranking requirements are standard, use one of them.

Build your own when:

You need sub-10ms p99.9 and the off-the-shelf solutions give you 20 to 50ms
You need tight control over memory (embedded, edge, constrained environments)
You need custom ranking that the plugin systems cannot express
You need to run on $700/month and Elasticsearch on the same workload costs $2000+
You want to understand your system

I have built search infrastructure three times. Each time, the first version took 3 months and was worse than Elasticsearch. The second version took 6 months and was equal. The third version, with the lessons from the first two, took 4 months and was 3x faster at half the cost. If you are going to build, commit to building it twice. The first version is the tuition.

Key takeaways

1M queries/month is 0.385 qps average and 5 q/s peak. Build for the peak, the tail, and the burst. Not the average.
Measure p99.9, not p99. At 8 queries per session, p99 hits 1 in 13 sessions.
Posting list compression and block skipping are 80% of performance. Get these right before anything else.
WAND with block-max skipping reduces scoring work by 95 to 99%. Score 0.5 to 2% of matching documents, not all of them.
Five cache layers. Negative cache zero-result queries. Use singleflight for stampede prevention.
mmap your index. Let the kernel manage the page cache. mlock the hot 5%.
Disable THP. It is the single biggest tail-latency win.
GOGC=200 and GOMEMLIMIT to trade RAM for GC pause time. Or don't allocate on the hot path at all.
Fan out to all shards. Over-fetch 5x. Merge. Accept 1% top-k error. Partial results beat slow results.
Bloom filters on shards to skip fan-out to shards that don't have your terms.
Instrument slow queries. The slow query log is how you find the 10x optimizations.
Buy RAM before CPU. Your index needs to mostly fit in RAM or you lose.

#search#infrastructure#inverted-index#go#performance#scaling#postings#bm25#wand

Frequently Asked Questions

How many queries per second is 1 million queries per month?

About 0.38 queries per second on average. But average is meaningless. Real traffic follows a diurnal curve with 3 to 5x peaks. You need to handle 2 to 5 q/s sustained and 10x burst spikes. The infrastructure you build for 1M/month is the same infrastructure you build for 10M/month, because the hard part is the tail, not the average.

What is an inverted index and why does it matter for search?

An inverted index maps each term to the list of documents that contain it, called a posting list. When you search for two words, the engine intersects two posting lists. The efficiency of that intersection, and the compression of those lists, is 90% of search performance. Everything else is secondary.

What is the WAND algorithm in search engines?

WAND (Weak AND) is an early-termination algorithm for top-k retrieval. Instead of scoring every document that matches the query, it uses upper-bound estimates from block-max scores to skip posting list blocks that cannot produce a result in the top-k. On typical queries it reduces scoring work by 80 to 95% compared to full evaluation.

Should I build my own search engine or use Elasticsearch?

If you need generic text search with minimal engineering, use Elasticsearch or Meilisearch. If you need tight control over latency, memory, ranking, or cost at scale, build on top of a library like Lucene or Bleve, or build your own. The post explains when each makes sense and what you give up either way.

Why is p99 latency misleading for search infrastructure?

Because search traffic is bursty and user sessions fire multiple queries in rapid succession. A p99 of 12ms sounds fine until you realize one user session hits 8 queries, and the chance of at least one hitting the p99 is 7.7%. p99.9 is where the real experience lives. At 1M queries/month, p99.9 means 1,000 slow queries, and those are the ones users remember.

Keep reading

Found this useful? Cite it as: webscraping.space. “Building search infrastructure for 1M+ queries a month: the stuff nobody tells you.” https://webscraping.space/blog/building-search-infrastructure-1m-queries. Published 2026-07-13.

Building search infrastructure for 1M+ queries a month: the stuff nobody tells you

Frequently Asked Questions

Keep reading

Scraping at scale: queues, caching, and not getting banned

Web scraping ethics and robots.txt: the lines you don't cross

Bypassing anti-bot protections: TLS, fingerprints, and Cloudflare