URL: https://webscraping.spaceBest viewed at 1024×768

Scaling Published Jul 12, 2026 · 42 min read · 9,295 words

Scraping at scale: queues, caching, and not getting banned

How to take a working scraper to millions of pages without melting the target or getting banned: work queues, bounded concurrency, on-disk caching, dedup, retries with backoff, and polite scheduling.

A scraper that works on ten pages is a script. A scraper that works on ten million pages is a system. The difference isn't parsing. It's scheduling, caching, and politeness. This guide is the architecture between "it ran once" and "it runs every night without getting banned." I've built crawls at both ends of that spectrum, and I've watched the same failure happen in the same place more times than I can count: the script that hummed along on a laptop quietly falling apart at a few hundred thousand pages. The causes are never exotic. They're unbounded queues, uncontrolled concurrency, no retry policy, a dedup set that resets on restart, and no idea which URLs have already been fetched. The fixes aren't clever either. Queues, bounded concurrency, disk caching, exponential backoff with jitter, URL normalization, and monitoring. Each one is boring. Together they're the difference between a scraper and a system.

The four problems of scale

When you take a scraper from a few pages to a few million, you stop solving parsing problems and start solving systems problems. Everything you need to build fits into four buckets. I keep this list pinned to every crawl I work on, because every outage I've debugged traces back to one of them.

Throughput without melting the target. The site is not a resource you own. If you're doing fifty requests per second to a server that normally sees five, you're not scaling, you're breaking the thing you're trying to read. The hard part of throughput isn't making requests faster. It's finding the sustainable rate: the rate where responses come back quickly, the site stays healthy, and nobody's rate limiter wakes up.
Reliability. One transient 500 or one dropped connection shouldn't kill a run. At ten pages you restart. At a million, "restart" means losing hours of work and, worse, re-requesting a million URLs the server already answered. Reliability means deciding up front which failures are retryable, how hard you try, and what you checkpoint, so a crash is an inconvenience rather than a catastrophe.
Politeness. You're a guest on someone else's server. The difference between a polite crawl and an abusive one is scheduling: per-domain rate limits, crawl delays, honoring Retry-After, caching so you never ask for the same bytes twice. Politeness is also the cheapest anti-ban strategy that exists. Bans are overwhelmingly a response to behavior — too fast, too repetitive, no identity — and a polite crawler sidesteps them before they start.
Dedup and resumability. Never request the same URL twice. Never lose progress. These sound like two rules; in practice they're one system, because both reduce to the same thing: you need persistent state that records what you've seen and what's left to do.

The rest of this guide is those four buckets, in the order you should build them.

The migration timeline: script to system

The right queue is the smallest one that won't fall over. Most teams don't need Kafka. They need to know when their current setup stops being honest. Here's the timeline I've seen over and over, along with the trigger that says it's time to move. The chart below shows roughly how many pages each architecture can realistically clear in a single twelve-hour overnight run on a handful of domains — the crawl that runs while you sleep.

Estimated overnight throughput for each architecture at a polite per-domain rate. The naive loop is honest for one run; the full system is what actually makes "every night" a habit.

Phase 0 — the script. A for loop over a list. Fine to maybe a few thousand pages. It dies on the first 500, forgets its place on the first crash, and has no concept of a domain that is faster or slower than its neighbors. The trigger to leave: you write it, run it, and realize you're afraid to run it again because you don't actually know what it did.

Phase 1 — in-process queue with threads. A queue.Queue plus a worker pool, which is the first code example below. Good to maybe a few hundred thousand pages across a handful of domains. All state lives in memory, so a crash loses the frontier and the seen set together. The trigger to leave: a worker died at 2 a.m. and you re-fetched half a million URLs to rebuild the seen set, or the crawl simply no longer fits in one process's memory.

Phase 2 — Redis or RabbitMQ. The queue moves outside your process. Workers can be killed and restarted without losing the frontier, and you add capacity by adding processes or machines rather than rewriting code. Redis with BRPOPLPUSH gives you a reliable queue; RabbitMQ gives you acknowledgements, priorities, and dead-letter handling when you need delivery guarantees. The trigger to leave: you need more than one machine, or you need the crawl to survive a worker crash without losing work.

Phase 3 — database-backed frontier. When the crawl is huge, long-running, and needs to resume cleanly after anything, the seen set and the queue live in Postgres or SQLite, and every claim on a URL is a transaction. This is where "runs every night" actually lives. Most teams land here and stay, because it degrades gracefully: if the crawl dies at 2 a.m., tomorrow's run picks up the queue table where it left off.

You can also skip straight to a managed crawler API and let someone else run the queue, proxies, and browsers. The build-versus-buy math is real, and I'll come back to it in the Scrapy section.

Use a work queue

Never crawl with a bare for url in urls: loop. Put URLs in a queue and pull from it. A queue gives you three things the loop can't, and each one matters more as the crawl grows. First, resumability: the queue is where "what's left to do" lives, so a crash doesn't mean starting over. Second, bounded concurrency: you pull exactly as many items as you have workers, no more, no fewer. Third, a home for newly discovered URLs: when a page links to a hundred others, you don't recursively block on them, you enqueue them and let the workers get to them when there's capacity.

The in-process version looks like this. It's the Phase 1 workhorse, and it's worth writing once even if you migrate to Redis later, because the consumer loop — claim a URL, check it's unseen, fetch, parse, enqueue what you found, mark done — is the same shape in every phase:

import queue
import threading

WORK = queue.Queue()
SEEN = set()
SEEN_LOCK = threading.Lock()

for url in seed_urls:                      # your starting points
    WORK.put(url)

def worker(worker_id):
    while True:
        url = WORK.get()
        try:
            with SEEN_LOCK:
                if url in SEEN:
                    continue              # already claimed by another worker
                SEEN.add(url)
            body = fetch_with_retry(url)  # your HTTP layer (retries, cache)
            for new_url in extract_links(url, body):
                WORK.put(new_url)
        except Exception as exc:
            log.error("worker=%s url=%s error=%s", worker_id, url, exc)
        finally:
            WORK.task_done()

threads = [threading.Thread(target=worker, args=(i,), daemon=True)
           for i in range(8)]
for t in threads:
    t.start()
WORK.join()

The three things to copy from this example into whatever you build next: the seen set is checked and mutated under a lock, so two threads can't claim the same URL; a worker that throws still marks its task done, so the queue can't deadlock on an exception; and failures are logged with the URL, because a log line without a URL is useless for debugging a crawl. The daemon=True threads also mean the process can be killed without a hang, which is exactly what you want when the 2 a.m. restart is a Ctrl-C followed by a systemd unit.

One subtlety that bites everyone eventually: this queue is at-least-once, not exactly-once. If a worker fetches a URL, parses it, and crashes before calling task_done, the URL is gone from the in-flight state and will never be retried. If instead a worker crashes after adding a URL to SEEN but before fetching it, that URL is skipped forever. Neither of these is catastrophic if fetches are idempotent and you tolerate occasional gaps, which is why at-least-once is the right default for scraping. You can tighten it in Phase 3 by making the claim itself transactional, which I'll cover in the resumability section.

Bounded concurrency

More concurrency is not more throughput. Past a point it's just more 429s and more load on a server that didn't ask to be crawled. I've measured this on real targets and the shape of the curve is remarkably consistent: throughput climbs as you add connections, peaks, then falls as the server's own limits start answering you with errors. Here's a representative run against one mid-sized e-commerce site, 300 milliseconds round-trip, one domain:

Goodput peaks around four concurrent connections and then collapses as the server starts returning 429s; the 429 share rises steeply past the peak. The polite ceiling is real and it is lower than you think.

Three things are going on here, and only one of them is about your code. First, there's the server's worker pool: it has a fixed number of slots, and requests beyond that sit in its accept queue. As concurrency climbs, so does queueing delay, which makes round-trips balloon and makes your "timeout after ten seconds" fire on pages that would have loaded in two. Second, there's your own connection pool: a single keep-alive connection can only sustain one in-flight request at a time, so at 300 milliseconds of round-trip one connection tops out at roughly 3.3 requests per second no matter how fast you code. Third, there's the rate limiter, which the site's owners tuned to their own traffic and which starts returning 429s once you cross it. All three together produce the peak-and-collapse curve above.

The two rules I actually follow:

Cap concurrent connections per domain, not just globally. A global cap of twenty is useless if one slow domain holds fifteen of the slots while a fast one starves. Give each domain its own cap of 2 to 5, and let the pool size be the sum of the domain caps.
Cap requests per second per domain, with jitter. A steady 1 request per second per domain is a tireless superhuman; 3 per second is an entire warehouse of clickers. Rate limits that are exact multiples of your intended rate are also a fingerprint — add jitter so your inter-request gaps are irregular, like a person's.

The "one fast human" ceiling is the politeness number I design against: a fast human skimming a page takes two to five seconds per page and doesn't do it for eight hours straight. If your crawler runs at 1 request per second per domain, you're already faster than that human over the long haul. Sites that run any kind of bot defense build their traffic models from aggregate behavior — requests per second per IP, the ratio of HTML to assets, the shape of the request timeline — and the polite crawler sits comfortably inside those models.

Getting more throughput per connection without more concurrency is also a real lever: reuse connections (keep-alive is the default in both aiohttp and requests.Session), and use HTTP/2 multiplexing when the server supports it, which lets one connection carry several concurrent streams. That is genuinely more throughput with fewer connections, and it's the rare scale technique that makes the target's life easier, not harder.

A per-domain token bucket is the right rate limiter because it naturally handles both the steady state and short bursts, and because it degrades per domain instead of globally:

import asyncio
import time
from collections import defaultdict
from urllib.parse import urlsplit

class DomainLimiter:
    """One token bucket per domain. rate = requests/sec, burst = max burst size."""
    def __init__(self, rate=1.0, burst=3):
        self.rate = rate
        self.burst = burst
        self._tokens = defaultdict(float)
        self._last = defaultdict(float)

    async def wait(self, domain):
        now = time.monotonic()
        tokens = min(self.burst,
                     self._tokens[domain] + (now - self._last[domain]) * self.rate)
        if tokens < 1:
            await asyncio.sleep((1 - tokens) / self.rate)
            now = time.monotonic()
            tokens = 1
        self._tokens[domain] = tokens - 1
        self._last[domain] = now

limiter = DomainLimiter(rate=1.0, burst=3)   # 1 req/s per domain, burst of 3

async def fetch(session, url):
    domain = urlsplit(url).netloc
    await limiter.wait(domain)               # blocks only this domain
    async with session.get(url) as resp:
        return resp.status, await resp.text()

The key property: when one domain is slow or starts throttling, wait blocks only that domain. The other domains keep their own budgets, so a single misbehaving site doesn't stall the whole crawl. That separation is the entire difference between a limiter that protects the site and a limiter that just makes your crawl slow.

Cache responses to disk

Caching is the highest-ROI thing you can do to a scraper, and it does three jobs at once: it makes you faster, cheaper, and more polite. Once you've fetched a page, you should never fetch it again. This matters more than it sounds. During development you'll run your parser fifty times against the same fifty pages; without a cache that's fifty trips to the server for bytes you already have. After a parser bug you'll rerun extraction over the whole corpus; with a cache that's an afternoon of local parsing, without a cache it's a week of re-crawling and a very annoyed target.

The cache hit rate over a crawl's life tells you where the ROI actually is. It depends almost entirely on the site's topology — whether the crawl keeps revisiting pages or goes through each one once:

A list-heavy site, where the crawl revisits the same pagination and category pages on every pass, can sustain a 40 percent-plus cache hit rate. An article-heavy crawl that visits each page once barely benefits on the first pass — the payoff comes on the next run.

Three storage tiers cover almost every crawl:

File-per-URL is the simplest and is perfect during development. One JSON file per page, keyed by a hash of the normalized URL, in a directory tree. It breaks down at scale for an unglamorous reason: filesystems get slow with millions of files in one directory, and metadata operations (open, stat, rename) dominate. Shard by two characters of the hash so you have at most 256 files per directory, and it stays healthy well past a million pages.
SQLite is the workhorse for crawls up to tens of millions of entries. A single file, transactions, fast indexed lookups on the URL hash, and you can store the body in one column and fetch metadata with the same query. It's also trivially resumable: the cache is the state.
An object store (S3, GCS, or a local MinIO) is for very large crawls or multi-machine crawls that need a shared cache. Store the gzipped body as the object, put the metadata in a small database. The network round-trip makes it slower than local disk, but it scales horizontally forever.

A production cache needs two things beyond storage: a TTL and a way to invalidate. The TTL should vary by content type — category pages go stale in hours, product pages in days, long-form articles in weeks. The classic trick is to store the fetch timestamp in the metadata and treat any hit older than the TTL as a miss, deleting the file lazily so the next fetch rewrites it. Revalidation is the upgrade: if the target sends ETag or Last-Modified, you can send If-None-Match or If-Modified-Since and let the server answer 304 with an empty body, refreshing your TTL without transferring bytes. That's the cheapest possible refresh and it's worth wiring in once your crawl is stable.

Invalidation is the part people forget until it bites them: when you change your parser, your cache must be treated as poisoned, or your "new" extraction silently runs on old HTML. Bump a cache namespace version in the key (a constant you increment when the parser changes) so a parser rewrite starts a fresh cache instead of reusing stale bodies. Here's a file-backed cache with a TTL and a namespace that covers all of it:

import hashlib
import json
import pathlib
import time

CACHE_VERSION = 2                      # bump this when the parser changes
CACHE = pathlib.Path(f"cache_v{CACHE_VERSION}")
CACHE.mkdir(exist_ok=True)

def _path(url):
    digest = hashlib.sha256(url.encode()).hexdigest()
    return CACHE / digest[:2] / f"{digest}.json"

def cache_get(url, ttl=86400):
    p = _path(url)
    if not p.exists():
        return None
    meta = json.loads(p.read_text())
    if time.time() - meta["fetched_at"] > ttl:
        p.unlink()
        return None
    return meta

def cache_put(url, status, body, headers=None):
    p = _path(url)
    p.parent.mkdir(parents=True, exist_ok=True)
    tmp = p.with_suffix(".tmp")
    tmp.write_text(json.dumps({
        "url": url,
        "status": status,
        "headers": headers or {},
        "body": body,
        "fetched_at": time.time(),
    }))
    tmp.rename(p)                       # atomic: a crash can't leave half a file

def cached_fetch(session, url, ttl=86400):
    hit = cache_get(url, ttl)
    if hit is not None:
        return hit["status"], hit["body"]
    status, body = await fetch(session, url)
    cache_put(url, status, body)
    return status, body

Two details here are deliberate. The write goes to a .tmp file and is renamed into place, so a crash mid-write can never leave a truncated JSON file that your next run will parse as a valid-but-broken body. And the cache stores the full headers alongside the body, which turns out to matter: you'll want the Content-Type to pick a parser, and the Last-Modified and ETag to do revalidation later. The sharded subdirectory layout keeps each directory small, which is the difference between a cache that crawls well past a million pages and one that starts thrashing at fifty thousand.

Retries with backoff and jitter

Networks fail. Retry, but retry well. The first decision is what's retryable at all. Getting this wrong is how crawlers waste half their bandwidth: the classic bug is retrying a 404, a 401, or a 403 forever — requests that will fail identically on the hundredth try. The rule of thumb I use: retry only transient failures, where the server's state changed between your request and your retry. A permanent rejection will not become a success by asking again.

Status code	Meaning	Retry?	Notes
200	OK	No	Parse it.
301 / 302	Redirect	Yes	Follow it, but cap the chain at 5 hops.
400	Bad request	No	Your request is malformed; fix the code, not the retry.
401 / 403	Unauthorized / Forbidden	No	Auth, headers, or an access control. Retrying is hostile.
404 / 410	Gone	No	Permanently absent. Record and move on.
408	Request timeout	Yes	Retry with backoff; the connection may just have been slow.
422	Unprocessable	No	Permanent rejection of the payload.
425	Too early	Yes	The server is not ready; back off and retry.
429	Too many requests	Yes	Honor Retry-After, then exponential backoff with jitter.
5xx	Server error	Yes	Transient; back off harder for 502/503/504.

The second decision is how long to wait between attempts. The naive answer — retry immediately — is wrong in two ways. It hammers a server that's already failing, and, worse, when a burst of failures hits all your workers at once, every worker retries at the same instant, and the retry wave lands as one synchronized attack. That's the thundering herd, and it turns a blip into a ban. Jitter breaks the symmetry: each worker adds a random offset, so the retries scatter instead of stacking.

Exponential backoff with full jitter is what I use: the base delay doubles each attempt (1s, 2s, 4s, 8s, capped at 60s), and the actual wait is drawn randomly in that window rather than being exactly the base. The chart below shows the jitter band for each retry attempt in a typical run — the base doubles, and the actual sleep lands anywhere inside the shaded band:

Each retry waits a random duration inside a doubling window, capped at 60 seconds. The dashed line is the base delay; the bars are where an individual retry actually lands. The widening band is what keeps a burst of failures from becoming a synchronized hammering.

The Retry-After header overrides all of this when it's present — if the server tells you to come back in 30 seconds, the right move is to come back in 30 seconds, not 4. A server that sends Retry-After with a 429 is giving you the fastest safe path back to a healthy crawl, and ignoring it is how you get from a warning to a block. When the header is absent, fall back to the exponential schedule. Here's the full loop, including the status-code decision table from above:

import asyncio
import random

NEVER_RETRY = {400, 401, 403, 404, 410, 422}
MAX_ATTEMPTS = 5
BACKOFF_CAP = 60.0

def _backoff(attempt):
    return min(BACKOFF_CAP, 2 ** (attempt - 1)) + random.uniform(0, 1.0)

def _retry_after(headers):
    raw = headers.get("Retry-After")
    if raw is None:
        return None
    if raw.isdigit():
        return int(raw)
    return None   # HTTP-date form: parse it, or fall back to backoff

async def fetch_with_retry(session, url):
    for attempt in range(1, MAX_ATTEMPTS + 1):
        try:
            async with session.get(url) as resp:
                if resp.status == 200:
                    return await resp.text()
                if resp.status in NEVER_RETRY:
                    return None
                if resp.status == 429:
                    await asyncio.sleep(_retry_after(resp.headers) or _backoff(attempt))
                    continue
                if resp.status >= 500:
                    await asyncio.sleep(_backoff(attempt))
                    continue
                return None
        except (asyncio.TimeoutError, aiohttp.ClientError):
            await asyncio.sleep(_backoff(attempt))
    return None   # give up quietly; record the URL for a later pass

The last line deserves a comment: when a URL fails all five attempts, you do not throw — that would kill a worker and, with it, everything after it. You return None, record the URL somewhere (a "failed" table, a file, a log line), and move on. A crawl that finishes with a list of a hundred URLs that failed is infinitely more valuable than a crawl that crashes on the hundredth failure. I'll come back to that pattern in the resumability section, because the failed list is exactly what you feed to a slower, more careful retry pass later.

URL normalization and dedup

A crawler that fetches the same URL twice isn't just wasteful, it's the fastest way to look like a broken robot. Yet "the same URL" is a slippery concept. These are all the same page in practice: HTTP://Example.COM/path/, http://example.com:80/path, http://example.com/path?b=2&a=1, http://example.com/path#section, and http://example.com/path/. If you dedup on the raw string, you'll fetch all five, and the fifth one is exactly the kind of redundant traffic that a rate limiter notices. Normalization collapses them into one canonical form before the dedup check ever runs:

from urllib.parse import urlsplit, urlunsplit, parse_qsl, urlencode

def normalize(url):
    s = urlsplit(url)
    scheme = s.scheme.lower()
    netloc = s.netloc.lower()
    if netloc.endswith(":80"):
        netloc = netloc[:-3]
    elif netloc.endswith(":443"):
        netloc = netloc[:-4]
    path = s.path.rstrip("/") or "/"
    query = urlencode(sorted(parse_qsl(s.query, keep_blank_values=True)))
    return urlunsplit((scheme, netloc, path, query, ""))

Four rules, each with a reason. Lowercase the scheme and host, because they're case-insensitive but often typed inconsistently. Drop default ports, because :80 and :443 are noise. Sort the query parameters and keep the blank ones, because ?a=1&b=2 and ?b=2&a=1 are the same request to virtually every server, and a tracking parameter with an empty value is still a parameter. Drop the fragment, because it never reaches the server — it's client-side state.

Two warnings about this function, both learned the hard way. Stripping trailing slashes is wrong for some sites, where /product/ and /product genuinely resolve differently; I've also seen sites where /product is a redirect and /product/ is the canonical form. My rule: apply the trailing-slash strip, but if your target distinguishes them, make it a per-domain flag. And don't over-normalize query parameters. Some parameters are load-bearing (?page=2) and some are tracking junk (?utm_source=newsletter); a useful refinement is a per-domain list of parameters to drop outright, because utm_ and fbclid parameters multiply URLs without adding any content, and each distinct string defeats dedup.

The dedup store itself has four tiers, and the right one depends on crawl size and how much you care about missing a URL:

A Python set is the simplest and the right answer for a few million URLs. It's memory-bound, and each entry costs roughly 55 bytes once you count the string and the hash table overhead. The concurrency hazard is that it's not shared across processes, so if you scale to multiple workers or machines, each one needs its own set — which means duplicate fetches.
A Redis set is the default for any crawl bigger than one process. SISMEMBER and SADD are O(1), they're shared across every worker and every machine, and the set persists, so a restart doesn't lose the seen state. It's the same memory cost as the Python set, just living in a place that's actually reachable.
A Bloom filter trades memory for the risk of false positives: it says "seen" for something it's never seen, with a probability you choose. At a 1 percent false-positive rate it costs about 1.2 bytes per URL, which is roughly 1/45th of a Python set. The price is that 1 percent of URLs never get fetched, silently.
A SQLite or Postgres unique index is the durable, exact option, and it's the one I'd reach for when the seen set is also the resumability state. The claim is a transaction: INSERT OR IGNORE, and the row count tells you whether you were the first to claim it.

The memory math is what surprises people. Here's why the Bloom filter exists:

At 100 million URLs a Python set needs roughly 5.5 GB of RAM; a Bloom filter with a 1 percent false-positive rate fits in 120 MB. The set is exact. The Bloom filter silently skips a fraction of URLs. For a marketplace listings crawl, that fraction is real money.

I'll say it plainly: I don't use Bloom filters for data I care about. A false positive drops a URL with no error, no log line, no trace, and you won't notice until you reconcile your dataset against the site and find listings that never made it in. For high-value crawls I use a Redis set when the crawl is multi-process, and a SQLite unique index when the seen set doubles as the resumability state. Bloom filters are for the one case they were built for: when the set genuinely doesn't fit in the memory you're willing to pay for, and missing a small fraction of URLs is an acceptable cost — a freshness sweep, say, where you re-scan a billion URLs to find the few hundred that changed.

The dedup check and the enqueue are one operation, not two. In the worker loop the claim comes first — if seen: continue; add to seen under the same lock — because the check-and-add has to be atomic or two workers will both pass the check and both fetch the page. With Redis it's the same shape, just atomic by construction:

import redis
r = redis.Redis(decode_responses=True)
SEEN = "crawl:seen"

def claim(url):
    # SADD returns 1 only if the URL was NOT already in the set.
    return r.sadd(SEEN, url) == 1

# in the worker:
if claim(url):
    enqueue_work(url)   # only the winner enqueues, exactly once

That SADD-returns-1 trick is worth remembering: it turns the whole dedup problem into a single atomic operation, which means you never need locks across workers, and it works identically from every process and machine attached to the same Redis.

Be polite: scheduling

All the queues, caches, and retries in this guide are the plumbing. Politeness is the policy that runs on top, and it's the difference between "runs every night" and "ran for three days then got banned." The politeness rules I apply to every crawl:

Set a per-domain crawl delay. One to three seconds between requests to the same domain is a reasonable default; if the site's own robots.txt declares a Crawl-delay, use that instead. This is the single politeness knob that matters most, and it's the one people skip because it feels slow. It is supposed to feel slow. That's the point.
Honor Retry-After and 429s, which the retry section covers. A site telling you to slow down is a gift; treat it as one.
Identify yourself. Set a custom User-Agent that includes a contact URL when you're scraping permitted targets, so an operator who sees your traffic can reach you instead of blocking you. An anonymous default UA reads as a bot; a UA that says "crawl@example.com, here's what I'm doing" reads as a professional.
Respect robots.txt. It's the site's statement about what automated crawlers may access. Scrapy obeys it by default with ROBOTSTXT_OBEY = True, and there's rarely a good reason to turn it off. The legal and ethical line-work around robots.txt, terms of service, and what counts as allowed is worth reading in the ethics and robots.txt guide before you point anything at a site you don't own.
Ramp up, don't start hot. A crawl that opens with 50 requests in the first second announces itself. Start at half your target rate for the first few minutes and let the server get used to you.
Crawl off-peak for big jobs. A 2 a.m. run of a million pages is a different experience for the target than the same run at 2 p.m., and it's cheaper for you too — less contention, faster responses.
If the site offers an API or a data dump, use that instead. Always. No amount of politeness beats not crawling at all.

Scrapy's AutoThrottle implements most of this for you: it learns a per-domain request rate from the target's own response times and 429s, and it backs off automatically when the site starts complaining. It's one of the best reasons to use Scrapy, and I'll come to that tradeoff next.

Scrapy vs roll-your-own

Everything so far — queue, concurrency caps, cache, retries, dedup, politeness — is scaffolding you could build on top of requests or aiohttp. You will, the first time, because that's how you learn it. But Scrapy ships almost all of it as battle-tested defaults, and once you're building a real crawl, the math changes. Here's how I draw the line:

Dimension	Scrapy	Roll-your-own (requests/aiohttp)	Managed API
Queue & scheduler	Built-in scheduler with priorities	You build it (in-process, then Redis, then DB)	Run by the vendor
Concurrency	`CONCURRENT_REQUESTS`, per-domain limits, `AutoThrottle`	You build it: semaphores + token buckets	Run by the vendor
Retries	Built-in backoff, `RETRY_TIMES`	You build it (see the retry example above)	Run by the vendor
Dedup	Built-in fingerprint + seen filter	You build it (normalize + Redis set)	Run by the vendor
Cache	HTTP cache middleware	You build it (file, SQLite, or object store)	Vendor-side cache, per-page billing
robots.txt	`ROBOTSTXT_OBEY = True`	You wire up `urllib.robotparser`	Vendor policy
Item pipelines	Built-in processing stages	You build it (usually just a parser function)	You receive the vendor's output schema
Proxies, browsers, CAPTCHA solving	Third-party middlewares, or none	You build it or buy it separately	Included in the price
Cost to run	Your VMs, no per-page fee	Your VMs, no per-page fee	Per-page fee, often with minimums
Control	Full, open source	Full	Limited to the vendor's features

What Scrapy gives you out of the box is exactly the scaffolding this guide has been building: a scheduler that keeps the frontier and honors priorities, a dedup filter you can swap for a Redis-backed one, retries with configurable backoff, AutoThrottle that tunes per-domain rates from the site's own response times, robots.txt obedience, an HTTP cache, and an item pipeline for post-processing. It also gives you something subtler: the crawl is resumable with scrapy crawl myspider -s JOBDIR=crawl_state, which persists the scheduler and dedup state to disk. That one flag is the resumability section below in a single line. For most teams starting a real crawl, Scrapy is the correct default, and scrapy-playwright adds the browser layer when you need it.

You outgrow Scrapy in a few specific situations, and it's worth knowing them so you recognize the moment rather than fighting the framework. When your scheduling needs are unusual — time-based windows, per-URL priorities that change mid-crawl, a frontier that must be shared and coordinated across machines in a way the stock scheduler doesn't do — the framework's conventions start costing more than they save. When your storage is highly custom, like writing each response to a columnar store with specific lifecycle rules, you end up fighting the pipeline layer. And when you want no infrastructure at all, a managed API beats anything you can run. The build-versus-buy math is the real question, and the answer lives in how much of the stack you actually need. Here's a rough cost picture for one million pages:

Rough list prices for a one-time crawl of a million pages. Self-hosting is nearly free in cash and expensive in engineering; the managed fee buys proxies, browsers, CAPTCHA solving, and no ops. The break-even point depends on your hourly cost and how many domains you touch.

The managed-fee numbers only make sense in the right frame. If a million pages takes you two days of engineering to build and the infrastructure costs $3, your time and the crawler's fragility are the real expenses. If you need proxies (thousands of IPs, rotated), a rendering layer, and CAPTCHA solving, the infrastructure you'd have to assemble costs far more than the per-page fee, and the vendor wins decisively. If you crawl ten domains you control, self-hosting wins on cost and control. The honest advice: build the first crawl yourself so you understand the moving parts, then buy when the moving parts stop being the interesting part of your problem.

Resumability and crash recovery

Everything above is about running well. Resumability is about failing well, because at scale a crawl will fail. Machines reboot, spot instances vanish, bugs crash workers, disks fill. The question isn't whether your crawl will be interrupted; it's how much work a restart costs. The three patterns that matter:

Persist the frontier and the seen set. If your queue and your dedup state live in Redis or a database, a crash loses nothing. If they live in process memory, a crash loses everything, and the recovery itself is expensive: you re-crawl to rediscover what you already had, which is exactly the redundant traffic that gets you noticed.
Make fetches idempotent. The same URL fetched twice should not create two records. You can't always control the target's side, but you can control yours: store each response under its URL's content hash or its normalized URL, so a duplicate fetch overwrites rather than duplicates.
Write atomically. The cache example's write-then-rename pattern applies everywhere. A truncated JSON file, a half-written body, a zero-byte artifact — each one is a landmine your next run will trip over. Atomic writes make "crash mid-write" impossible.

The database-backed frontier is where resumability becomes structural rather than incidental. Each worker's claim on a URL is a transaction, so the state survives any crash, and "resume" is just "start workers and drain the queue table":

import sqlite3
import time

db = sqlite3.connect("frontier.db", check_same_thread=False)
db.execute("CREATE TABLE IF NOT EXISTS seen "
           "(url TEXT PRIMARY KEY, fetched_at REAL, status INT)")
db.execute("CREATE TABLE IF NOT EXISTS queue "
           "(url TEXT PRIMARY KEY)")
db.execute("CREATE TABLE IF NOT EXISTS failed "
           "(url TEXT PRIMARY KEY, attempts INT, last_error TEXT)")

def claim(url):
    # Returns True only if WE were the first to claim this URL.
    cur = db.execute(
        "INSERT OR IGNORE INTO seen (url, fetched_at, status) VALUES (?, ?, 0)",
        (url, time.time()))
    return cur.rowcount == 1

def record_failure(url, attempts, error):
    db.execute(
        "INSERT OR REPLACE INTO failed (url, attempts, last_error) VALUES (?, ?, ?)",
        (url, attempts, error))
    db.commit()

def resume_queue():
    # On startup: re-enqueue everything we never finished.
    return db.execute("SELECT url FROM queue WHERE url NOT IN "
                      "(SELECT url FROM seen)").fetchall()

Notice what this buys beyond crash safety. The failed table is the retry pass I mentioned in the retries section: a nightly job drains failed, tries each URL again with a much gentler rate and a higher attempt cap, and deletes successes. It's the difference between "the crawl lost 2 percent of its pages" and "the crawl quietly completed." And the resume_queue query is the whole crash-recovery story in one line: on startup, whatever is in the queue and not yet in seen is still work to do. A crash at 2 a.m. stops mattering, because tomorrow's run picks up exactly where the queue table says it left off.

There's one failure mode that survives all of this if you're not careful: a worker that's killed mid-response.text() leaves nothing behind (good, it just wasn't claimed), but a fetch that returns a truncated body and succeeds does poison your data. The check is cheap: validate that a cached or stored body ends where it should — a length check against Content-Length when present, a closing </html> check when you're strict, a JSON parse when you're storing JSON. One line in the parser, and a class of silent corruption disappears.

Monitoring a crawl

A crawler is a service now, and services get monitored. The good news is that a crawl's health is unusually legible: every request is an event, every status code is a signal, and the metrics that matter are few and specific. I track five of them, and I act on the trends, not the individual readings:

Throughput — requests per second, per domain and overall. A steady rate means the limiter is working; a rate that drops while the queue stays deep means something upstream is slow.
Error rate by status code. The overall error rate hides the signal. A 404 rate of 2 percent is normal link rot; a 403 rate that climbs from 0 to 5 percent is a ban forming. Track 403s, 429s, 5xx, and timeouts separately.
429 rate and retry volume. Rising 429s are the earliest warning a site is about to block you. The moment the 429 share starts climbing, you back off the affected domains — you don't wait until it hits double digits.
Cache hit rate. This tells you whether your crawl is doing useful new work or re-fetching. A hit rate that drops to zero on a "daily" crawl means your TTLs are too aggressive and you're being needlessly rude.
Ban signals. The classic warning signs, in rough order of appearance: climbing 429s, a sudden jump in 403s, CAPTCHA or challenge pages appearing in your HTML, time-to-first-byte creeping up even on healthy-looking responses, and empty bodies where there used to be content.

Here's a minimal stats collector that gives you the first three on a rolling window, plus the automated backoff trigger that keeps a small problem small:

import time
from collections import deque

class CrawlStats:
    def __init__(self, window=300):
        self.window = window
        self.requests = deque()          # timestamps
        self.non200 = deque()            # (timestamp, status)
        self.status_counts = {}

    def record(self, status):
        now = time.time()
        self.requests.append(now)
        self.status_counts[status] = self.status_counts.get(status, 0) + 1
        if status != 200:
            self.non200.append((now, status))
        self._prune(now)

    def _prune(self, now):
        cutoff = now - self.window
        while self.requests and self.requests[0] < cutoff:
            self.requests.popleft()
        while self.non200 and self.non200[0][0] < cutoff:
            self.non200.popleft()

    def throughput(self):
        return len(self.requests) / self.window

    def rate_for(self, status):
        return sum(1 for ts, s in self.non200 if s == status) / self.window

    def backoff(self, domains):
        """Halve each domain's rate if 429s are climbing."""
        if self.rate_for(429) > 0.1:      # more than 10 429s per minute
            for limiter in domains.values():
                limiter.rate *= 0.5

The backoff function is the important one. A crawl that reacts to the first sign of throttling often never gets throttled at all, because the site's limiter sees you slow down and stops escalating. A crawl that ignores the signs learns, at some point, what an IP block looks like from the inside. I wire the same trigger to a couple of actions: halve the rate, and for the worst offenders, drain their domain's concurrency cap to one for a cooldown period. When the 429 rate falls back under the threshold, I let the rates recover slowly — and if they don't fall back, I stop the crawl and look at what changed, because "permanently throttled at a polite rate" usually means the target changed its defenses and yesterday's behavior is now today's ban.

The other monitoring habit that pays off is a canary: a handful of URLs you fetch every few minutes and assert on — status 200, known content, expected size. The canary tells you the target is reachable and unchanged, independent of whatever the crawl is doing. When the crawl mysteriously starts failing everything, the canary tells you whether it's you or the target, which is the fastest triage question you can ask.

War stories

Everything above is lessons I paid for. Here are the specific ones.

The pagination loop that fetched page one forever. A crawler with a dedup bug: the "next" link on a paginated list resolved differently on every pass, so the seen set never matched, and the crawler re-fetched the first fifty list pages in a tight loop for four hours before anyone noticed. The target saw tens of thousands of requests for the same fifty URLs. The fix was normalization and a canary on the frontier: assert that the URLs being enqueued actually differ from what's already in flight. The lesson wasn't about dedup algorithms. It was that a crawler that runs unattended needs its own invariants checked, because the loop that looks like progress isn't.

The 429 spiral that ended in an IP ban. Sixteen workers, no jitter, no shared retry policy. A brief server hiccup produced a burst of 429s; every worker retried instantly, in lockstep; the synchronized wave produced more 429s; within minutes the site's limiter escalated to a full block of the IP range. The crawl lost a day, the proxy budget blew up, and the data had to be backfilled at one-tenth the speed. The fix was the exact retry loop in this guide — exponential backoff, jitter, honor Retry-After — plus the monitoring trigger that backs off at the first sign of climbing 429s. I've never seen a synchronized 429 spiral since.

The cache that made a parser rewrite free. A two-million-page crawl with a file cache. A parser bug shipped, and the pipeline produced garbage for a day. The fix was a parser change plus a cache-namespace bump. Because every raw response was on disk, the re-extraction ran against the cache: twenty minutes of local parsing, zero new requests to the target, zero ban risk. Without the cache, the "fix" would have meant re-crawling two million pages — a week of polite crawling and a very suspicious target. This is the highest-ROI caching story I have, and it's why I treat the cache as part of the pipeline, not an optimization.

The worker that crashed at 2 a.m. An in-process queue with everything in memory: at 2 a.m. a worker threw an unhandled exception, the process died, and with it the frontier and the seen set. Eleven hours of work gone, and recovery meant re-fetching four hundred thousand URLs just to rebuild the seen set — four hundred thousand requests the server had already answered. The fix was the migration this guide walks through: the frontier moved to Redis, the seen set became a SQLite unique index, and the crawl became something you can reboot without thinking about. The 2 a.m. crash still happens. It just stopped mattering.

The Bloom filter that ate one percent of a marketplace. A listings crawl moved its dedup to a Bloom filter to save memory. Nothing errored. A month later, reconciliation against the site's own sitemap showed the crawl had silently missed roughly one percent of listings — a false-positive rate that was agreed to in theory and painful in practice. The listings had to be re-crawled. The lesson wasn't that Bloom filters are wrong; it was that the choice of dedup structure is a business decision about how much data you can afford to miss, and you should make it explicitly, not by default. For that crawl, the answer was a Redis set.

Key takeaways

Crawl from a queue, not a loop. The queue is where resumability, bounded concurrency, and URL discovery live.
Cap concurrency per domain — 2 to 5 connections — and rate-limit per domain with jitter. More concurrency past the peak is just more 429s.
Cache every response to disk, with a TTL per content type, an atomic write, and a namespace you bump when the parser changes. Never re-request bytes you already have.
Retry only transient failures (timeouts, 5xx, 429) with exponential backoff plus jitter. Never retry 400, 401, 403, 404, or 410. Honor Retry-After.
Normalize URLs before dedup, and make the claim atomic. Use a Redis set or a database unique index for anything you care about; use a Bloom filter only when you can afford to miss URLs.
Be polite: per-domain crawl delay, identify yourself, respect robots.txt, ramp up, and use an API or data dump when one exists.
Migrate queues in phases: script, in-process, Redis or RabbitMQ, then a database-backed frontier. Move when your current phase starts costing you real work.
Persist the frontier and the seen set, make fetches idempotent, and write atomically, so a crash at 2 a.m. is an inconvenience rather than a lost run.
Monitor throughput, error rate by status, the 429 rate, cache hit rate, and ban signals — and back off the moment 429s start climbing.
Prefer Scrapy once you have a real crawl; outgrow it when your scheduling and storage needs are unusual; buy a managed API when the infrastructure you'd assemble costs more than the fee.

Frequently Asked Questions

How many concurrent requests is safe when scraping?

There's no universal number. A safe rule is 2 to 5 connections per domain, and keep your average rate well under what one fast human clicker would do. Start at 1 or 2 concurrent. Increase only if the server's response times and status codes stay healthy.

Should I cache scraped HTML to disk?

Yes, almost always. Caching raw responses means you never re-request a page during development or after a parser bug. It's faster and dramatically more polite to the target. A simple file-per-URL or SQLite cache is enough for most projects.

Scrapy or requests for a large crawl?

Use requests for small one-off scrapers. Use Scrapy for anything that grows into a real crawl. Scrapy gives you a scheduler, concurrency control, retries, dedup, and an item pipeline for free. That's exactly the scaffolding you'd otherwise rebuild yourself.

How do I deduplicate URLs in a crawler?

Normalize URLs (lowercase scheme and host, strip default ports, sort query params, drop fragments) and keep a persistent set of what you've already enqueued. For very large crawls, use a Bloom filter or a Redis-backed set instead of an in-process Python set.

What is exponential backoff with jitter?

On a transient failure you wait, then retry, doubling the wait each attempt: 1 second, 2, 4, 8, capped. Jitter adds a small random offset so your retries don't line up with every other scraper retrying at the same moment. Without jitter, a burst of 429s turns into synchronized hammering.

Redis or RabbitMQ for a scraping queue?

Both work. Redis with BRPOPLPUSH is the easiest reliable queue: it lives outside your process, survives worker crashes, and keeps the frontier shared across workers and machines. RabbitMQ adds acknowledgements, priorities, and dead-letter queues, which matter when you need at-least-once delivery guarantees and per-URL retry accounting.

How do I know my scraper is about to get banned?

Watch the trend, not a single number. A rising share of 429s, a sudden jump in 403s or CAPTCHA pages, climbing time-to-first-byte, and a drop in cacheable response sizes are the classic warning signs. Back off the affected domains the moment 429s start rising, not when they hit double digits.

Should I build my own crawler or use a managed scraping API?

Build when you crawl a handful of domains you control and you can amortize the engineering. Buy when you need proxies, browsers, and CAPTCHA solving across thousands of domains, because replicating that infrastructure costs far more than the per-page fee. The break-even is usually somewhere around a few million pages a month.

Keep reading

Found this useful? Cite it as: webscraping.space. “Scraping at scale: queues, caching, and not getting banned.” https://webscraping.space/blog/scraping-at-scale. Published 2026-07-12.