URL: https://webscraping.spaceBest viewed at 1024×768

Anti-Bot Published Aug 8, 2026 · 31 min read · 6,780 words

Web Scraping Without Getting Blocked: The 2026 Anti-Ban Playbook

How to scrape without getting blocked: the behavioral and operational playbook — politeness and rate shaping, realistic headers and fingerprints, caching, retries, ban detection, IP strategy, and recovery. What actually keeps you unbanned, not proxy marketing.

Everyone who asks me how to scrape without getting blocked is expecting proxy marketing. They want a pitch about residential IP pools and captcha solvers, and they want it to be complicated, because they assume the arms race is technical. The truth is more boring and more useful: most bans are caused by behavior, not by forensics. You don't get blocked because you're a bot. You get blocked because you act like a bot — too fast, too regular, too predictable, and too careless about re-requesting pages you already have.

This post is the operations playbook: the behavioral and operational layer that decides whether a scraper lives or dies. It covers the levels of enforcement, the signals that get you caught, rate shaping, caching, ban detection, IP strategy, and recovery. It is deliberately not the TLS-and-fingerprint deep dive — that's Bypassing anti-bot protections, which owns the curl_cffi and JA3 territory. This post owns everything around it: the discipline that makes a good fingerprint unnecessary in the first place.

I'm going to give you concrete numbers, runnable code, and the honest tradeoffs. I have run crawlers that have survived on the same IP for months and crawlers that died in an afternoon. The difference was almost never the proxy.

The ban pyramid: what "getting blocked" actually looks like

"Getting blocked" is not one thing. It's a ladder of enforcement with four rungs, and each rung is cheaper to detect and recover from than the one above it. Most people don't realize they're on the ladder until they're at the top.

The four enforcement levels. Everything below the top two is survivable with rate discipline; the top of the pyramid is where a session dies and no proxy pool revives it.

Level 1 — soft rate limiting. The site throttles you but doesn't block you. You see 429 Too Many Requests, Retry-After headers, and occasionally pages served 5-30 seconds slow to punish you. This is the site saying slow down, and it's the only level that's entirely recoverable. If you see 429s, you're already past your sustainable rate; back off now.

Level 2 — challenge pages. A JavaScript challenge, a CAPTCHA, Cloudflare's "Just a moment" interstitial, DataDome blocks, Turnstile widgets. The site isn't sure you're a bot but wants proof you're a browser. This is a warning shot. It happens when your rate is fine but your fingerprint or your reputation is borderline. It's solvable, but every challenge you solve is a data point the vendor keeps, and repeated challenges push you up the ladder.

Level 3 — hard IP ban. Every request from your IP returns 403 with a blank or generic "access denied" body. No challenge, no captcha, no retry that helps. This is a firewall rule, usually with a TTL of hours to months. The IP is done until it expires or you rotate it.

Level 4 — fingerprint/account ban. The site has built a reputation profile around your client fingerprint or your account. A 403 follows you across every IP you try because the detection is on the TLS handshake, the HTTP/2 settings, or the account itself, not the address. This is the level where a proxy pool stops mattering entirely, and where you have to fix the client or the account and wait out a long cooldown.

Here's the full table of what each level looks like, how to detect it, and what to do about it:

Level	How it looks	How to detect it	Response
1 — Soft rate limiting	429s, `Retry-After` headers, throttled/slow pages	status 429; `Retry-After` present	Slow down, honor `Retry-After`, add jitter
2 — Challenge pages	"Just a moment", CAPTCHA iframe, Turnstile, DataDome block	challenge markers in body (`cf-chl-*`, captcha tokens, `x-datadome`)	Pause, reduce rate, check fingerprint quality
3 — Hard IP ban	403 on every request, empty or generic body	403 with no challenge, on all paths, from same IP	Change exit IP, wait out TTL, review cadence
4 — Fingerprint/account ban	403 from fresh IPs too; login/account blocked	new IP still blocked; challenge on every path	Fix client fingerprint or account; long cooldown

The operational point is simple: detect each level as soon as it starts, because the cost of recovery multiplies as you climb. A level-1 429 costs you thirty seconds. A level-3 ban costs you an IP and a day. A level-4 ban costs you the whole target.

Why scrapers get caught

Anti-bot systems don't run on magic. They run on signals, and nearly all of those signals are behavioral. Here is the honest list of what gets you caught, in rough order of how often I see it in real crawler logs:

1. Request rate and cadence. This is the big one. A naive scraper fires a request every 400 milliseconds, on a metronome, for hours. No human does that. A human reading a page leaves gaps of 3 to 8 seconds, and the gaps are irregular — sometimes you pause to read, sometimes you scroll, sometimes you get distracted and stop for forty seconds. The inter-arrival time distribution is the single most distinguishing feature between a human session and a bot session, and every serious detection system models it.

A naive bot produces a single tall spike at the timer interval; a human produces a broad, skewed distribution with a long tail. Detection systems fit a model to this shape, so the fix is to make your gaps look like the right distribution, not to make them exactly equal.

2. Request patterns in the URL space. Bots crawl in structured order — page 1, 2, 3, ... sequentially, or all the category URLs in a burst, or every ID in a numeric sequence. Humans jump around: they open a listing, click a result, go back, click another. If your crawler walks pagination linearly at a steady rate, the site sees a perfect sweep. Two practical mitigations: shuffle the crawl frontier so you don't hit URLs in sorted order, and interleave different paths rather than finishing one section before starting another.

3. Zero dwell time. A human spends tens of seconds looking at a page before the next request. A scraper downloads and moves on in 200 milliseconds. Combined with the gap between requests, dwell time is cheap for you to simulate and trivial for them to measure: just look at the distribution of time-on-page. Even a modest fake dwell of 1-3 seconds changes the distribution a lot.

4. Fingerprint inconsistencies. This is the layer the other post owns, so I'll be brief: your TLS handshake (JA3/JA4), your HTTP/2 settings frame, your header order, and your actual header values must all be consistent with the same real browser. A curl UA with a Python TLS fingerprint is a self-describing bot. Python's requests will get identified on sight no matter what headers you send. If you need the deep details, read the bypassing anti-bot protections post — the short version is that you want curl_cffi with impersonate="chrome" or a stealthed browser client.

5. Behavioral tells. No mouse movement, no scroll events, no JS execution at all, no referrer flow. A scraper requesting the product page directly with no referer and no history looks like a crawler even at a polite rate. Sending the right Referer that matches your crawl path (the listing page, then the product page it linked to) is a cheap, underrated fix.

6. IP reputation. Datacenter ASNs are flagged, some hosting ranges are blocked wholesale, and an IP that other scrapers have hammered is already on a watchlist before you touch it. If your requests per IP stay polite, reputation matters less — but if you rotate through a pool of already-abused datacenter IPs, you inherit their history. This is where residential proxies genuinely earn their money, and it's also where most people over-invest.

The honest framing: rate and cadence get you caught 90% of the time. Fingerprint and reputation matter, but they're usually the reason the site confirms the suspicion that your cadence already created.

Politeness is the biggest lever

I've never once seen a crawler get banned at one request every three seconds with jitter. Not once. The bans I've debugged always trace back to a sustained rate the site's operators would never produce themselves, run for long enough that the error rate climbed past the threshold.

So let's do the rate math properly. What does "one fast human" actually mean? A person clicking through a catalog, reading, skimming, going back — you get maybe 0.5 to 1.5 page views per second at the absolute extreme, for a few minutes. Sustain that over an hour and you're at roughly 2,000-4,000 requests. Most humans are far slower. The polite rule I use: keep sustained traffic at or under about 1 request per second per IP, with gaps of 1-3 seconds that vary. That's already 3,600 requests an hour from one IP. For nearly every target, that's more than enough throughput for the data you actually need.

The relationship between rate and block probability is not linear. Below a threshold — somewhere around 1-2 requests per second per IP — detection systems mostly ignore you, because the cost of false positives outweighs the benefit. Above 4-5 requests per second, you're outside human bounds and the probability of a block within 30 days climbs sharply.

Block risk is low and flat below about 2 requests per second, then turns sharply upward past 4-5. The whole strategy is to live left of the knee and spend your effort on throughput that doesn't require more requests per second.

Two concepts from the politeness toolbox that are worth naming precisely:

Crawl-delay. The classic robots.txt-era concept: a minimum gap between requests to the same host. It's what every polite crawler implements, and it's the single most effective anti-ban control you have. One request per 1-3 seconds per domain, with jitter, and you are indistinguishable from a busy human by rate alone.

Burst vs sustained. Humans burst. They click three pages in a row (following a link chain), then pause for 20 seconds. A token bucket with a small burst capacity (say 2) and a low sustained rate (1/s) reproduces exactly that shape: it allows short bursts, then forces the pause. This is much more realistic than a fixed 1-second timer, because the distribution of gaps looks human even though the average is the same.

Politeness also has a second-order benefit: it makes the site's operators not care. A crawl at 3,600 requests a day that follows robots.txt and never triggers an alert costs a site almost nothing. A crawl at 3,600 requests an hour costs them real CPU and makes them open the blocklist. You're not just evading detection; you're avoiding becoming worth detecting.

Rate shaping in code

Everything above reduces to a few small pieces of code. Here's the token bucket limiter I use, which gives you bursts with a controlled sustained rate:

import threading, time

class TokenBucket:
    """rate tokens/sec sustained, capacity burst. take() returns seconds to sleep."""
    def __init__(self, rate=1.0, capacity=2.0):
        self.rate, self.capacity = rate, capacity
        self.tokens = capacity
        self.updated = time.monotonic()
        self.lock = threading.Lock()

    def take(self, n=1):
        with self.lock:
            now = time.monotonic()
            self.tokens = min(self.capacity,
                              self.tokens + (now - self.updated) * self.rate)
            self.updated = now
            if self.tokens >= n:
                self.tokens -= n
                return 0.0
            wait = (n - self.tokens) / self.rate
            self.tokens = 0.0
            return wait

bucket = TokenBucket(rate=1.0, capacity=2.0)   # 1 req/s, burst of 2

def polite_get(session, url):
    time.sleep(bucket.take())
    return session.get(url, timeout=15)

That's the whole mechanism: the bucket refills at rate tokens per second, allows up to capacity as a burst, and take() tells the caller how long to sleep before the next request. time.sleep(bucket.take()) is your politeness layer.

Next, per-domain jitter. Rate limiting per domain matters because one polite crawler on one domain shouldn't pay for a fast crawl on another. And the irregularity is the point, so add jitter:

import random, time
from collections import defaultdict

last_request = defaultdict(float)   # domain -> last request time

def wait_for_domain(domain, floor=1.0, ceiling=4.0):
    """Sleep so the gap to the previous request for this domain lands
    uniformly in [floor, ceiling] seconds. Irregular, human-ish."""
    now = time.monotonic()
    gap = random.uniform(floor, ceiling)
    time.sleep(max(0.0, gap - (now - last_request[domain])))
    last_request[domain] = time.monotonic()

Two things to notice. First, the sleep is gap - elapsed, not gap — if another thread already waited recently for this domain, we don't stack delays. Second, random.uniform(floor, ceiling) produces a flat distribution, which is not exactly the human shape from the histogram, but it's close enough to defeat fixed-interval detection, which is the common case. If you're being specifically profiled, fit a lognormal distribution instead; for everyone else, uniform jitter is 90% of the value.

Now the retry policy. Transient errors are normal, and how you retry them is part of your fingerprint. Retry only what's transient, honor the server's instructions, and back off exponentially:

429: the server told you exactly how long to wait. Sleep the Retry-After value, or 2^n if it's absent.
5xx: transient server error. Back off with 2^n seconds plus jitter.
4xx (other than 429): a permanent refusal. Retrying a 404 or a 401 is how you look stupid and hostile. Do not retry.

This is worth restating because it's the highest-frequency mistake in scraper logs: people retry 403s. A 403 is not transient. If the site blocked you, hammering the block with retries is how a temporary block becomes a permanent one. Log it, mark the IP bad, and move on.

Caching: never re-request what you already have

Here's a number that surprises people: in a typical development cycle, 70% of your requests are repeats. You run the scraper, the parser throws, you fix the parser, you run it again — and every run re-fetches the same pages. At scale, you re-crawl the same catalog every night, and 95% of it didn't change. Each of those requests is a chance to look like a bot and load on the target. None of them are necessary.

Caching is the highest-ROI anti-ban control there is, because it's the only one that reduces your request volume while also making your scraper faster and your operators happier. A disk cache is twenty lines:

import hashlib, json, pathlib, time

CACHE = pathlib.Path("response_cache")
CACHE.mkdir(exist_ok=True)

def cache_path(url):
    key = hashlib.sha256(url.encode()).hexdigest()
    return CACHE / f"{key}.json"

def cache_get(url, max_age_s=None):
    p = cache_path(url)
    if not p.exists():
        return None
    entry = json.loads(p.read_text())
    if max_age_s is not None and time.time() - entry["ts"] > max_age_s:
        return None
    return entry["body"]

def cache_set(url, body, meta=None):
    cache_path(url).write_text(json.dumps({
        "url": url, "ts": time.time(), "body": body, "meta": meta or {},
    }))

Use it everywhere: check the cache before any network call, write to it after every successful fetch. During development, the cache alone cuts origin hits by 60-70%. In production, a nightly crawl of pages that mostly don't change turns into a diff — fetch only what changed, replay the rest from disk. If you're building this into a real crawler, the scraping at scale post has the queue, dedup, and concurrency scaffolding that pairs with it.

The compounding effect over a development cycle looks like this:

Ten days of a dev loop and a nightly refresh. Without a cache the origin sees 10,000 requests; with a disk cache it sees roughly 3,350. That's 6,650 requests that never gave the anti-bot system a look at your traffic.

The dedup half of caching is normalizing URLs before you touch the cache or the seen-set. The same page reachable as /product?id=5 and /product?id=5&utm_source=newsletter and PRODUCT/ID=5 is three requests to you and one to the site. Normalize:

from urllib.parse import urlsplit, urlunsplit, parse_qsl, urlencode

def normalize(url):
    s = urlsplit(url)
    netloc = s.netloc.lower()
    if netloc.endswith(":80"): netloc = netloc[:-3]
    if netloc.endswith(":443"): netloc = netloc[:-4]
    query = urlencode(sorted(parse_qsl(s.query)))   # sort params, drop dups
    return urlunsplit((s.scheme.lower(), netloc, s.path.rstrip("/") or "/", query, ""))

Lowercase the scheme and host, drop default ports, sort query params, strip the fragment. For very large crawls, swap the in-memory set for a Redis set or a Bloom filter — the scraping at scale post covers that.

Headers and fingerprints: the basics

I'm going to keep this section short because the bypassing anti-bot protections post is the deep dive. But you need the basics here, because a polite rate is necessary but not sufficient — if your client looks like a robot at the TLS layer, no amount of pacing saves you.

User-Agent. Send one real, current browser UA that matches your TLS fingerprint. A recent Chrome or Firefox string, matching the client you're impersonating.
Accept-Language and standard headers. A real Accept-Language (en-US,en;q=0.9) and the standard request headers a browser sends, in a sane order.
Referer flow. When you crawl a listing page and then a product page it linked to, send the listing URL as the Referer. A direct request with no referer is a crawler tell.
Do not randomize User-Agents. Rotating UA strings from a list of a hundred scraped UAs is itself a fingerprint — real browsers don't change identity mid-session, and the distribution of UAs vs your TLS fingerprint will be inconsistent. Pick one real profile, keep it stable, and pair it with a client that reproduces that browser's TLS handshake.
When you need a real browser client. Python's requests has a TLS fingerprint that no browser produces; it gets identified instantly. If the target checks the TLS or HTTP/2 layer, use curl_cffi with impersonate="chrome", or a stealthed Playwright/Camoufox for JS challenges. The rule: match the client to the target's scrutiny.

The consistency point is the one most people miss: everything about a session should be coherent. One browser's UA, one browser's TLS fingerprint, one browser's header order, one IP, one cookie jar. Mixed signals are how you look assembled rather than real.

Ban detection and monitoring

Politeness keeps you out of the pyramid's top two levels. Detection keeps you out of level 3. The skill is simple: watch the error rate in a sliding window, and act before the ban lands, not after. By the time every request is a 403, you've lost the IP.

First, classify responses. A 429 is not the same as a 403, and a challenge page isn't a block:

import collections, time

WINDOW_SECONDS = 300
hits = collections.defaultdict(collections.deque)   # ip -> [(ts, verdict)]

def classify(status, text):
    if status == 429:
        return "rate_limited"
    if status == 403 and "captcha" in text.lower():
        return "captcha"
    if status == 403:
        return "blocked"
    if status >= 500:
        return "server_error"
    return "ok"

def error_rate(ip, verdict):
    """Sliding-window share of non-ok responses for one exit IP."""
    now = time.time()
    q = hits[ip]
    while q and q[0][0] < now - WINDOW_SECONDS:
        q.popleft()
    q.append((now, verdict))
    n = len(q)
    bad = sum(1 for _, v in q if v != "ok")
    return bad / n if n else 0.0

The window size matters. Five minutes is short enough to react quickly and long enough to smooth out single glitches. The verdict categories matter too: if you see mostly rate_limited, the fix is speed; if you see captcha and blocked, the fix is fingerprint and IP. A rising share of challenge pages is the early warning — challenges precede blocks, because the site gives you a chance to go away before it escalates to a firewall rule.

Second, act on the rate with an auto-pausing circuit breaker. When the error rate for an IP crosses a threshold, pause that IP for a cooldown that grows with each strike:

class Breaker:
    def __init__(self, threshold=0.15, pause_base=30, pause_max=1800):
        self.threshold = threshold
        self.pause_base, self.pause_max = pause_base, pause_max
        self.paused_until = 0.0
        self.strikes = 0

    def wait_if_paused(self):
        while time.time() < self.paused_until:
            time.sleep(5)

    def trip(self, seconds=None):
        delay = seconds or min(self.pause_base * (2 ** self.strikes), self.pause_max)
        self.paused_until = time.time() + delay
        self.strikes = min(self.strikes + 1, 6)
        _alert(f"error rate > {self.threshold:.0%}: pausing {delay:.0f}s")

    def ok(self):
        if self.strikes and time.time() > self.paused_until:
            self.strikes = max(0, self.strikes - 1)   # decay after cooldown

def _alert(msg):
    print(f"ALERT {time.strftime('%H:%M:%S')}: {msg}")   # wire to Slack/email

The thresholds I default to: pause the crawl at a 15% error rate, escalate the pause from 30 seconds to 30 minutes, and treat a sustained 25%+ as a hard stop for that IP. Those numbers come from watching what actual bans look like: a healthy crawl sits at under 2% errors. The moment you see 10%+, something changed, and it's almost always you.

What a ban looks like in the telemetry. The error rate climbs from 2% to over 40% in under an hour. A circuit breaker that pauses at 15% preserves the IP; waiting until every request 403s throws it away.

The third piece is alerting, and it's the one people skip. A crawler that runs unattended at 2 a.m. needs to page someone when it trips the breaker. My minimum bar: an alert on the first breaker trip, an alert on a full IP ban, and a daily digest of per-IP error rates. The alert message should include the IP, the error mix (how many 429 vs 403 vs captcha), and the URL that triggered it — because the URL usually tells you which behavior is the problem.

IP strategy without the marketing

The proxy industry wants you to believe you need a rotating pool of 10,000 residential IPs. The truth is more economical: one polite IP is enough for most targets, and rotation is a tool for specific problems, not a default.

When one IP is enough: the target doesn't IP-filter aggressively, your rate is under 1-2 req/s, and the fingerprint is clean. I've run crawlers that pulled a hundred thousand pages a month off a single static IP for years. Every time someone asks me "why am I banned after a week?" and the log shows 8 req/s from one IP, the answer isn't more IPs, it's the rate.

When you need rotation: the target has a strict per-IP cap that's below your needed throughput; you're hitting geo-restrictions; or one IP got flagged and you need a clean exit while it cools down. In those cases, the residential proxies for scraping post covers the provider and protocol tradeoffs. The operational rules you need here are:

Sticky beats rotating. Keep one IP per session, per target, for as long as it behaves. A session that flips IPs every request is itself a bot fingerprint — no human changes address mid-session. Sticky sessions also keep cookies, rate limits, and reputation coherent.
Respect per-IP limits even when rotating. If you have 50 IPs, the polite thing is 50 IPs each under the human threshold, not 50 IPs each hammering. Rotation multiplies throughput; it doesn't multiply the rate you're allowed per IP. A 500-IP pool at 10 req/s each is still 5,000 req/s hitting one origin, and the origin will notice the aggregate.
Retire IPs on their own merits. Track error rate per IP. When an IP crosses the threshold, drop it from rotation and let it cool down, instead of rotating it back into the pool an hour later with the same behavior.
The exit IP is part of the fingerprint. The UA, the TLS profile, and the IP should look like they belong together. A residential IP with a datacenter-grade TLS fingerprint is incoherent. Pair the proxy type with the client you're impersonating.

The honest tradeoff: proxies cost money and add failure modes (dead IPs, slow hops, provider instability). Before you buy them, spend the two days on rate shaping and caching, because they're free and they fix the actual cause of 80% of bans.

The robots.txt contract

robots.txt is not a security control — anti-bot systems don't read it before deciding you're a bot. But it's the consent contract, and staying inside it is the easiest way to stay unbanned for a reason most people miss: the paths a site excludes with robots.txt are exactly the paths it's most willing to defend. If a site says Disallow: /search, hammering /search is how you get rate-limited into an IP ban while the site's operators nod and don't lift a finger to help you.

The practical reading:

User-agent groups. Match your crawler's UA against the applicable group. If the site has a rule for your bot, it's telling you what's permitted. Ignoring a specific Disallow for your own UA is the clearest possible way to get on a watchlist.
Crawl-delay. Non-standard but widely honored: the minimum seconds between requests. If a site sets Crawl-delay: 2, treat it as the floor for your per-domain delay. It's the site telling you its tolerance in advance.
Disallow vs Allow. The contract says which paths are off-limits for automated access. Staying out of them is cheap, keeps your crawl defensible, and — practically — keeps you off the defended paths where blocks actually get issued.
Sitemaps. Sitemap: entries aren't permission, but a site that publishes a sitemap is signaling it's OK with being crawled in a structured way. Crawling the sitemap at a polite rate is the most cooperative behavior available.

I wrote the full legal and ethical picture in the web scraping ethics and robots.txt guide. The operational summary is: robots.txt doesn't protect you from detection, but it protects you from the consequences of detection. A site that catches a crawler that respected robots.txt says "fine, slow down." A site that catches a crawler ignoring it says "block." When you're choosing which hills to fight on, don't pick the one where the site has already written down its terms.

Recovery: what to do after the ban lands

Even with all of this, bans happen. The recovery playbook matters as much as the prevention, and it's short:

1. Stop. The instant you confirm a level-3 or level-4 ban, halt the crawl for that target. Continuing to hammer a blocked endpoint is what converts a temporary block into a permanent one. Your breaker should already be doing this; if you didn't have a breaker, now is when you write one.

2. Change the exit. If the ban is per-IP (level 3), switch to a fresh IP and verify with one request that it's clean. If the ban follows you across IPs (level 4), you have a fingerprint or account problem, and changing IPs won't help — fix the client first.

3. Review the cadence log. The ban didn't happen by accident. Look at the last hour of requests before the first 403: what rate were you sustaining, what paths were you hitting, what did the error mix look like on the way up? The answers tell you which section of this post you skipped. In my experience it's almost always the rate or the retry policy, and the fix is a one-line change to a constant.

4. Wait out the TTL. Most per-IP bans are 24-72 hours, though some last months. Plan around the range: if you need the data now, rotate; if you can wait, wait — a cooled-down IP that you resume politely beats a scorched-earth rotation campaign. A level-4 ban often needs a full cooldown of a week or more while the reputation profile ages out.

5. Resume slower than you think you need to. Come back at half the previous rate, with jitter, and watch the error rate for the first hour. The first hour after a ban is the probation period; one more burst and you're back on the blocklist, possibly for much longer.

The uncomfortable truth about recovery is that there's no trick. The tools that get you unbanned are the same tools that keep you unbanned: slow down, be irregular, cache, detect early, and don't retry blocks.

The complete polite scraper

Here's everything above assembled into one scraper: rate limiting, jitter, disk cache, ban detection, and a circuit breaker. It's about seventy lines, it's runnable, and it has kept the pattern of crawler alive for me for years.

import hashlib, json, pathlib, random, time
from collections import defaultdict, deque
from urllib.parse import urlsplit, urlunsplit, parse_qsl, urlencode

CACHE_DIR = pathlib.Path("cache"); CACHE_DIR.mkdir(exist_ok=True)

def normalize(url):
    s = urlsplit(url)
    netloc = s.netloc.lower()
    if netloc.endswith(":80"): netloc = netloc[:-3]
    if netloc.endswith(":443"): netloc = netloc[:-4]
    q = urlencode(sorted(parse_qsl(s.query)))
    return urlunsplit((s.scheme.lower(), netloc, s.path.rstrip("/") or "/", q, ""))

def cache_get(url):
    p = CACHE_DIR / f"{hashlib.sha256(url.encode()).hexdigest()}.json"
    if p.exists():
        return json.loads(p.read_text()).get("body")

def cache_set(url, body):
    p = CACHE_DIR / f"{hashlib.sha256(url.encode()).hexdigest()}.json"
    p.write_text(json.dumps({"url": url, "ts": time.time(), "body": body}))

# 1-3s jitter per domain
last_req = defaultdict(float)
def pace(domain):
    gap = random.uniform(1.0, 3.0)
    time.sleep(max(0.0, gap - (time.monotonic() - last_req[domain])))
    last_req[domain] = time.monotonic()

# Token bucket: 1 req/s sustained, burst of 2
tokens, refill = 2.0, time.monotonic()
def throttle():
    global tokens, refill
    now = time.monotonic()
    tokens = min(2.0, tokens + (now - refill))
    refill = now
    if tokens < 1.0:
        time.sleep(1.0 - tokens); tokens = 0.0
    else:
        tokens -= 1.0

# Ban detection: sliding 5-minute window, per IP
recent = defaultdict(deque)
def verdict_of(status, text):
    if status == 429: return "rate_limited"
    if status == 403 and "captcha" in text.lower(): return "captcha"
    if status == 403: return "blocked"
    if status >= 500: return "server_error"
    return "ok"

paused_until, strikes = 0.0, 0
def check_ban(ip):
    global paused_until, strikes
    now = time.time()
    q = recent[ip]
    while q and q[0][0] < now - 300: q.popleft()
    rate = sum(1 for _, v in q if v != "ok") / max(1, len(q))
    if rate > 0.15 and now > paused_until:
        delay = min(30 * 2 ** strikes, 1800)
        paused_until = now + delay; strikes += 1
        print(f"[breaker] {ip} error rate {rate:.0%} — pausing {delay:.0f}s")

def fetch(session, url, ip):
    while time.time() < paused_until:
        time.sleep(5)                     # cool down, don't hammer
    body = cache_get(url)
    if body is not None:
        return body
    pace(urlsplit(url).netloc)
    throttle()
    try:
        r = session.get(url, timeout=15)
    except Exception:
        time.sleep(5); return None        # network blip, skip for now
    text = r.text
    recent[ip].append((time.time(), verdict_of(r.status_code, text)))
    check_ban(ip)
    if r.status_code == 200:
        cache_set(url, text); return text
    if r.status_code == 429:
        retry = float(r.headers.get("Retry-After", "30"))
        paused_until = max(paused_until, time.time() + retry)
    return None                            # 403/404/5xx: log, don't retry

def crawl(session, urls, ip="1.2.3.4"):
    for raw in urls:
        url = normalize(raw)
        body = fetch(session, url, ip)
        if body:
            yield url, body

The design decisions are all deliberate. The cache is checked before the rate limiter, so cached pages cost nothing and never touch the network. The pace and throttle run together — jitter shapes the gaps, the bucket enforces the sustained rate and allows small bursts. 429s pause the crawl using the server's own Retry-After. 403s are recorded and not retried — a retried block is how you lose an IP. And the breaker watches the sliding-window error rate, pausing with exponential cooldowns before a hard ban finishes forming.

Swap session for requests.Session() (or curl_cffi's session if the target checks TLS), pass the seed URLs, and this is a working, polite crawler. From here, the scraping at scale post shows you how to grow it into a queue-driven system with bounded concurrency and dedup for millions of pages.

Or just buy an API

I've been writing this whole post about how to do it yourself, so let me be honest about when not to. There is a class of scraping work where the correct answer is to buy an API, and refusing to is ego and cost, not engineering.

You should build your own when: the target is stable, you control the contract (your own site, an API you're permitted to use, a well-behaved public target), the volume is modest, and the data shape is something you'll iterate on. The playbook in this post is cheap to implement, and self-hosting keeps you independent.

You should buy when: the target is hard (Cloudflare, DataDome, heavy JS), the volume is large, the pages change shape constantly, or your time is worth more than the bill. A managed extraction or crawler API handles rotation, fingerprint maintenance, and challenge-solving as a service — and it passes the liability problem along too, which is real. The build-vs-buy math is covered properly in Best Web Crawler APIs in 2026: Build vs Buy, and the extraction layer that turns raw HTML into clean data is covered in Website Content Extraction API: The 2026 Guide.

My actual rule, after years of running crawlers: build the polite engine yourself, because it's a durable skill and it's cheap; buy the anti-bot arms race, because it's a treadmill and it's never done. And if you buy, buy for the targets that need it — don't pay a vendor per request to crawl the easy 80% that your own polite scraper would handle for free.

Key takeaways

Bans are mostly behavioral, not forensic. Rate and cadence get you caught long before fingerprints do.
Live below the knee: about 1-2 requests per second sustained per IP, with irregular 1-3 second gaps. Bursts are fine; sustained speed is the killer.
Jitter is the point. A fixed timer is a bot fingerprint; a flat or lognormal distribution of gaps is a human one.
Cache everything and normalize URLs. Caching cuts origin requests 60-70% in development and makes every re-crawl a diff.
Detect early, act early: watch the sliding-window error rate, pause at 15%, halt at 25%, and never retry a 403.
One polite IP beats a thousand rotating ones behaving badly. Rotate sticky, respect per-IP limits, and retire bad IPs.
Stay inside robots.txt. It won't stop detection, but it keeps you off the paths the site defends hardest.
After a ban: stop, change the exit, review the cadence log, wait out the 24-72 hour TTL, and resume slower.
Match the client to the target — polite pacing plus a browser-accurate TLS fingerprint (see the anti-bot post) is the complete recipe.

Frequently Asked Questions

How do I avoid getting blocked while scraping?

Most bans are behavioral, not forensic. Stay under about 1 to 2 requests per second sustained per IP, add jitter so gaps are irregular, cache every response so you never re-request, honor 429s and Retry-After, use a real browser-accurate fingerprint, and detect rising error rates early so you back off before a hard block lands. Politeness and monitoring beat proxy rotation every time.

Why does my scraper keep getting banned?

Almost always one of four things: you're requesting too fast and too regularly, you're re-requesting the same pages in every run, your TLS/HTTP-2 fingerprint doesn't match any real browser, or you're hammering paths the site protects. A custom User-Agent no longer matters in 2026 — detection reads your request cadence and your TLS handshake before it reads your headers.

How fast is too fast when scraping?

Treat one fast human as the ceiling: roughly 1 to 2 requests per second, with irregular gaps. Sustained traffic above about 4 to 5 requests per second per IP makes a block within 30 days statistically likely on most protected sites. Start at one request every 2 to 3 seconds with jitter, and only go faster after watching the error rate.

How do I know if I'm about to be blocked?

Track your error rate over a sliding window: if 429s appear, you're at level one and should slow down. If 403s climb past about 15% of requests, a hard block is forming — pause. If you suddenly get 403 on every URL from a fresh IP, the block already landed. A rising share of challenges and CAPTCHAs is the other early signal. Watch these and you act before the ban, not after.

Do rotating proxies stop bans?

Rotation spreads load across IPs and fixes IP reputation, but it does not fix behavior. Aggressive rotation — a new IP per request — is itself a bot fingerprint, and most anti-bot systems detect the client fingerprint before they route on IP. Use a browser-accurate client, stay polite per IP, and rotate only when a single IP genuinely can't hold the rate you need.

What's a good crawl delay when scraping?

Start with a per-domain delay of 1 to 3 seconds and add jitter so gaps are irregular. If the site serves you 200s at one request every 2 seconds, you're well inside what one fast human does. Only lower the delay after watching the error rate for a while, and never go below about one request per second sustained on a single IP.

Does respecting robots.txt keep me from getting blocked?

Not by itself — anti-bot systems don't read robots.txt before deciding you're a bot. But staying inside it keeps you off the paths a site is most likely to defend, keeps your crawl defensible, and removes the most common reason an operator reports you. It's the cheapest risk reduction available, and it's usually the difference between 'slow scrape' and 'incident'.

How long should I wait after my IP gets blocked?

Plan on 24 to 72 hours. Most per-IP bans carry TTLs in that range, though some run for months. Change the exit IP, review the request log to find which behavior tripped the ban, and go back slower. Waiting a day is cheaper than rotating a thousand proxies while still behaving like a bot.

Does caching actually prevent bans?

It reduces request volume, and volume is what trips rate-based detection. During development, caching cuts origin hits by 70% or more, because you parse the same pages over and over. Fewer requests means fewer chances to look like a bot and less load on the target. Cache is the highest-ROI anti-ban lever most teams skip.

Keep reading

Found this useful? Cite it as: webscraping.space. “Web Scraping Without Getting Blocked: The 2026 Anti-Ban Playbook.” https://webscraping.space/blog/web-scraping-without-getting-blocked. Published 2026-08-08.