URL: https://webscraping.spaceBest viewed at 1024×768

Proxies Published Aug 8, 2026 · 35 min read · 7,680 words

Residential Proxies for Web Scraping: The Honest Guide

What residential proxies actually do for web scraping, when you genuinely need them, and when they're wasted money: how IP rotation works, residential vs datacenter vs mobile, real pricing, integration with requests and Playwright, and why proxies don't fix a bad fingerprint.

The proxy industry is the single most oversold corner of the scraping world, and residential proxies are its flagship product. Every vendor's homepage promises the same thing: rotate an IP per request and you'll never be blocked again. That's a lie, and the lie costs people real money — thousands of dollars a month for bandwidth they don't need, solving a block that was never IP-based.

Let me be blunt about what this guide will and won't do for you.

A proxy changes which IP address the server sees when your request arrives. That's the core fact. Everything else a proxy marketer tells you is either a corollary of that fact or a fantasy built on top of it. Residential proxies specifically give you IPs that belong to real Internet Service Provider customers — the same pool your home internet connection sits in — which makes them hard to flag on IP reputation alone. They are genuinely useful. They are also genuinely expensive, and they do nothing for the fingerprints that actually get most scrapers blocked in 2026.

This guide is the honest technical explainer: what a proxy physically does, how the three proxy types compare in 2026 pricing, how residential networks are plumbed, exactly which blocks a proxy fixes and which it doesn't, when you need one and when you're burning cash, how to wire proxies into requests, curl_cffi, and Playwright, and the cost math that decides whether renting IPs beats renting a scraping API.

If you came here hoping for "buy proxies, never get blocked," you can leave now. If you want to know precisely what the tool does so you can spend money only where it actually moves your success rate, stay.

What a proxy actually does

A proxy is a middleman for your TCP connection. Your client opens a connection to the proxy server, the proxy opens a second connection to the target, and bytes flow through the proxy in both directions. The target server terminates a connection that came from the proxy's IP. Your client never talks to the target directly, so the target has no way to learn your real IP.

That's the entire mechanism. There is no magic. A proxy is a relay, and a relay changes four things about your traffic:

Your source IP. The one that matters. The server's rate-limit counters, IP reputation lookups, and geo logic all key on this address.
Latency. Every relay adds a hop. Datacenter proxies add maybe 20-60ms on a good day. Residential proxies are worse: your bytes exit through an actual home broadband or mobile connection with variable quality, so figure 50-300ms, sometimes more. A scrape that was 150ms per request direct becomes 350ms per request through residential, and at a million requests that's real wall-clock time.
Reliability. You inherit every failure mode of a network you don't control. Exit IPs die mid-request, residential devices drop off the pool, providers throttle bandwidth, gateways time out. A proxy changes a single point of failure into a distributed one that fails in interesting ways. Your retry logic had better be good.
Routing and reputation. The server doesn't just see an IP. It sees what network that IP belongs to. A datacenter IP advertises "this is a server" by its very ASN. A residential IP advertises "this is a normal person at home." And because the routing path is different, CDNs that correlate your traffic across a session may see a different pattern of TLS connections, request timing, and behavior — sometimes in your favor, sometimes not.

Everything else proxies are marketed on — bypassing Cloudflare, solving CAPTCHAs, evading fingerprinting — is downstream of the IP change and mostly doesn't follow from it. Keep the four-item list in your head. When someone promises you a proxy solution to a problem that isn't in that list, they're overselling you.

The three proxy types, honestly compared

There are three flavors of proxy, and they form a strict ladder of price and cleanliness: datacenter at the bottom, residential in the middle, mobile at the top. There's also a hybrid worth knowing about (ISP/static residential) that sits between residential and mobile in price and behavior.

Datacenter proxies. IPs carved out of cloud and hosting providers — AWS, DigitalOcean, Hetzner, and hundreds of smaller shops. Cheap: roughly $1.50-8 per GB, or $2-4 per IP per month for dedicated IPs, with plenty of flat-rate "unmetered" plans. Fast, because the whole path is data-center-to-data-center. The problem is flaggability: the IP ranges of every hosting provider are public knowledge, and threat-intel feeds list them as "hosting" or "ASN: cloud." Any site that does even basic IP reputation filtering can block an entire hosting ASN with one rule. Datacenter proxies are fine for targets that don't IP-filter at all, and they are the correct first stop for almost every scrape.

Residential proxies. IPs from real ISP customers' connections, aggregated into a pool by the provider (more on the plumbing below). Price: roughly $5-15 per GB in 2026, with volume discounts past 100GB/month. Slower than datacenter and less reliable, because the exits are other people's devices. But the IP reputation is genuinely different: to a target server, a residential IP looks like a normal home user in the country you picked. No hosting-ASN blocklist flags it. That's the entire value proposition, and it's real — up to the limit of the fingerprint problem we'll get to.

Mobile proxies. IPs from cellular carriers, which come out of carrier CGNAT pools. These have the cleanest reputation of all — mobile IPs are rarely in blocklists, and the same public IP is shared by many real subscribers, so a server that "remembers" an IP can't cleanly tie it to one user. Price: $15-40 per GB, the most expensive category by a wide margin. Mobile is for the hardest targets only, and honestly, in most scrapes it's overkill. If your TLS fingerprint is clean and your behavior is sane, residential gets you there for a third of the price.

The hybrid is ISP / static residential: a residential IP that's assigned to you persistently rather than pulled from a rotating pool. It behaves like a residential IP but holds still across sessions, which makes it the right tool for login flows, cookie persistence, and anything that needs to look like "the same person, coming back." Pricing is usually per IP per month, $4-10, plus bandwidth. It's the least talked-about category and often the one you actually want.

Proxy type	2026 cost	Block resistance	Best use case
Datacenter	$1.50-8/GB, or $2-4/IP/mo	Low — hosting ASNs are flagged by default	High-volume crawls of sites that don't IP-filter; the cheap default
ISP / static residential	$4-10/IP/mo plus bandwidth	High — looks like a real home user, stays put	Login flows, cookie sessions, anything needing "the same person"
Residential (rotating)	$5-15/GB	High at the IP level; pool is finite and shared	Large crawls, geo-targeting, search engines, per-IP rate-limit bypass
Mobile	$15-40/GB	Highest — carrier CGNAT, rarely blocklisted	The hardest targets, when residential still fails on IP grounds

The honest framing: datacenter is cheap and flagged, residential is expensive and hard to flag, mobile is cleanest and most expensive — and each step up the ladder buys you a smaller increment of success while charging you a bigger increment of money.

Each step up the ladder buys a smaller increment of block resistance at a much bigger increment of price. Start at the bottom-left and escalate only when you can point at an IP-level block.

How residential proxy networks actually work

Before you pay for one, it's worth understanding what you're buying at the plumbing level. A residential proxy provider is not a datacenter with a bigger IP block. It's an aggregator of other people's connections.

The pool. The provider signs up real internet users through apps, browser extensions, and SDKs embedded in third-party software. Those users consent — often buried in an app's terms, sometimes compensated with a "free VPN" or points — to let a small slice of their connection's bandwidth be used as exit nodes. The provider aggregates these consented connections into a pool of millions of exit IPs, organized by country, city, and often ASN. This is the consent/broker model, and it's the entire reason residential IPs exist as a product: you can't buy a million real home IPs, so you rent other people's.

The gateway. Your scraper doesn't connect to the pool directly. It connects to the provider's proxy gateway — a hostname and port you put in your client's proxy config — and authenticates with a username and password. The gateway looks up your account, checks which country and rotation mode you asked for, and then picks an exit IP from the pool for each connection. The username is a config knob: most providers let you encode the geo and rotation behavior in the username string, so user-country-us-session-30m and user-country-de-rotate hit the same gateway but get routed to different parts of the pool.

Rotation rules. Two modes, and you must understand the difference because it determines both your cost and your behavior. In rotating mode, the gateway hands you a new IP per request (or per N seconds). This is what most people picture when they say "rotating proxies," and it's right for stateless bulk fetches where you want the per-IP counters to never build up. In sticky mode, the gateway pins you to one IP for a session you define — 1 minute, 30 minutes, an hour — and keeps you on it until it dies. Sticky is right for anything with state: a login flow, a cookie jar, a paginated session where a new IP mid-stream would look like someone else took over your browser.

Why per-GB. Here's the business reality nobody says out loud: your bytes physically traverse other people's broadband connections. That costs the provider money per byte, so you're billed per byte. There is no such thing as an "unlimited" residential plan, and any provider advertising one is lying about something. It's also why a misconfigured scraper that fetches images and assets instead of just HTML will blow through a residential balance in hours — you're paying real money for every kilobyte that flows through someone's home connection.

One more thing worth knowing: because the pool is finite and shared, heavy use degrades quality. When thousands of customers hammer the same pool, the good IPs get burned out and the provider has to serve you worse ones. A provider's "pool size" number on a marketing page is a ceiling, not a guarantee. In practice what matters is how many unburned IPs they can hand you in the region you care about at the hour you're scraping.

The consent model is the industry's soft spot and, increasingly, your legal exposure. The people whose connections you're using opted in through an app or a browser extension, but the quality of that consent varies a lot. Reputable providers use explicit opt-in apps where users understand they're contributing bandwidth in exchange for a "free VPN" or a points perk. Less reputable ones bury it in SDKs that ride along in software nobody reads, and their IPs come from devices that never meaningfully agreed to carry other people's traffic. You often can't tell which you're buying, which is why the provider's consent story belongs in your due diligence — the checklist below treats it as a first-class item rather than an afterthought.

From your side of the connection, the practical rule is proportionality: the more personal the data you route through someone else's home connection, the more uncomfortable the arrangement should make you. Routing public product pages through a residential pool is one thing — the exit device sees only a few anonymous HTTP requests. Routing account logins, session cookies, or personal data through a consent-adjacent pool is a different animal: you're transferring risk onto a population of users who signed up to share bandwidth, not to be the middle hop in your credential traffic. Keep the scrape public and the traffic boring, and the ethics stay manageable.

What proxies fix and what they don't

This is the section that saves you money, so read it slowly.

Modern bot detection is a stack, and it's evaluated roughly in the order below. Here's the thing most proxy vendors will never tell you: the IP layer is deep down in that stack, and by the time a sophisticated detector routes your request by IP reputation, it has already scored your TLS handshake, your HTTP/2 settings, and your headers. The detection happens before IP reputation even gets a vote.

Layer	What it inspects	Does a proxy fix it?
TLS handshake (JA3/JA4)	cipher + extension order	No
HTTP/2 SETTINGS frame	header table size, window, streams	No
Header order + casing	the order headers arrive	No
User-Agent string	the UA you claim	No
Behavioral	JS execution, mouse, timing, scroll	No
Per-IP fingerprint correlation	does this fingerprint show up on other IPs?	Partial
IP reputation / ASN	hosting range? flagged?	Yes
Per-IP rate limits	how many requests from this IP?	Yes
Geo-restriction	is this IP in the allowed country?	Yes

That "Partial" row is subtle and worth unpacking. CDNs like Cloudflare correlate: they see a specific TLS fingerprint hammering one IP, then the same fingerprint shows up on fifty other IPs in rapid succession. The pattern is itself a bot signal. A residential proxy breaks the per-IP correlation — each IP only sees a few requests — but it does nothing about the fingerprint those requests carry. If your client's fingerprint is wrong, rotating IPs just makes you a distributed bot with a consistent fingerprint, which is exactly the correlation the detector is looking for.

Everything a proxy can fix lives at the bottom of the detection stack, and it's evaluated after the fingerprint layers on top. If your client's TLS and HTTP/2 fingerprint is wrong, a proxy is buying you nothing but IP diversity.

Here is the practical consequence, stated as a rule: proxies change who you are at the IP layer, and nothing else. Every layer above the IP layer — TLS, HTTP/2, headers, behavior, CAPTCHAs — is your client's job. If you're blocked because your Python requests TLS fingerprint is a dead giveaway, buying 10,000 residential IPs doesn't move the needle. It just means you're now getting 403s from 10,000 different IPs.

The single biggest proxy mistake in 2026 is buying residential proxies to fix a fingerprint problem. The diagnosis is free: hit the target with a browser-accurate client like curl_cffi (which reproduces Chrome's JA3 and HTTP/2 settings) on your plain home IP. If you still get blocked, the block is fingerprint-based or behavioral, and proxies won't fix it. If curl_cffi sails through on your home IP and your old client gets 403s, you have an IP-reputation or rate-limit problem, and that is what proxies are for.

And CAPTCHAs deserve special mention, because vendors love to sell proxies as the CAPTCHA solution. A CAPTCHA is a symptom, not a cause. You triggered one because something upstream — fingerprint, behavior, IP reputation — made you look like a bot. A fresh IP sometimes sidesteps the trigger because the new IP has no history. But if your fingerprint and behavior are bot-like, the CAPTCHA comes back on the residential IP just as fast. The cheapest CAPTCHA solver is not triggering one.

How to find which layer is actually blocking you

The four-step diagnostic loop below is how you keep a proxy budget honest. It isolates which layer of the stack is failing before you spend a dollar, and it takes about fifteen minutes.

Fix the client first. Hit the target with curl_cffi impersonating Chrome from your plain home IP. If it returns 200, your client is fine and any remaining block is IP-based — a proxy can help. If it 403s, the block lives in the fingerprint, the headers, or a JS challenge, and no proxy fixes that.
Prove the fingerprint is the variable. For a site that passed step one with curl_cffi but fails with your old requests client, keep your IP constant and swap only the client. Same IP, different fingerprint, different outcome: that's an airtight fingerprint experiment.
Add the proxy with everything else held constant. When you introduce IP rotation, change nothing else — same client, same headers, same pacing. Compare success rates. If the proxy moves you from 0% to 90%, the block was IP-layer and the money is earned.
Read the result honestly. If the proxy moves you from 0% to 5%, you bought IP diversity for a fingerprint block. Stop buying IPs and fix the client.

Every time a proxy vendor tells you "you need more IPs," run this loop first. In my experience, the majority of scrapers that reach for residential proxies skip step one entirely and end up paying residential prices for a problem the client could have fixed for free.

Here's what the loop's "it's IP-based" branch looks like when it's true. When a site rate-limits per IP, a static IP follows a predictable decay: success holds, then the counter trips, then the IP is effectively dead for the crawl. Rotation's whole value is flattening that decay — each fresh IP restarts the counter. The graph below is illustrative, but the shape is exactly what you'll measure on a per-IP-limited target:

When a block is per-IP, rotation is the difference between a crawl that dies at request 40 and one that runs all night. When the block is fingerprint-based, both lines look like the orange one and rotation is money spent on nothing.

When you genuinely need residential proxies

Residential proxies are worth the money in exactly these situations:

Scale past a handful of IPs. When you're moving tens of thousands of pages a day from one site, the per-IP rate limits become the binding constraint, and a datacenter IP or two won't hold. This is the honest core use case.
Geo-targeted scraping. Country-specific search results, regional pricing, localized storefronts. You need exit IPs physically in the target country, and residential pools have the country/city coverage that datacenter providers mostly don't.
Search-engine scraping. Google and Bing enforce strict per-IP query limits — ballpark a few hundred SERP requests per IP per day before challenges ramp up. The only way to run large SERP crawls is thousands of IPs, and the IPs need to look residential because Google aggressively flags hosting ranges.
Sites that block all hosting ASNs. Plenty of targets run a blanket rule: drop any request whose ASN is a cloud provider. Datacenter proxies never work there, full stop.
Bypassing per-IP rate limits on large crawls. Same mechanism as scale, but the trigger is explicit 429s on one IP. When you measure that the 429s follow your IP and not your fingerprint, rotation is the fix.

And here's the "you don't need them" list, which is where most people actually live:

Small crawls. A few thousand pages a day from one site fits fine on one or two datacenter IPs, or on your own IP with polite pacing. Paying residential prices for that is burning money.
Your own target or your own API. You control the rate limits. Just be polite.
Most public APIs. Rate limits are keyed to your API key, not your IP. Rotating IPs doesn't help and can get your key banned for abuse.
Any block that is fingerprint-based. Diagnosed above. Proxies won't fix it, so they're wasted money.
Anything behind a login. A proxy gets you a new IP; it doesn't get you credentials. If the content requires an account, the account is your bottleneck, not the IP.

The escalation rule I actually use: start on my own IP with a clean client. When I measure an IP-related block (hosting-ASN rejection, per-IP rate limiting, geo mismatch), step up to datacenter. Only when datacenter fails on IP grounds do I pay for residential. Mobile is the step after that, and I've taken it maybe twice in the last three years. Each escalation should be justified by a block you can point at, not by marketing.

A worked example: 100,000 SERP results a day

Let me make the decision concrete with the case where residential proxies genuinely earn their keep: a price-comparison tool that needs 100,000 US-targeted search results per day.

Google allows roughly a few hundred organic queries per IP per day before challenges ramp up, so 100,000 results needs on the order of 300-600 distinct IPs per day, and they need to look residential because Google flags hosting ASNs on sight. At 0.3MB per SERP response, that's about 30GB of traffic a day, call it 900GB a month. At a $6/GB volume price, the residential bill lands near $5,400 a month.

Now run the same job through a SERP API at $0.003 per result: 100,000 results a day is $300 a day, $9,000 a month. The proxy build genuinely beats the API here — but notice exactly why. The IPs were the scarce resource, the volume was high enough to amortize the engineering, and the block was provably IP-based. Drop any one of those conditions and the answer flips. Halve the volume and the API wins. Make the target a site that doesn't per-IP limit and your own IP wins. That's the discipline: the "right" answer is a function of three numbers, not a brand preference.

Integrating proxies into your scraper

The integration is boring plumbing, which is exactly how it should be. Five patterns cover 99% of real scrapers.

Pattern 1: `requests` with a proxy dict

requests takes a proxy per request via a dict:

import requests

proxies = {
    "http": "http://user:pass@res.provider.com:4444",
    "https": "http://user:pass@res.provider.com:4444",
}

resp = requests.get("https://example.com/", proxies=proxies, timeout=15)
print(resp.status_code)

The user:pass in the URL is the auth. With most providers, the username encodes rotation behavior — user-country=us-session=30m gives you a sticky US IP for 30 minutes, user-country=us-rotate gives you a new US IP per request. Read your provider's username format; it's the control surface for the whole product.

Pattern 2: a proxy pool with session rotation

For a bulk crawl you want bounded IP churn, not a new IP per request. A small pool class with a sticky window keeps you sane:

import itertools
import requests

class ProxyPool:
    """Round-robins proxies, holding one for `sticky` requests at a time."""

    def __init__(self, proxies, sticky=25):
        self._cycle = itertools.cycle(proxies)
        self._sticky = sticky
        self._current = None
        self._used = 0

    def next(self):
        if self._current is None or self._used >= self._sticky:
            self._current = next(self._cycle)
            self._used = 0
        self._used += 1
        return {"http": self._current, "https": self._current}

pool = ProxyPool(
    [f"http://user-{i}:pass@res.provider.com:4444" for i in range(50)],
    sticky=25,
)

for url in urls:
    resp = requests.get(url, proxies=pool.next(), timeout=15)
    handle(resp)

Fifty distinct usernames against a rotating gateway give you fifty different IPs; holding each for 25 requests gives you ~1,250 requests before the cycle repeats. That's plenty of diversity for a mid-size crawl without looking like a tornado of one-request IPs.

Pattern 3: `curl_cffi` with a proxy

When the target fingerprint-checks, pair the proxy with a browser-accurate client. This is the combination that actually works on protected sites:

# pip install curl_cffi
from curl_cffi import requests

resp = requests.get(
    "https://protected.example.com/search?q=widgets",
    impersonate="chrome",   # Chrome's JA3 + HTTP/2 settings
    proxies={"https": "http://user-country=us-rotate:pass@res.provider.com:4444"},
    timeout=20,
)
print(resp.status_code, resp.text[:200])

The order matters: get the fingerprint right first, then add the proxy. If you do it the other way around you'll never know which one fixed it.

Pattern 4: Playwright with a proxy per context

Playwright puts the proxy at the browser context level. One context, one IP. New context, new IP:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)

    context = browser.new_context(
        proxy={
            "server": "http://res.provider.com:4444",
            "username": "user-session-1",
            "password": "pass",
        },
        user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                   "AppleWebKit/537.36 (KHTML, like Gecko) "
                   "Chrome/126.0.0.0 Safari/537.36",
    )
    page = context.new_page()
    page.goto("https://target.com", wait_until="domcontentloaded")

    # A different IP is a different context with a different username.
    context2 = browser.new_context(
        proxy={
            "server": "http://res.provider.com:4444",
            "username": "user-session-2",
            "password": "pass",
        }
    )

Notice the sticky-session usernames: user-session-1 and user-session-2 hold different IPs for the life of the context, which is what you want for a browser that maintains cookies and state. If you rotate IPs mid-session, cookies and login state follow the old IP and you silently break the flow.

Pattern 5: testing a proxy before you trust it

Never wire a proxy into a production crawl untested. A 20-line check tells you the basics — IP, geo, latency, and whether it's leaking:

import json, time, requests

def test_proxy(proxy_url):
    proxies = {"http": proxy_url, "https": proxy_url}
    start = time.monotonic()
    try:
        r = requests.get("https://api.ipify.org?format=json",
                         proxies=proxies, timeout=10)
        data = r.json()
        latency_ms = (time.monotonic() - start) * 1000
    except Exception as e:
        return {"ok": False, "error": str(e)}

    direct = requests.get("https://api.ipify.org?format=json", timeout=10).json()

    return {
        "ok": True,
        "ip": data.get("ip"),
        "country": data.get("country", "?"),
        "latency_ms": round(latency_ms, 1),
        "leaking_my_ip": data.get("ip") == direct.get("ip"),
    }

print(json.dumps(test_proxy("http://user:pass@res.provider.com:4444"), indent=2))

Check the IP is different from your direct connection, the country matches what you paid for, and latency is in the expected ballpark. For fingerprint-level verification, hit the actual target with curl_cffi and compare status codes with and without the proxy.

Rotation strategy: per-request vs sticky

The strategy question is really a question of what you're faking. Per-request rotation is the "be nobody in particular" strategy: no single IP ever accumulates enough requests to trip a counter. It's right for stateless bulk fetches where each request stands alone. Sticky is the "be one consistent person" strategy: same IP, same cookies, same session, normal-looking behavior. It's right for search sessions, logins, and any flow with state.

Here's the anti-pattern: rotating aggressively as a general policy. A scraper that shows up with a fresh IP every single request is itself a bot signal, because no human's IP changes every 300 milliseconds. One request per IP per second is aggressive; one IP per request is a banner that says "scraper." If a site allows a few requests per IP, let each IP do its full allowance before moving on. Rotate on demand — when you hit a 429 or a block — not for the aesthetic.

Cost and capacity math

Now the part where most proxy purchases die: the actual numbers.

Residential bandwidth is metered and expensive, so the question "how many pages can I scrape for my money" reduces to "how many bytes does one page cost you."

A SERP page is roughly 100-300KB of HTML. A typical product or article page is 200-500KB. If you fetch HTML only — no images, no CSS, no JS — you can call it 0.3MB per page on average and you won't be far off. At that rate:

1 GB of residential bandwidth ≈ 3,000-3,500 HTML pages.
At the common $10/GB price, that's about $0.003 per page.
A million pages ≈ 300 GB ≈ $3,000/month, before any volume discount.

The moment someone's scraper fetches images and assets through a residential proxy, the math collapses. A page with a few images is 1-3MB, so 1 GB now buys 300-1,000 pages and your cost per page triples. The fix is boring and essential: route only the HTML fetches through the proxy, or strip assets with a content filter. Paying residential rates for a target's JPEGs is the most expensive mistake in this entire post.

A few pricing-model traps to watch for:

Per-IP pricing hides the bandwidth cost. "500 IPs for $50/month" is a real product, but you still pay for bytes on top. The IPs are cheap; the GBs are the product.
Geo multipliers. US and EU IPs are baseline. Premium geos (Japan, Korea, some of Western Europe) can be 2-5x the base per-GB rate.
Concurrency caps. Many plans limit concurrent connections. A bursty scraper that opens 100 connections will hit the cap and get throttled even with bandwidth to spare.
"Premium" tiers. Providers sell "premium residential" pools with cleaner IPs at a 1.5-3x markup. Sometimes worth it, often a way to double the bill for IPs that were already fine.

Here's the build-vs-buy crossover that decides most projects, and it's not the one vendors want you to see. Residential proxies at $0.003/page beat a scraping API only if the API's per-result fee is higher. A SERP API runs roughly $0.002-0.005 per result. So for raw per-page cost, residential proxies are competitive with APIs — but an API includes the parsing, the CAPTCHA solving, the browser rendering, and the engineering that goes into not getting blocked, which proxies do not.

Below roughly 1.3M pages a month, a scraping API is cheaper than residential proxies before you count a single hour of engineering. Datacenter proxies stay cheap because their per-GB price is a fraction of residential, which is why the escalation ladder starts there.

Add engineering to that chart and the crossover moves hard. Building and operating your own residential-proxy pipeline — pool management, retries, CAPTCHA handling, fingerprint hygiene, geo balancing, monitoring — is realistically two engineer-months to start and a permanent maintenance tax. At a fully-loaded engineer cost of $10-15K a month, that's $20-30K of engineering to save on per-page fees that only exceed the API's fee once you're past a few million pages a month. The honest build-vs-buy answer for most teams: under a few million pages a month, buy the API. The proxy pipeline only wins at the scale where you have a team whose job is the pipeline anyway.

The build-vs-buy decision rule, distilled

Here is the rule I actually apply when a project lands on my desk. Compute your effective per-result cost three ways — datacenter proxies, residential proxies, and the API — at the volume you realistically expect in the first year, not the volume on the slide deck. Then add engineering to whichever path you'd build yourself: two engineer-months to stand up the pipeline, plus a permanent maintenance tax of 10-20% of an engineer's month, every month, forever.

Under a few million results a month, the API wins on total cost almost every time, because the per-result difference between residential and the API is small — a couple of tenths of a cent — while the engineering difference is an order of magnitude. Past a few million results a month, the build wins on unit economics, but only if your team is already in the pipeline business and will still be maintaining it next year. And always keep the third option on the table: polite, low-rate scraping from your own IP costs nothing and is the correct answer for the majority of small jobs. The most expensive proxy is the one bought to solve a problem that was never IP-based, and the second most expensive is the pipeline built at a volume that never materialized.

Evaluating a provider and avoiding the scams

If you do need residential proxies, the quality difference between providers is enormous, and it's not visible on the marketing page. Here's what actually matters:

Uptime and exit quality. The real product is "does a request through a random pool IP succeed and return the right geo." Buy a small package and measure: success rate per 1,000 requests, time-to-first-byte, and how often exit IPs fail mid-request. A provider with 95%+ success on your target is worth 2x the price of one at 85%.
Pool size that you can verify. "60M IPs" is a number on a page. What you can verify is how many distinct, unburned IPs you actually see in the region you need, and how they degrade under your load pattern. Run your real workload for a day on the smallest plan and look at IP diversity per 10,000 requests.
Rotation control. Do they give you per-request rotate, sticky sessions with a time window you set, and country/city/ASN targeting in the username? If rotation is a black box, you can't implement the strategies above.
Pricing model transparency. Per-GB, per-IP, geo multipliers, concurrency caps, minimum commits. Any provider that can't tell you their effective per-GB price in one sentence is hiding something.
Support that answers at your hours. You will hit a 3am "all exits in Germany are dead" moment. Find out if support is there before you need it.
The consent story. A provider that can describe how their pool is assembled — opt-in, disclosed, region-appropriate — is one you can defend using. A provider that's vague about where the IPs come from is a legal risk you inherit. This matters more as anti-scraping law sharpens.

And the scams. The free-proxy ecosystem is not "low quality," it's hostile: a large share of public proxy lists are run by defenders specifically to catch scrapers, and a large share of the rest are run by people monetizing your traffic, injecting ads, or intercepting credentials. A free proxy list is the one place where "it's free" and "it's actively against you" are the same sentence.

The share of free proxies that is already blacklisted, run against you, or actively hostile dwarfs the share worth using. The "2% that works" is still a security risk you can't audit. This is the one place the paid option is also the safety option.

If you must test cheap proxies, treat them like untrusted network traffic: never send them credentials, never route anything logged-in through them, keep them in a sandboxed process, and expect them to fail. The realistic answer is that the cost of a small residential package is lower than the debugging time of a free proxy stack, and that's before you value your credentials.

When to skip proxies entirely

Let me end where the vendors won't: with the list of cases where the correct answer is no proxy at all.

Skip proxies entirely for small crawls — anything under a few thousand pages a day from one site. Your own IP with a polite client, a cache, and jittered delays handles that, and it's free. Skip them for APIs and your own targets, where rate limits are keyed to keys and accounts, not IPs. Skip them for any target whose block is fingerprint-based, because the proxy is buying you nothing. And skip them whenever the content is behind a login, because the account is the bottleneck.

When you do need IP diversity, climb the ladder one rung at a time: datacenter first, residential only on a measured IP-level block, mobile almost never. And always, always pair any proxy with a browser-accurate client — the fingerprint is the price of admission in 2026, and the proxy is just your seat.

The honest summary fits in one sentence: residential proxies are an IP-layer tool, priced like a silver bullet, and they only pay for themselves when IP-layer blocking is what's actually hurting you. Diagnose the block before you buy the fix. Most proxy bills are spent on the wrong layer.

Key takeaways

A proxy changes your source IP and nothing else. Everything it can fix lives at the IP layer of the detection stack.
Datacenter proxies are cheap, fast, and easily flagged; residential are expensive, slower, and hard to flag; mobile are cleanest and priciest. Climb the ladder one rung at a time.
Detection reads TLS, HTTP/2, and headers before it routes by IP. Proxies do not fix a bad fingerprint — pair them with curl_cffi or a stealthed browser.
You need residential proxies only for scale, geo-targeting, search engines, and per-IP rate-limit bypass — and you should be able to point at the block that justifies them.
Use sticky sessions for anything with state; use per-request rotation sparingly, because aggressive rotation is itself a fingerprint.
Residential billing is per-GB for a reason: your bytes traverse real people's connections. Fetch HTML only, or the asset traffic will triple your bill.
Below roughly a few million pages a month, a scraping API is cheaper than a residential-proxy pipeline once you count engineering. Build only at scale.
Free proxies are a hostile environment, not a bargain. Test every paid proxy before you trust it, and verify IP, geo, latency, and leaks.

Frequently Asked Questions

Do I need residential proxies for web scraping?

Probably not at first. Residential proxies earn their cost when you scrape at scale (tens of thousands of pages a day from one site), need geo-targeted IPs, or hit a site that blocks hosting ranges. For a small crawl of a few thousand pages, a datacenter proxy or no proxy at all is the right answer. Buy residential only after you can point at a measured IP-related block.

What's the difference between residential, datacenter, and mobile proxies?

Datacenter proxies come from cloud and hosting providers: cheap, fast, and easy to flag because the IP ranges are known. Residential proxies are real ISP customers' connections: expensive, slower, and hard to flag at the IP level. Mobile proxies are carrier IPs, the cleanest reputation and the highest price. Start with datacenter. Escalate only when IP reputation is your actual blocker.

Why do I still get blocked with residential proxies?

Because the block isn't at the IP layer. Detection reads your TLS fingerprint (JA3/JA4), HTTP/2 settings, and header order before it ever routes by IP. A residential IP fixes your IP reputation, not your client fingerprint. Pair proxies with a browser-accurate client like curl_cffi or a stealthed browser, or the proxy money is wasted.

How much do residential proxies cost?

Roughly $5-15 per GB in 2026, with volume discounts above 100 GB/month. One GB of HTML-only fetches is roughly 2,000-4,000 pages. At $10/GB that's about a third of a cent per page. Static residential IPs billed per IP run $4-10 per IP per month. Mobile is $15-40 per GB.

Can I use free proxies for scraping?

No. Free proxy lists are dominated by endpoints that are already blacklisted, run by defenders as honeypots, or injecting malware and ads. The ones that work are slow, unreliable, and likely being sold to the site you're scraping. A free proxy costs more in debugging time than a paid one costs in money.

How do proxies fix IP-based rate limits?

A rate limit is a counter keyed to your source IP. A proxy changes your source IP, so the counter resets. Rotating residential proxies spread requests across thousands of IPs, so no single IP trips the limit. That only helps if the block is actually per-IP. If the site limits by fingerprint or account, a new IP doesn't help.

What is sticky vs rotating residential proxy?

Rotating mode assigns a new IP per request (or per short time window), which spreads load but looks erratic. Sticky mode holds one IP for the duration of a session — typically 1-30 minutes — which looks more like a normal user and is required for login flows and cookies. Use sticky for sessions, rotating for stateless bulk fetches.

Are residential proxies legal?

They're a tool, not inherently legal or illegal. Legal risk comes from what you scrape and how you use it, not the proxy type. The catch with residential is that the IPs belong to real people who consented, often via an app's fine print, and some providers' consent models are shakier than others. Check the target's ToS, respect robots.txt, and don't route stolen credentials or personal data through someone's home connection.

How do I test whether a proxy is working?

Fetch a service like api.ipify.org through the proxy and check the IP, country, and latency, then compare the IP to your direct connection. A working proxy shows a different IP that matches the region you paid for. To check detection, hit the target once with and once without the proxy and compare status codes. A proxy that still gets 403 with a browser-accurate client means the block isn't IP-based.

Datacenter or residential proxies for my use case?

Rule of thumb: start with datacenter. If the target blocks hosting ASNs or rate-limits your IP, move to residential. If the target blocks your residential IPs too, the problem is your fingerprint, not your proxy — fix the client first. Mobile is the last resort and should be rare.

Keep reading

Found this useful? Cite it as: webscraping.space. “Residential Proxies for Web Scraping: The Honest Guide.” https://webscraping.space/blog/residential-proxies-for-scraping. Published 2026-08-08.