Scaling Published Aug 14, 2026 · 34 min read · 7,519 words
The Complete Proxy Guide for Web Scraping (2026)
The honest 2026 proxy guide: what a proxy does, the four types compared, rotation, detection, real pricing, cost math, when you need one, and working Python code.
Every proxy vendor's homepage sells the same fantasy: rotate an IP per request and you'll never be blocked again. It's the most oversold claim in the scraping industry, and it costs people real money — thousands of dollars a month for bandwidth they don't need, solving a block that was never IP-based.
This guide is the honest version. I run scrapers in production for a living, and I've bought proxies from most of the major providers. Here's what I've learned: a proxy changes which IP address the server sees. That's the entire mechanism. Everything else a proxy marketer tells you is either a corollary of that fact or a fantasy built on top of it.
What follows is the complete 2026 field guide: what a proxy physically does, the four proxy types (datacenter, residential, mobile, ISP) with honest pros and cons, how rotation actually works, how proxies get detected, the real cost math (per-GB vs per-request), when you genuinely need proxies and when you're burning cash, a decision framework for picking one, and a working Python example. Where the honest answer is "you don't need a proxy," I'll say so. Where the honest answer is "yes, and here's the exact number that justifies it," I'll show the math.
The site's ground rules apply throughout: prefer an official API or dataset when one exists, respect robots.txt, rate-limit politely, and don't collect personal data you don't need. Everything here assumes public, ToS-aware, educational use.
Key takeaways
- A proxy changes your source IP and nothing else. It fixes IP reputation, per-IP rate limits, and geo restrictions — and nothing above the IP layer of the detection stack.
- Four types, a strict ladder of price and cleanliness: datacenter (
$0.5-2/GB, easily flagged), ISP/static (per-IP, sticky), residential ($4-15/GB, hard to flag), mobile ($15-40/GB, cleanest, rarely worth it). - Detection reads TLS, HTTP/2, and behavior before it routes by IP. A proxy does not fix a bad fingerprint. Pair any proxy with a browser-accurate client like
curl_cffi. - Rotation is a per-IP counter reset, not a magic shield. Rotating mode spreads load; sticky mode holds one IP for stateful sessions. Aggressive rotation is itself a bot signal.
- You need proxies only for scale, geo-targeting, search engines, and per-IP rate-limit bypass — and you should be able to point at the block that justifies them.
- Per-GB pricing is a volume discount that flips at scale. Below roughly 10k-50k pages a month, a scraping API is cheaper than a proxy pipeline once you count engineering.
- Diagnose before you buy. The four-step loop in this guide isolates which layer is blocking you in about fifteen minutes, and it's saved more money than any provider discount.
What a proxy actually is
A proxy is a middleman for your TCP connection. Your client opens a connection to the proxy server, the proxy opens a second connection to the target, and bytes flow through the proxy in both directions. The target server terminates a connection that came from the proxy's IP. Your client never talks to the target directly, so the target has no way to learn your real IP.
That's it. There is no magic. A relay changes four things about your traffic:
- Your source IP. The one that matters. Rate-limit counters, IP reputation lookups, and geo logic all key on this address.
- Latency. Every relay adds a hop. Datacenter proxies add maybe 20-60ms. Residential proxies are worse — your bytes exit through a real home broadband or mobile connection, so figure 50-300ms, sometimes more. At a million requests, that's real wall-clock time.
- Reliability. You inherit every failure mode of a network you don't control. Exit IPs die mid-request, residential devices drop off the pool, providers throttle bandwidth. Your retry logic had better be good.
- Routing and reputation. The server doesn't just see an IP; it sees what network that IP belongs to. A datacenter IP advertises "this is a server" by its ASN. A residential IP advertises "this is a normal person at home."
Everything proxies are marketed on — bypassing Cloudflare, solving CAPTCHAs, evading fingerprinting — is downstream of the IP change and mostly doesn't follow from it. Keep the four-item list in your head. When someone promises you a proxy solution to a problem that isn't in that list, they're overselling you.
The four proxy types, honestly compared
There are four flavors of proxy, and they form a strict ladder of price and cleanliness: datacenter at the bottom, ISP/static in the middle, residential above it, mobile at the top.
Datacenter proxies. IPs carved out of cloud and hosting providers — AWS, DigitalOcean, Hetzner, and hundreds of smaller shops. Cheap: roughly $0.5-2 per GB, or $2-4 per IP per month for dedicated IPs, with plenty of flat-rate "unmetered" plans. Fast, because the whole path is data-center-to-data-center. The problem is flaggability: the IP ranges of every hosting provider are public knowledge, and threat-intel feeds list them as "hosting" or "ASN: cloud." Any site that does even basic IP reputation filtering can block an entire hosting ASN with one rule. Datacenter proxies are fine for targets that don't IP-filter at all, and they are the correct first stop for almost every scrape.
ISP / static residential proxies. A residential IP that's assigned to you persistently rather than pulled from a rotating pool. It behaves like a residential IP but holds still across sessions, which makes it the right tool for login flows, cookie persistence, and anything that needs to look like "the same person, coming back." Pricing is usually per IP per month — $4-10 — plus bandwidth. It's the least talked-about category and often the one you actually want.
Residential proxies. IPs from real ISP customers' connections, aggregated into a pool by the provider. Price: roughly $4-15 per GB in 2026, with volume discounts past 100GB/month. Bright Data lists around $4/GB; Oxylabs around $5/GB. Slower than datacenter and less reliable, because the exits are other people's devices. But the IP reputation is genuinely different: to a target server, a residential IP looks like a normal home user in the country you picked. No hosting-ASN blocklist flags it. That's the entire value proposition, and it's real — up to the limit of the fingerprint problem we'll get to.
Mobile proxies. IPs from cellular carriers, which come out of carrier CGNAT pools. These have the cleanest reputation of all — mobile IPs are rarely in blocklists, and the same public IP is shared by many real subscribers, so a server that "remembers" an IP can't cleanly tie it to one user. Price: $15-40 per GB, the most expensive category by a wide margin. Mobile is for the hardest targets only, and honestly, in most scrapes it's overkill. If your TLS fingerprint is clean and your behavior is sane, residential gets you there for a third of the price.
The honest framing: datacenter is cheap and flagged, ISP is sticky and mid-priced, residential is expensive and hard to flag, mobile is cleanest and most expensive — and each step up buys you a smaller increment of success while charging a bigger increment of money.
What proxies actually cost in 2026
Let me put real numbers on the ladder, because the price gap between types is the single most important fact in this guide. These are published July 2026 list prices, and they're the numbers I use in every cost conversation.
The per-GB framing matters because of what a GB buys. A typical HTML-only product page is 200-400KB. At 0.3MB per page, one GB is roughly 3,300 pages. So residential at $4/GB works out to about $0.0012 per page — a tenth of a cent. Datacenter at $1/GB is a quarter of that. The numbers only get scary when you fetch images, scripts, and assets instead of just HTML, or when you let a misconfigured scraper re-download the same page. A residential balance that should last a month can evaporate in an afternoon of asset traffic.
How proxies get detected
Before you spend money on IPs, you need to understand what the other side is actually looking at. Modern bot detection is a stack, and it's evaluated roughly in the order below. The IP layer is deep down in that stack — by the time a sophisticated detector routes your request by IP reputation, it has already scored your TLS handshake, your HTTP/2 settings, and your headers.
IP reputation. The oldest and crudest signal. Threat-intel feeds classify IP ranges by ASN: hosting providers, known proxy services, VPN endpoints, and Tor exits are all flagged. A datacenter IP advertises its nature by its ASN, which is why a single rule can block an entire cloud provider. Residential and mobile IPs don't carry that flag — they look like normal consumers. This is the only layer a proxy fully controls.
TLS fingerprinting. Every HTTP client negotiates TLS with a specific set of ciphers, extensions, and ordering. That combination — JA3, and its successor JA4 — is a fingerprint that identifies the library you're using. Python's requests has a TLS fingerprint that is instantly recognizable, and it's the same on every IP you route through. A WAF that fingerprints TLS doesn't care how many IPs you rotate; it sees the same bot client on all of them. This is the layer that defeats most proxy purchases in 2026.
HTTP/2 and header fingerprinting. The HTTP/2 SETTINGS frame — header table size, window, stream concurrency — plus the order and casing of your headers, form a second fingerprint. Browsers send headers in a specific order; libraries send them in another. Same story as TLS: it follows your client, not your IP.
Behavioral detection. Request rate, timing, mouse movement, scroll, and the pattern of what you fetch. A scraper that hits 50 pages a second with metronomic regularity is a bot no matter what IP it comes from. This is the layer that the web scraping without getting blocked post owns in depth.
Here's the practical consequence, stated as a rule: proxies change who you are at the IP layer, and nothing else. Every layer above the IP layer is your client's job. If you're blocked because your requests TLS fingerprint is a dead giveaway, buying 10,000 residential IPs doesn't move the needle. It just means you're now getting 403s from 10,000 different IPs.
The single biggest proxy mistake in 2026 is buying residential proxies to fix a fingerprint problem. The diagnosis is free: hit the target with a browser-accurate client like curl_cffi (which reproduces Chrome's JA3 and HTTP/2 settings) on your plain home IP. If you still get blocked, the block is fingerprint-based or behavioral, and proxies won't fix it. If curl_cffi sails through on your home IP and your old client gets 403s, you have an IP-reputation or rate-limit problem — and that is what proxies are for.
How rotation works: sticky vs rotating
Rotation is the feature every vendor leads with, and it's the one most people misunderstand. Here's the plumbing, then the decision.
Your scraper doesn't connect to the provider's IP pool directly. It connects to the provider's gateway — a hostname and port you put in your client's proxy config — and authenticates with a username and password. The gateway looks up your account, checks which country and rotation mode you asked for, and picks an exit IP from the pool for each connection. The username is a config knob: most providers let you encode the geo and rotation behavior in it, so user-country-us-session-30m and user-country-de-rotate hit the same gateway but get routed to different parts of the pool.
Two rotation modes, and the difference determines both your cost and your behavior:
Rotating mode hands you a new IP per request (or per N seconds). This is what most people picture when they say "rotating proxies," and it's right for stateless bulk fetches where you want the per-IP counters to never build up. The catch: aggressive rotation is itself a signal. A detector that sees a consistent TLS fingerprint hammering fifty different IPs in rapid succession has found a distributed bot — the correlation is the tell.
Sticky mode pins you to one IP for a session you define — 1 minute, 30 minutes, an hour — and keeps you on it until it dies. Sticky is right for anything with state: a login flow, a cookie jar, a paginated session where a new IP mid-stream would look like someone else took over your browser. Sticky also costs less in practice, because you're not burning a fresh IP on every request.
Session control is the knob that makes sticky work. Most providers let you set the session length in the username — user-country-us-session-30m holds one IP for 30 minutes, user-country-us-session-1h for an hour — and some let you pin a specific session ID so the same IP is reused across separate requests from different processes. That's the pattern for anything where the target keys state to your IP: a search session, a paginated listing, a checkout flow. The session dies when the IP dies, so your retry logic should treat a dead session as a fresh start, not a continuation. And a practical warning: a session that outlives its usefulness is a session that gets rate-limited. If a sticky IP starts returning 429s, rotate it — holding a burned IP "for consistency" is how people turn a working session into a blocked one.
The cost math: per-GB vs per-request
The pricing-model question — per-GB bandwidth vs per-request API — is where most proxy decisions actually get made, and it's a crossover, not a preference. Here's the honest math.
A per-GB proxy charges you for bytes. The cost per page therefore scales with page size: a 0.1MB page costs a tenth of a 1MB page. A per-request scraping API charges a flat rate per page regardless of size. So the crossover is a function of your average page size and the API's price.
At $4/GB (Bright Data residential), 1,000 pages of 0.3MB each cost $1.20. The same 1,000 pages through ScraperAPI at $1.10/1k cost $1.10 — the API is cheaper. But 1,000 pages of 1MB each cost $4.00 through the proxy and still $1.10 through the API — wait, that's wrong. Let me redo this.
Per-GB: 1,000 pages × 0.3MB = 300MB = 0.3GB. At $4/GB that's $1.20. At 1MB per page, 1,000 pages = 1GB = $4.00. Per-request: ScraperAPI $1.10/1k flat. So the proxy beats ScraperAPI only when pages are large enough that $4/GB × size < $1.10/1k, i.e., size < 0.275MB. For pages under ~0.28MB, the API is cheaper; above that, the proxy is cheaper. And against a flat $0.25/1k API like Keirolabs, the proxy loses at every realistic page size — $4/GB only beats $0.25/1k when pages are under 0.06MB, which is almost nothing.
But cash per page is only half the math. The other half is engineering. A proxy pipeline is infrastructure: you write the fetcher, the retry logic, the rotation client, the parser, and you maintain all of it when targets change. A scraping API bundles fetch, render, proxy, and parse into one call and moves the maintenance to the provider's team. The web scraping APIs post does the full provider roundup; the decision rule here is simple: below roughly 10k-50k pages a month, the API is cheaper than your time. Above it, a self-built proxy stack starts to pay — unless your targets are protected, which shoves the crossover back toward buying.
A worked example: 100,000 pages a day
Let me make the crossover concrete with the case where a proxy pipeline genuinely earns its keep. Say you run a price-comparison tool that needs 100,000 US-targeted product pages a day, and the target rate-limits per IP at a few hundred requests before serving 429s.
At 0.3MB per page, 100,000 pages is 30GB of traffic a day, call it 900GB a month. At Bright Data's ~$4/GB volume price, the residential bill lands near $3,600 a month. Add a small box to run the fetcher — $50-100 — and you're at roughly $3,700 a month for the whole pipeline, plus the engineering you already wrote.
Now run the same job through a per-request API. At ScraperAPI's $1.10/1k, 100,000 pages a day is $110 a day, $3,300 a month — about the same, before you count that the API also handles rendering, retries, and anti-bot. At Keirolabs' flat $0.25/1k, the same volume is $25 a day, $750 a month, and the proxy pipeline loses on cash alone.
The honest conclusion: the proxy build wins only when the block is provably IP-based, the volume is high enough to amortize the engineering, and the per-page price of the API is high. Drop any one of those conditions and the answer flips. Halve the volume and the API wins. Make the target a site that doesn't per-IP limit and your own IP wins. That's the discipline: the "right" answer is a function of three numbers, not a brand preference.
When you actually need proxies
Proxies are worth the money in exactly these situations:
- Scale past a handful of IPs. When you're moving tens of thousands of pages a day from one site, per-IP rate limits become the binding constraint, and one or two datacenter IPs won't hold. This is the honest core use case.
- Geo-targeted scraping. Country-specific search results, regional pricing, localized storefronts. You need exit IPs physically in the target country, and residential pools have the country/city coverage that datacenter providers mostly don't.
- Search-engine scraping. Google and Bing enforce strict per-IP query limits — ballpark a few hundred SERP requests per IP per day before challenges ramp up. Large SERP crawls need thousands of IPs, and the IPs need to look residential because search engines aggressively flag hosting ranges.
- Sites that block all hosting ASNs. Plenty of targets run a blanket rule: drop any request whose ASN is a cloud provider. Datacenter proxies never work there, full stop.
- Bypassing per-IP rate limits on large crawls. Same mechanism as scale, but the trigger is explicit 429s on one IP. When you measure that the 429s follow your IP and not your fingerprint, rotation is the fix.
And here's the "you don't need them" list, which is where most people actually live:
- Small crawls. A few thousand pages a day from one site fits fine on one or two datacenter IPs, or on your own IP with polite pacing. Paying residential prices for that is burning money.
- Your own target or your own API. You control the rate limits. Just be polite.
- Most public APIs. Rate limits are keyed to your API key, not your IP. Rotating IPs doesn't help and can get your key banned for abuse.
- Any block that is fingerprint-based. Diagnosed above. Proxies won't fix it, so they're wasted money.
- Anything behind a login. A proxy gets you a new IP; it doesn't get you credentials. If the content requires an account, the account is your bottleneck, not the IP.
The escalation rule I actually use: start on my own IP with a clean client. When I measure an IP-related block (hosting-ASN rejection, per-IP rate limiting, geo mismatch), step up to datacenter. Only when datacenter fails on IP grounds do I pay for residential. Mobile is the step after that, and I've taken it maybe twice in the last three years. Each escalation should be justified by a block you can point at, not by marketing.
How to pick a provider
If you do need proxies, the quality difference between providers is enormous, and it's not visible on the marketing page. Here's what actually matters:
- Uptime and exit quality. The real product is "does a request through a random pool IP succeed and return the right geo." Buy a small package and measure: success rate per 1,000 requests, time-to-first-byte, and how often exit IPs fail mid-request. A provider with 95%+ success on your target is worth 2x the price of one at 85%.
- Pool size you can verify. "60M IPs" is a number on a page. What you can verify is how many distinct, unburned IPs you actually see in the region you need, and how they degrade under your load pattern. Run your real workload for a day on the smallest plan and look at IP diversity per 10,000 requests.
- Rotation control. Do they give you per-request rotate, sticky sessions with a time window you set, and country/city/ASN targeting in the username? If rotation is a black box, you can't implement the strategies above.
- Pricing model transparency. Per-GB, per-IP, geo multipliers, concurrency caps, minimum commits. Any provider that can't tell you their effective per-GB price in one sentence is hiding something.
- Support that answers at your hours. You will hit a 3am "all exits in Germany are dead" moment. Find out if support is there before you need it.
- The consent story. A provider that can describe how their pool is assembled — opt-in, disclosed, region-appropriate — is one you can defend using. A provider that's vague about where the IPs come from is a legal risk you inherit.
And the scams. The free-proxy ecosystem is not "low quality," it's hostile: a large share of public proxy lists are run by defenders specifically to catch scrapers, and a large share of the rest are run by people monetizing your traffic, injecting ads, or intercepting credentials. A free proxy list is the one place where "it's free" and "it's actively against you" are the same sentence. If you must test cheap proxies, treat them like untrusted network traffic: never send them credentials, never route anything logged-in through them, keep them in a sandboxed process, and expect them to fail.
A working example: requests with a rotating proxy pool
Here's the pattern I actually run in production. It's deliberately boring: a proxy pool, a per-request pick, retries with backoff, and polite pacing. The rotation is handled by the provider's gateway — you just rotate which gateway URL you use, or let the gateway rotate for you via the username.
import random
import time
import requests
# One entry per gateway/account. The username encodes geo and rotation:
# "user-country-us-rotate" = US exits, new IP per request.
PROXY_POOL = [
"http://user-country-us-rotate:YOUR_PASSWORD@gateway.brightdata.com:22225",
"http://user-country-us-rotate:YOUR_PASSWORD@pr.oxylabs.io:7777",
]
HEADERS = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/126.0 Safari/537.36",
"Accept-Language": "en-US,en;q=0.9",
}
def fetch_with_proxy(url, retries=3, base_delay=1.0):
"""Fetch a URL through a random proxy, retrying with exponential backoff."""
for attempt in range(retries):
proxy = random.choice(PROXY_POOL)
proxies = {"http": proxy, "https": proxy}
try:
resp = requests.get(
url,
headers=HEADERS,
proxies=proxies,
timeout=20,
)
if resp.status_code == 200:
return resp
if resp.status_code in (429, 403):
# Rate-limited or blocked: back off, then try a fresh IP.
delay = base_delay * (2 ** attempt) + random.uniform(0, 0.5)
time.sleep(delay)
except requests.RequestException:
# Dead exit IP, gateway timeout, connection reset: retry.
time.sleep(base_delay * (2 ** attempt))
return None
def crawl(urls, delay=0.5):
"""Crawl a list of URLs with polite pacing and a shared session."""
session = requests.Session()
results = {}
for url in urls:
resp = fetch_with_proxy(url)
if resp is not None:
results[url] = resp.text
# Polite pacing: jittered delay so request timing isn't metronomic.
time.sleep(delay + random.uniform(0, delay))
return results
Three things to notice. First, the retry logic is not optional — exit IPs die mid-request and gateways time out, and a proxy pipeline without retries is a pipeline that silently loses data. Second, the pacing is deliberate: a jittered delay between requests is the difference between "looks like a person" and "looks like a loop." Third, this uses requests, which is fine for targets that don't fingerprint TLS. For anything behind a strict WAF, swap the client for curl_cffi with impersonate="chrome" — the proxy is your seat, but the fingerprint is the price of admission in 2026.
One more production note: fetch HTML only. If your scraper pulls images, scripts, and stylesheets through a per-GB residential proxy, you're paying real money for bytes you don't need. Parse the HTML and let the browser-less client skip the assets. This single habit cuts residential bills by 60-80% in my experience.
The ethics line
Proxies are a tool, and the ethics live in how you use them. The ground rules this site operates by apply with extra force when you're routing traffic through other people's connections: prefer an official API or dataset when one exists, respect robots.txt, rate-limit politely, and don't collect personal data you don't need.
The proxy-specific consideration is proportionality. Routing public product pages through a residential pool is one thing — the exit device sees only a few anonymous HTTP requests. Routing account logins, session cookies, or personal data through a consent-adjacent pool is a different animal: you're transferring risk onto a population of users who signed up to share bandwidth, not to be the middle hop in your credential traffic. Keep the scrape public and the traffic boring, and the ethics stay manageable. The web scraping ethics and robots.txt post is the full treatment.
FAQ
What is the best proxy for web scraping? There is no single best proxy — there's a best for your situation. Datacenter proxies are the right default for most scrapes: cheap, fast, and fine for targets that don't IP-filter. Residential proxies (Bright Data ~$4/GB, Oxylabs ~$5/GB) earn their cost only when you hit per-IP rate limits, need geo-targeted IPs, or scrape search engines. Mobile is the most expensive and almost never needed. Start with datacenter, escalate only when you can point at an IP-level block.
Residential vs datacenter proxies: which should I use? Datacenter proxies come from cloud and hosting providers: roughly $0.5-2/GB, fast, and easy to flag because hosting ASNs are blocklisted by default. Residential proxies are real ISP customers' connections: $4-15/GB, slower, and hard to flag at the IP level. Use datacenter first. Move to residential only when the target blocks hosting ranges or rate-limits your IP. If residential still fails, the problem is your TLS fingerprint or behavior, not your proxy.
How much do proxies cost? In 2026, datacenter proxies run about $0.5-2 per GB (or $2-4 per IP per month for dedicated IPs). Residential proxies run about $4-15 per GB — Bright Data lists ~$4/GB, Oxylabs ~$5/GB — with volume discounts past 100GB/month. Mobile is the most expensive at $15-40 per GB. One GB of HTML-only fetches is roughly 2,000-4,000 pages, so residential works out to about a tenth of a cent per page.
Do I need proxies to scrape? Usually not at first. A few thousand pages a day from one site fits fine on your own IP with polite pacing, or on one or two datacenter IPs. You need proxies when you measure an IP-related block: hosting-ASN rejection, per-IP rate limiting, or geo restrictions. Most scrapers that buy residential proxies are actually solving a fingerprint problem, which proxies don't fix. Diagnose the block before you buy the fix.
What is a rotating proxy and how does rotation work? A rotating proxy assigns you a new IP per request (or per short time window) from a provider's pool. Your scraper connects to the provider's gateway, which picks an exit IP for each connection based on the country and rotation mode you encoded in the username. Rotating spreads load across thousands of IPs so no single IP trips a rate-limit counter. Sticky mode, by contrast, pins you to one IP for a session you define — right for logins and cookie flows.
How do websites detect proxies? Three ways, in rough order of sophistication. IP reputation: hosting ASNs and known proxy ranges are blocklisted, which is how datacenter proxies get caught. TLS fingerprinting: your client's TLS handshake (JA3/JA4) and HTTP/2 settings identify the library you use, regardless of IP. Behavioral: request rate, timing, and patterns that don't look human. A proxy only fixes the first. The other two are your client's job.
Are proxies legal for web scraping? A proxy is a tool, not inherently legal or illegal. Legal risk comes from what you scrape and how you use it. The ground rules this site operates by: prefer an official API or dataset when one exists, respect robots.txt, rate-limit politely, and don't collect personal data you don't need. One proxy-specific caveat: residential IPs belong to real people who consented, often via an app's fine print, so check the provider's consent story and keep the traffic public and boring.
How do I use proxies with Python requests?
Pass a proxy dict to each request: requests.get(url, proxies={"http": proxy, "https": proxy}). For a rotating pool, keep a list of gateway URLs, pick one per request (or per session), and retry with backoff on 429 and 5xx. The full working example is in this guide. Pair it with a browser-accurate client like curl_cffi for protected targets — requests' TLS fingerprint is a giveaway on strict sites.
Further reading
- Residential Proxies for Scraping — the deep dive on the residential category: pool plumbing, consent, and when the price is worth it
- Web Scraping Without Getting Blocked — the operations playbook for staying alive, including the behavioral layer proxies can't fix
- Web Scraping APIs — the per-request alternative to a proxy pipeline, with the full provider roundup and cost math
- Scraping at Scale — queues, dedup, caching, and the self-host build path for 1M+ pages
- Web Scraping Ethics and robots.txt — the ground rules this site operates by, in full
Frequently Asked Questions
What is the best proxy for web scraping?
There is no single best proxy — there's a best for your situation. Datacenter proxies are the right default for most scrapes: cheap, fast, and fine for targets that don't IP-filter. Residential proxies (Bright Data ~$4/GB, Oxylabs ~$5/GB) earn their cost only when you hit per-IP rate limits, need geo-targeted IPs, or scrape search engines. Mobile is the most expensive and almost never needed. Start with datacenter, escalate only when you can point at an IP-level block.
Residential vs datacenter proxies: which should I use?
Datacenter proxies come from cloud and hosting providers: roughly $0.5-2/GB, fast, and easy to flag because hosting ASNs are blocklisted by default. Residential proxies are real ISP customers' connections: $4-15/GB, slower, and hard to flag at the IP level. Use datacenter first. Move to residential only when the target blocks hosting ranges or rate-limits your IP. If residential still fails, the problem is your TLS fingerprint or behavior, not your proxy.
How much do proxies cost?
In 2026, datacenter proxies run about $0.5-2 per GB (or $2-4 per IP per month for dedicated IPs). Residential proxies run about $4-15 per GB — Bright Data lists ~$4/GB, Oxylabs ~$5/GB — with volume discounts past 100GB/month. Mobile is the most expensive at $15-40 per GB. One GB of HTML-only fetches is roughly 2,000-4,000 pages, so residential works out to about a tenth of a cent per page.
Do I need proxies to scrape?
Usually not at first. A few thousand pages a day from one site fits fine on your own IP with polite pacing, or on one or two datacenter IPs. You need proxies when you measure an IP-related block: hosting-ASN rejection, per-IP rate limiting, or geo restrictions. Most scrapers that buy residential proxies are actually solving a fingerprint problem, which proxies don't fix. Diagnose the block before you buy the fix.
What is a rotating proxy and how does rotation work?
A rotating proxy assigns you a new IP per request (or per short time window) from a provider's pool. Your scraper connects to the provider's gateway, which picks an exit IP for each connection based on the country and rotation mode you encoded in the username. Rotating spreads load across thousands of IPs so no single IP trips a rate-limit counter. Sticky mode, by contrast, pins you to one IP for a session you define — right for logins and cookie flows.
How do websites detect proxies?
Three ways, in rough order of sophistication. IP reputation: hosting ASNs and known proxy ranges are blocklisted, which is how datacenter proxies get caught. TLS fingerprinting: your client's TLS handshake (JA3/JA4) and HTTP/2 settings identify the library you use, regardless of IP. Behavioral: request rate, timing, and patterns that don't look human. A proxy only fixes the first. The other two are your client's job.
Are proxies legal for web scraping?
A proxy is a tool, not inherently legal or illegal. Legal risk comes from what you scrape and how you use it. The ground rules this site operates by: prefer an official API or dataset when one exists, respect robots.txt, rate-limit politely, and don't collect personal data you don't need. One proxy-specific caveat: residential IPs belong to real people who consented, often via an app's fine print, so check the provider's consent story and keep the traffic public and boring.
How do I use proxies with Python requests?
Pass a proxy dict to each request: requests.get(url, proxies={'http': proxy, 'https': proxy}). For a rotating pool, keep a list of gateway URLs, pick one per request (or per session), and retry with backoff on 429 and 5xx. The full working example is in this guide. Pair it with a browser-accurate client like curl_cffi for protected targets — requests' TLS fingerprint is a giveaway on strict sites.
Keep reading
Residential Proxies for Web Scraping: The Honest Guide
What residential proxies actually do for web scraping, when you genuinely need them, and when they're wasted money: how IP rotation works, residential vs datacenter vs mobile, real pricing, integration with requests and Playwright, and why proxies don't fix a bad fingerprint.
E-commerce Scraping: The Complete 2026 Guide
The complete 2026 guide to e-commerce scraping: what data people scrape, platform anti-bot profiles, price-monitoring pipelines, cost math, working code, and the legal reality.
Web Scraping APIs in 2026: The Complete Guide
The honest 2026 guide to web scraping APIs: when to buy vs build, the five stages inside a call, real pricing for every major provider, cost math, code, and verdicts.
Found this useful? Cite it as: webscraping.space. “The Complete Proxy Guide for Web Scraping (2026).” https://webscraping.space/blog/complete-proxy-guide. Published 2026-08-14.