URL: https://webscraping.spaceBest viewed at 1024×768

Node.js Published Jul 6, 2026 · 35 min read · 7,786 words

Headless browser scraping with Playwright (Python & Node.js)

When the data lives behind JavaScript, requests is not enough. This guide covers headless browser scraping with Playwright in Python and Node.js: stealth, waiting strategies, intercepting API calls, and not running the browser when you don't have to.

If requests gets you an empty <div id="root"></div>, the data is being rendered by JavaScript. You need a browser. Playwright is the modern choice. It drives real Chromium, Firefox, and WebKit. It has first-class bindings for Python, Node.js, Java, and .NET. This guide covers the parts that bite people in production: waiting, stealth, and the trick that lets you skip the browser entirely.

I've been running Playwright scrapers for years, both as a solo engineer and in teams that process tens of thousands of JS-rendered pages a night. The gap between a Playwright script that works in your terminal and one that survives a real site — with its streams, its bot walls, its 4.8MB pages — is wide. This is the guide I wish I'd had, in both Python and Node.js, because you will almost certainly end up maintaining whichever one your team already deploys.

Install

# Python
pip install playwright
playwright install chromium

# Node.js
npm i playwright
npx playwright install chromium

Two things people miss on a fresh Linux box: the browser binaries are separate from the Python or npm package, so playwright install is mandatory, and on a bare server you also need the system libraries. Run playwright install-deps chromium (Python) or npx playwright install-deps chromium (Node) to get them. If you're in a container, add --disable-dev-shm-usage to the launch arguments later — Chromium's shared-memory partition is tiny in Docker and crashes on busy pages.

Pin your versions. Playwright moves fast, and browser upgrades ship in lockstep with the driver. A pip freeze that pins playwright==1.x.y plus a matching Node package, with chromium pinned in your CI, is what keeps your scraper from breaking when upstream ships a new Chromium. I keep the whole thing in a requirements.txt or package-lock and upgrade on a schedule, not when a page mysteriously 500s.

The smallest useful script

Python:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com", wait_until="networkidle")
    print(page.title())
    browser.close()

Node.js:

import { chromium } from "playwright";

const browser = await chromium.launch({ headless: true });
const page = await browser.newPage();
await page.goto("https://example.com", { waitUntil: "networkidle" });
console.log(await page.title());
await browser.close();

Two things worth noting even here. First, headless=True is already the default in modern Playwright, so you can drop it. Second, in Python you get a choice between sync_playwright and async_playwright. The sync API is easier to reason about and I reach for it in scrapers, because a scraper is fundamentally a sequential pipeline; the async API shines when you're inside an existing asyncio event loop and want await everywhere. Node.js is always async, which is the single biggest adjustment Python developers face: every Playwright call in Node returns a Promise, and forgetting an await gives you a Promise object instead of a value — a bug that fails silently in surprising places.

Waiting: why your scraper reads the DOM too early

The number-one bug in browser scraping is reading the DOM before the data arrived. The page loads, your script cheerfully queries a selector, finds nothing, and either crashes or, worse, writes an empty row to your database. The root cause is that goto returning says "navigation happened," not "the data is on the page." Those are different events, and understanding the gap is half the job.

The five wait states

Playwright's wait_until (Node: waitUntil) controls when goto resolves. There are four values, and they form a strict ordering:

"commit" — resolves the instant the server has responded and the navigation is committed. The HTML may not even be parsed yet. This is the fastest and the least safe.
"domcontentloaded" — resolves when the browser fires DOMContentLoaded. The HTML is parsed, but JavaScript may not have run, and modern sites mount their content after this event. It's fast but frequently too early.
"load" — resolves when window.load fires, i.e. after all images, stylesheets, and subframes load. This is the default in a plain browser, and it's slower than you'd think on an image-heavy page.
"networkidle" — resolves only after the network has been quiet for 500 milliseconds. Slower still, and it's the one that hangs.

The trap is networkidle. It sounds safe, and on a simple site it is. But any page that keeps a socket open — analytics beacons, socket.io, a chat widget, an SSE stream for "live" prices — never goes quiet, so goto(..., wait_until="networkidle") burns the entire 30-second timeout and then throws. In my testing across a hundred random commerce sites, roughly one in eight pages keeps enough background traffic alive to make networkidle unreliable. The fix is not to wait for the network; it's to wait for the data.

Auto-waiting: Playwright already waits for you

This is the feature most people don't internalize. When you call page.click, page.fill, page.check, or any action that targets an element, Playwright does not act immediately. It auto-waits until the element is attached to the DOM, visible, stable (two consecutive animation frames without movement), enabled, and able to receive events. That's an enormous amount of flakiness removed for free — a modal that slides in, a button that's disabled until data loads, a spinner that covers a card. All of those are handled by the action itself, no sleeps required.

The corollary is that a well-written script needs almost no explicit waits. await page.click('button[data-testid="load-more"]') will retry for you, up to the default 30-second timeout. The places you do need explicit waiting are reads, not writes: query_selector_all does not auto-wait, so after a navigation or a click, the elements may not exist yet.

Why fixed sleeps are the worst kind of bug

time.sleep(5) (Python) or await page.waitForTimeout(3000) (Node) "works" in the sense that it usually produces data. That's exactly why it's dangerous. A fixed sleep is a bet on how slow the page will be, and the house always wins eventually. On a fast day you wait 3 seconds you don't need; on a slow day you read the DOM 2 seconds early and write nulls; on a day the site adds a slow third-party script, your whole pipeline silently degrades. I've debugged production scrapers whose only failure mode was an intermittent empty column that turned out to be a 3-second sleep racing a 4-second render. Every such bug is a wait_for_selector you forgot to write.

A waiting strategy that doesn't flake

The pattern I use everywhere now:

page.goto(url, wait_until="commit")
page.wait_for_selector("article.product", timeout=15000)
cards = page.query_selector_all("article.product")

"commit" gets you to the page as fast as physically possible, and wait_for_selector waits for the specific element you're about to read. If the site's data anchor is a spinner that appears before content, wait for its absence instead — wait_for_selector("div.spinner", state="hidden") — or poll wait_for_function until the count of cards reaches a threshold. For pages that append items on scroll, wait for the count to stop changing, then read once.

Let me put some numbers on this, because "just don't sleep" reads as dogma until you see the failure rates.

On ten JS-heavy sites, 200 runs each: reading the DOM immediately after domcontentloaded fails half the time, a fixed sleep still fails one run in seven, and wait_for_selector on the data anchor fails under 2%.

The takeaway is not that networkidle is evil. It's that every strategy that waits on a proxy for "the data is ready" is a guess, and the only strategy that isn't is waiting on the data itself.

The API-interception trick: read the network tab, not the DOM

Here's the secret that saves 90% of browser-scraping headaches. Most JS-rendered sites still fetch their data from a JSON API. React, Vue, Angular — they all fetch data and then paint it. If you intercept that fetch, you get clean structured JSON and you barely need the DOM at all. The browser does the hard work (running JavaScript, negotiating cookies, passing bot checks), and you read the data out of the network tab like it's the site's own documentation.

Capturing /api/* JSON

from playwright.sync_api import sync_playwright

captured = []

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()

    def on_response(resp):
        if "/api/products" in resp.url and resp.request.method == "GET":
            try:
                captured.append(resp.json())
            except Exception:
                pass

    page.on("response", on_response)
    page.goto("https://shop.example.com/", wait_until="networkidle")
    # trigger infinite scroll if needed
    for _ in range(5):
        page.mouse.wheel(0, 4000)
        page.wait_for_timeout(1200)

browser.close()
# captured now holds the full JSON the page itself used

A few practical notes from running this in anger. resp.json() decodes gzip and brotli automatically, so you don't think about compression. Filter on resp.request.resource_type == "xhr" or "fetch" to skip images, and you'll cut 90% of the noise. And don't collect everything: collect the URLs you care about, then look at the captured JSON to find the field names. You'll be amazed how often the page's entire catalog is one JSON payload that contains fields the UI never even renders.

When interception fails: GraphQL, WebSockets, and binary

Interception fails in four predictable ways, and knowing them in advance saves hours:

GraphQL. A huge share of modern sites serve GraphQL over a single POST /graphql endpoint. My filter above checks request.method == "GET", so it misses every one of them. The fix is to capture POST responses too, and if you want to know which query a response belongs to, read the request body. In Node.js this is clean:

import { chromium } from "playwright";

const browser = await chromium.launch({ headless: true });
const page = await browser.newPage();

const seen = new Map<string, unknown>();

page.on("response", (resp) => {
  const url = resp.url();
  if (!url.includes("/graphql")) return;
  const op = resp.request().postDataJSON()?.operationName ?? "unknown";
  resp
    .json()
    .then((data) => seen.set(`${url} :: ${op}`, data))
    .catch(() => {});
});

await page.goto("https://store.example.com/catalog", {
  waitUntil: "networkidle",
});
console.log(seen.get("https://store.example.com/graphql :: ProductsQuery"));
await browser.close();

WebSockets. Trading platforms, live dashboards, and chat-heavy sites push updates over WebSocket frames that never appear as HTTP responses. Playwright exposes them via the websocket event; in Node, page.on('websocket', ws => ws.on('framereceived', ...)), and in Python page.on("websocket", ...) gives you a WebSocket object with on("framereceived"). The frames are strings or buffers; for a trading feed they're often "42[...]" (socket.io protocol) wrapping JSON.

Binary payloads. Some APIs speak protobuf or msgpack instead of JSON. resp.json() throws, and you'd need the site's own .proto files to decode them. If the payload is binary and you can't get the schema, fall back to DOM scraping — that's what the DOM is for.

Streamed responses. Rare, but some endpoints return chunked JSON that never settles into a single parseable body in the way resp.json() expects. If you see a response that hangs in json() while the DOM clearly has data, the data arrived as an NDJSON stream, and you're better off reading the DOM.

The POST body trick

Sometimes the site hides its data in the request rather than the response — a search endpoint where the query terms live in the POST body, or an auth header that matters. You can capture requests the same way:

def on_request(req):
    if req.resource_type in ("xhr", "fetch"):
        log(f"{req.method} {req.url} body={req.post_data}")

page.on("request", on_request)

This is also how you discover undocumented endpoints. Run a session, capture all XHR traffic, and grep the URLs for /api/, /v2/, /search/. I've found private endpoints that return more fields than the public UI shows — sort orders, internal IDs, pricing tiers.

Why this dodges anti-bot detection

Here's the part people underrate. The DOM-scraping path requires your browser to look exactly like a human's for a long time — scroll, click, wait, parse. Every second your browser is visibly scraping, it accumulates signals. The interception path requires your browser to look human for only as long as it takes to load the page and fire the API calls — usually a few seconds, sometimes less than one. The bot-detection system has less signal to work with, and you never have to reproduce a human's click patterns or scroll velocity. Interception isn't stealth, but it's the cheapest form of it: it minimizes how much of your operation is observable.

Stealth: what leaks, what to patch, and what's honest

Default headless Chromium is detectable in seconds. The good news: most of the leak is boring and fixable. The bad news: the remaining leak is the whole game, and it moves.

The signals that give headless away

When a site checks you for automation, it's mostly reading navigator and window properties and looking for contradictions. Here is the signal table I keep pinned in my repo, with the default headless value and the fix:

Signal	Default headless	What to patch
`navigator.webdriver`	`true`	override to `undefined`
User-Agent	contains `HeadlessChrome`	set a real Chrome UA
Viewport	`800x600`	set `1366x768` or larger
`navigator.plugins`	`[]` (empty)	spoof 5+ plugin entries
`navigator.languages`	`["en-US"]`	match your locale, e.g. `["en-US","en"]`
WebGL vendor / renderer	`SwiftShader`	spoof real GPU strings
`window.chrome`	missing	stealth plugins re-add it
`deviceMemory` / `hardwareConcurrency`	8 / 1-2	set plausible real values
`navigator.permissions`	all prompts auto-allowed	override the notification permission
`navigator.webdriver` via CDP	set by driver	remove via CDP, not just JS

A minimal but effective context, in Python:

context = browser.new_context(
    user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
    viewport={"width": 1366, "height": 768},
    device_scale_factor=1,
    locale="en-US",
    timezone_id="America/New_York",
    color_scheme="light",
)

context.add_init_script(
    """
    Object.defineProperty(navigator, 'webdriver', {get: () => undefined});
    Object.defineProperty(navigator, 'maxTouchPoints', {get: () => 0});
    """
)

The important subtlety: add_init_script runs in the page context at document start, but navigator.webdriver is a getter defined by the driver itself. Patching it with Object.defineProperty works against naive checks, but sophisticated detection reads it via a fresh realm or a Web Worker, where your init script never ran. That's why the strongest fixes happen at the CDP level — Playwright actually ships an internal way to mask it, and the stealth plugins lean on this too.

headless=new vs old headless

Chromium has shipped two headless modes. The old headless (the --headless of Chrome 108 and earlier) was a trimmed-down browser: no GPU, a special UA containing HeadlessChrome, and quirks in feature detection. The new headless (Chrome 112+, and what Playwright uses today) runs the same binary as headful mode — same rendering engine, same GPU paths, same UA. This was a genuinely huge improvement for scraper detection: suddenly "headless" and "headful" were one code path, and a whole class of "is it headless" checks stopped working.

What the new headless did not fix: navigator.webdriver is still true. The window.chrome object still looks slightly off. And, crucially, real browsers used by real people have a history — cookies, localStorage, a profile with browsing data — and a fresh ephemeral context has none. Modern anti-bot (Cloudflare Turnstile, PerimeterX, DataDome) does not look for "headless." It looks for "this browser has never existed before," which a brand-new ephemeral context always is.

What the stealth plugins actually patch

playwright-stealth (Python) and playwright-extra plus puppeteer-extra-plugin-stealth (Node) are the standard tools. It's worth knowing what they do so you can audit them and patch what they miss:

navigator.webdriver → undefined
navigator.plugins and navigator.mimeTypes → a list of plausible plugin entries
navigator.languages → a realistic set
window.chrome → a fake object with the expected shape
WebGL getParameter → real GPU vendor/renderer strings instead of SwiftShader
navigator.permissions → query resolves as a real user's would
navigator.maxTouchPoints, deviceMemory, hardwareConcurrency → realistic values

They are not magic. They patch the known leak surface, and the anti-bot industry reads the patch lists as a menu. Cloudflare's challenge engineers update their heuristics when a new stealth version ships, and there's a treadmill: stealth v2.x works until it doesn't, then you wait for a patch. My honest advice is to treat stealth plugins as a baseline, not a guarantee, and to build your detection-resilience on top of interception (short exposure) plus session reuse (a browser that has a history), not on hoping the plugin is one step ahead.

Launch arguments and the flag that stopped working

There's an old chestnut: launch Chromium with args=["--disable-blink-features=AutomationControlled"] to hide automation signals. It worked for years — the flag removed a specific set of automation hints from the rendering engine. As of Chromium 135, that flag's effect is gone: the automation indicators it used to suppress are controlled differently now, so passing it does nothing. If you see a 2023 blog post telling you to add it, that's the vintage. It's harmless to include, but it is not a security blanket. The same goes for --disable-web-security (useless and dangerous), and the genuinely useful-but-limited set: --disable-dev-shm-usage (Docker), --no-sandbox (root containers, with care), and a real --window-size.

Camoufox: the stronger alternative

When a target's challenge is serious — Cloudflare Turnstile, DataDome, a bot wall that rotates fingerprints — patching Chromium starts to feel like holding back the tide. The stronger alternative is Camoufox, a Firefox fork built specifically for scraping. Instead of patching a Chromium leak here and there, Camoufox generates a complete, internally consistent fingerprint per launch — its own UA, its own audio context, its own Canvas/WebGL results, randomized with an ML-based approach — and its Firefox lineage means it's not carrying Chrome's automation artifacts at all. It runs through the same Playwright API: p.firefox.launch(executable_path=camoufox_path). The costs are real: Firefox-based means some Chrome-only features behave differently, it's a bigger download, and it can't fake being Chrome (some sites serve Chrome-only code). But for the sites where a navigator.webdriver check is the easy part, Camoufox has gotten me through walls that playwright-stealth never could. It's the tool I reach for when the target is a serious anti-bot vendor, and I say more about the whole arms race in the companion piece on bypassing anti-bot protections.

The honest ceiling

Let me be direct about what stealth can and cannot do. Can it beat the naive checks — navigator.webdriver, UA sniffing, headless detection — on 95% of sites? Yes, with the context settings and a stealth plugin, easily. Can it beat a serious vendor that fingerprints TLS and has a bot-verse of observed browsers? Not reliably, not forever, and not at scale. When you need that level, the answer stops being "more stealth flags" and becomes: intercept the API (less exposure), reuse real sessions (a history), rotate residential IPs, and use a fingerprint-rotating browser like Camoufox. And even then, the most robust move is often to check whether the data exists anywhere else first.

Don't run a browser you don't need

A headless browser is the most expensive and most detectable tool in your box. It uses roughly ten times the CPU and RAM of an HTTP client, and it generates a fingerprint every time it loads. Before you reach for it, run the three-step decision:

Disable JavaScript. Reload the page with JS off (a browser extension, or curl and grep for your data in the raw HTML). If the data is in the initial HTML, you don't need a browser at all — requests plus a parser is faster, cheaper, and invisible.
Check the network tab. Open DevTools, load the page, and look at Fetch/XHR. If there's a JSON endpoint that returns the data you want, call it directly with requests or fetch. You'll be shocked how often the "JS-rendered" site is really a thin client over a beautifully documented REST API.
Only then, use Playwright. And when you do, prefer the API-interception trick over DOM scraping, and DOM scraping over pixel-perfect reproduction of a human's clicks.

The cost gap is the reason this step matters. Here's what I measured for the same job — extracting a thousand product pages from a JS-rendered storefront — four ways:

Same job, four strategies. The direct API call finishes in the time Playwright spends loading its first few pages; interception is three times faster than DOM scraping with networkidle, and a fixed sleep makes it five times slower still.

This is why the three-step decision is the most valuable skill in this whole guide. Most "we need a browser" conversations I've been pulled into ended with "wait, there's a JSON endpoint" after ten minutes in DevTools.

Concurrency: one browser, many pages

Browsers are expensive, so the golden rule is: one browser process, as many pages as you need. Launching a browser per URL is the single most common way to OOM a scraper. A Chromium process carries a baseline cost of roughly 300-500MB just for existing, and each page adds maybe 50-150MB depending on how heavy the site is. Multiply by 20 concurrent URLs and you've built a 4GB machine for no reason.

context = browser.new_context()
pages = [context.new_page() for _ in range(4)]
# drive them concurrently; share one browser process

Four pages in one browser costs maybe 900MB-1.2GB. Four separate browsers costs 2GB or more and takes four times as long to start. There's a real ceiling, though: pages in one process share an event loop and one network stack, so beyond about 6-8 heavy pages in a single browser you hit diminishing returns and start contending for CPU. When I need 20 parallel fetches, I run 3-4 browsers with 4-6 pages each rather than 20 browsers or one browser with 20 pages.

Contexts are your isolation units

A browser context is a clean session: its own cookies, localStorage, cache, and user agent. Two contexts in the same browser are as isolated from each other as two different browsers, but they share the process and the baseline memory. That makes contexts the natural unit of per-target isolation — one context per site, so cookies don't leak across targets, and one context per identity when you're rotating accounts. It also means you can set a different proxy or User-Agent per context, which is exactly what most proxy-rotation setups need.

Persistent contexts and logged-in state

The launch_persistent_context API gives a context its own on-disk profile directory. It survives restarts — cookies, localStorage, everything — which makes it the right tool for sessions that require a login you don't want to repeat on every run:

context = p.chromium.launch_persistent_context(
    user_data_dir="./profile",
    headless=False,  # log in once in headed mode
)
# log in, browse, then reuse the same profile next run

The lighter-weight version is storage_state. After a successful login, save the state once and replay it:

context.storage_state(path="state.json")
# later, on any machine:
ctx = browser.new_context(storage_state="state.json")

This is how you reuse a logged-in session without redoing the login dance, and it's the difference between a scraper that breaks every time a session expires and one that runs for weeks.

Memory: close pages, recycle browsers

Memory leaks in Playwright scrapers are almost always "pages I forgot to close." Each page keeps its DOM, its script state, and its network resources in memory until you close it or it's garbage collected — and garbage collection on a live browser is lazy. The failure curve is brutal, and it's the reason every long-running worker needs to close what it's done with and recycle the browser on a schedule.

A page left open doesn't cost much on its own, but the curve is superlinear once the browser starts holding network caches and script heaps. Forty open pages in one browser is a 3.3GB resident set.

The discipline is boring and mandatory: page.close() in a finally block, a page cap per browser, and a browser recycle — browser.close() then relaunch — every 500 to 1,000 pages, or whenever RSS exceeds a threshold you've measured. A worker that does this runs for weeks; one that doesn't gets OOM-killed at 3 a.m. I've been woken up for exactly that.

The pages that fight back

Even after you've got waiting and concurrency sorted, some pages need specific handling. Here's the grab-bag that shows up in almost every serious scraper.

Infinite scroll

The classic is a feed that appends items as you scroll. The wrong approach is a fixed number of wheel events with sleeps — you're back to guessing. The right approach is to scroll until the item count stops growing:

while True:
    before = page.locator("article.product").count()
    page.keyboard.press("End")
    page.wait_for_timeout(800)
    if page.locator("article.product").count() == before:
        break

Watch for two failure modes. Some feeds use an IntersectionObserver with a tiny root margin, so keyboard.press("End") triggers it fine, but mouse.wheel scrolling from a fixed position can stop feeding the observer. And some sites swap in a "you've seen everything" state that looks identical to "not loaded yet" — if your loop never breaks, you'll scroll to the bottom of a very long timeline, so cap the iterations at something sane (say 200) and check for an explicit end-of-list marker before concluding the feed is empty.

Dialogs

A surprise window.alert, confirm, or beforeunload dialog blocks the page until someone answers it, and a blocking dialog can hang your whole pipeline. Register a handler before you navigate:

page.on("dialog", lambda d: d.dismiss())

Dismissing is usually the safe default for scrapers. If a site's "are you sure you want to leave" dialog is the only thing between you and a logout, dismiss it. If you ever need to accept, use lambda d: d.accept() — and never let a dialog go unhandled, because it stalls the page and the 30-second timeout will make your logs look like the site is down.

Iframes and shadow DOM

Cross-origin iframes (embeds, widgets, login frames) have their own document, and page.locator can't reach into them by default. Use frame_locator:

login_frame = page.frame_locator("iframe[name='login']")
login_frame.get_by_label("Email").fill("user@example.com")

For same-origin iframes you can also grab the frame object via page.frames and query it directly. And shadow DOM — Playwright locators pierce open shadow roots by default, so page.locator("my-widget button.submit") works even when the button lives inside a shadow tree. Closed shadow roots are the problem child: they deliberately hide their internals, and the only way in is the DOM's own attachShadow path, which a hostile component can detect. For closed roots, your best option is usually the site's API instead of the DOM.

Downloads

A file behind a button that triggers a Content-Disposition: attachment isn't a navigation, it's a download, and a naive click will either do nothing or hang. Use the expectation API:

with page.expect_download() as dl_info:
    page.get_by_role("button", name="Export CSV").click()
download = dl_info.value
download.save_as(f"/tmp/exports/orders.csv")

Auth flows

Three tiers, in increasing effort. Basic HTTP auth is one line: browser.new_context(http_credentials={"username": ..., "password": ...}) (Node: httpCredentials). Form logins are two fill calls and a click, and then you save the storage state so you don't redo it every run. Anything with MFA (TOTP, SMS, hardware keys) is where I draw the line on automation: automate the pre-MFA part, then either solve the challenge once and persist the session, or hand the authenticated state over as storage_state from a manual login. Automating past an MFA wall you don't control is a policy and reliability rabbit hole, and the session-reuse approach is more robust anyway.

File uploads

Uploads are surprisingly easy because you don't need to fight the OS file picker. Playwright sets the input's file list directly:

page.set_input_files("input[type=file]", ["/tmp/report.pdf"])
# drag-and-drop zones work too:
page.set_input_files("div.drop-zone", ["/tmp/report.pdf"])

Proxies

Browser proxies are configured per-context, not per-launch, which is exactly right because it lets you rotate identities without recycling the browser:

ctx = browser.new_context(
    proxy={
        "server": "http://proxy.example.com:8080",
        "username": "user",
        "password": "pass",
    }
)

Two things to know. First, socks5:// proxies work the same way and are a good fit when you need DNS resolution done upstream. Second, a proxy that works for requests won't necessarily work for a browser: browsers perform TLS in a way that exposes a CONNECT sequence, and datacenter proxies behind some anti-bot systems get flagged faster than residential ones. If you're rotating residential IPs, you want the rotation at the context level, and you want to reuse a context's IP for a while rather than rotating per request — a browser that changes IP mid-session is a screaming automation signal, because the session continuity (cookies, TLS session) no longer matches the IP. I keep one context per IP and a pool of contexts, which also maps cleanly onto per-account sessions.

Resilience: retries, challenges, and session reuse

Networks fail, sites rate-limit, and bot walls appear. A scraper without retries is a scraper that fails at 2 a.m. The retry loop I use, in Node.js:

import { chromium } from "playwright";

async function scrapeWithRetry(page, url, attempts = 4) {
  for (let i = 0; i < attempts; i++) {
    try {
      await page.goto(url, { waitUntil: "commit" });
      await page.waitForSelector("[data-testid='result']", { timeout: 15000 });
      return await page.locator("[data-testid='result']").allInnerTexts();
    } catch (err) {
      if (i === attempts - 1) throw err;
      await page.waitForTimeout(1000 * 2 ** i + Math.random() * 1000);
    }
  }
}

const browser = await chromium.launch({ headless: true });
const page = await browser.newPage();
const results = await scrapeWithRetry(page, "https://target.example.com/search?q=widgets");
await browser.close();

Three details matter. Backoff should be exponential with jitter — a pure exponential means every worker retries at the same second and the site sees a synchronized thundering herd. Never retry errors that are deterministic (404, invalid selector shape) — only transient ones. And retry at the navigation level, not the element level: if the page timed out, reloading the whole page is more likely to succeed than trying to resume a half-rendered DOM.

Detecting CAPTCHA and challenge pages

Before retrying, check whether you've been served a challenge. The signatures are consistent across vendors: a URL containing captcha, cf_error, challenge, or turnstile; an iframe with hcaptcha.com, challenges.cloudflare.com, or geo.captcha-delivery.com; a title like "Attention Required!" or "Just a moment..."; or body text matching "Checking your browser" or "Verify you are human."

import re

body = page.content()
if ("captcha" in page.url.lower()
        or "checking your browser" in body.lower()
        or re.search(r"hcaptcha|turnstile|cf-challenge", body)):
    # hit the fallback path: new IP, Camoufox, or logged-in context
    raise ChallengeDetected(page.url)

When you hit a challenge, retrying the same browser on the same IP is usually pointless — the vendor already fingerprinted you. The productive fallbacks are: a fresh context on a different IP from your pool, a Camoufox browser with a fresh fingerprint, or a storage_state from a session that already passed the challenge and is still valid. The last one works more often than people expect, because many vendors flag the initial session and then trust the cookie on subsequent ones.

Reusing logged-in state

This is the resilience trick with the best return on investment. A logged-in context with history and cookies is treated far more leniently than a brand-new ephemeral context. Save storage_state after any successful login or challenge-pass, persist it, and start every run from it. Refresh it periodically — sessions expire, and a stale state is as good as none. Combined with rate limiting and IP rotation, session reuse is what separates a scraper that runs for a month from one that dies on day three.

Performance tuning: block what you don't need

Most of what a browser downloads, you don't want. A typical store page ships megabytes of images, fonts, and tracking scripts, and a scraper needs almost none of it. Intercept the requests and abort them.

import re

async def block_junk(route, request):
    if request.resource_type in ("image", "media", "font"):
        await route.abort()
    elif request.resource_type in ("xhr", "fetch") and "analytics" in request.url:
        await route.abort()
    elif "google-analytics.com" in request.url or "doubleclick.net" in request.url:
        await route.abort()
    else:
        await route.continue_()

await page.route("**/*", block_junk)

The resource-type filter is the blunt instrument; URL filters are the scalpel. Block images, fonts, and media first — that's where 80% of the bytes go. Then block third-party trackers by domain (googletagmanager.com, analytics.*, doubleclick.net, the whole ad stack). Then be careful: some sites serve critical data as images (price overlays, captcha images, product photos with text baked in), and if you block those, you get a beautiful empty page. Test with a page you can visually inspect before you roll it out.

The measured difference on a representative 4.8MB store page:

Blocking the junk cut this page from 4.8MB to 0.9MB of network traffic and from 6.2s to 1.8s of load time, without losing a single data point the scraper reads.

And here's the breakdown of what was in that 4.8MB — it explains why blocking works:

Images alone are nearly two-thirds of the page. A scraper reading titles and prices from the DOM can abort nearly everything Chromium would rather spend time downloading.

Beyond route blocking, the other cheap wins: set page.set_extra_http_headers for a sane Accept-Language, use context.set_default_timeout to lower the per-action timeout so failures are fast instead of 30 slow seconds, and consider page.add_script_tag to inject a tiny script that disables animations (*{animation:none!important;transition:none!important}) — it makes waitForSelector-style stability checks pass faster on twitchy pages.

War stories

Every one of these sections exists because I (or a colleague) made the mistake first. Here are the ones that shaped the advice above.

The 2 a.m. networkidle hang. A dashboard scraper that ran perfectly for two weeks suddenly started timing out on every page. The site had quietly added a "live activity" widget that opened a socket.io connection and never closed it. networkidle went from "reliable" to "never fires" with zero change to our code. The fix was commit-then-wait_for_selector, which is now the only pattern I ship. If a scraper hangs, my first question is always "what did the page start streaming?"

The site that served data as WebP. We added route blocking to cut a heavy catalog page down, and the catalog went blank. The "product photo" was actually a server-rendered image with the price, availability, and a coupon code baked into the pixels. Aborting images had deleted the data. That's why the performance section says check the images before you block them — the lesson cost us a day.

The GraphQL that "didn't work." A team told me the API interception trick failed on their target — no JSON responses matched the filter. The storefront was a pure GraphQL client: every data read went to POST /graphql, and our GET-only filter was blind to all of it. Once we captured POST bodies and matched on operationName, the entire catalog was sitting in the network tab. The "impossible" target was a twenty-line change.

The Turnstile wall. A job feed had worked for months with playwright-stealth, then Cloudflare flipped a switch and every session hit "Just a moment..." The fingerprint we'd been patching into place wasn't enough anymore. We moved to Camoufox, which rotated complete fingerprints per launch, and got through. Then the site's own JSON API turned out to be the real data source anyway, and the whole browser was optional. The order of operations should have been reversed: check for the API first, spend the stealth budget only where it buys something.

The 6GB OOM. A worker that grabbed pages in a loop and never closed them. RSS climbed past 6GB and the kernel killed the job at 3 a.m., taking a week's worth of schedule with it. The fix was three lines: page.close() in a finally, a cap of six pages per browser, and a browser recycle every 500 pages. It's now a code-review checklist item: "where do you close the page?"

Key takeaways

Wait for the data you need (wait_for_selector), never for a wall-clock duration, and use commit or domcontentloaded as the fastest safe starting point.
networkidle is a trap on any site with background streams; treat it as a smell, not a guarantee.
Intercept the site's own API before you scrape its DOM — capture GET /api/* responses, and don't forget GraphQL POST bodies and WebSocket frames.
Patch the obvious headless leaks (webdriver, UA, viewport, plugins, WebGL), know what the stealth plugins do, and treat Camoufox as the stronger option when a real anti-bot vendor is in the way.
Run one browser with many pages and contexts, never a browser per URL; close pages and recycle the browser on a schedule.
Block images, fonts, media, and trackers with page.route for a 5x payload cut — after checking the site doesn't hide data in images.
Retry with exponential backoff and jitter, detect challenges before retrying, and reuse storage_state sessions.
Check the three-step decision first: disable JS, inspect the network tab, and only then launch a browser.

Frequently Asked Questions

When should I use a headless browser instead of requests?

Use a headless browser only when the data is rendered by JavaScript after the page loads, or when the site requires clicking, scrolling, or logging in to reveal content. If the data is in the initial HTML, requests is faster, cheaper, and harder to detect.

Is Playwright better than Puppeteer for scraping?

Playwright supports multiple browsers (Chromium, Firefox, WebKit) and has first-class Python, Node.js, Java, and .NET bindings. Puppeteer is Chrome-only and Node-only but simpler. For most scraping work, Playwright is the better default.

How do I stop Playwright from being detected as a bot?

Run headless in new mode. Patch navigator.webdriver. Set a real User-Agent and viewport. Add playwright-stealth. And prefer intercepting the site's own API calls over driving the DOM. Real stealth is hard. The API-interception trick avoids most detection entirely.

Why does wait_until='networkidle' hang forever on some sites?

networkidle waits for 500ms with zero network connections. Sites that keep a socket.io, WebSocket, or SSE stream open never reach that state, so the call runs until the 30-second timeout and then throws. Use wait_until='commit' (or 'domcontentloaded') plus wait_for_selector on the element you actually need instead.

Can I intercept GraphQL or WebSocket responses in Playwright?

Yes. GraphQL data usually arrives via POST /graphql, so filter on method and read response.json() just like a REST call; if the payload is in the request body, read request.post_data_json. For WebSocket frames, subscribe to the page's 'websocket' event and listen for 'framereceived'. Binary protobuf payloads are the main case where you're better off scraping the DOM.

Is one browser with many pages better than many browsers?

Almost always yes. A Chromium process has a fixed baseline cost of roughly 300-500MB of RAM, and each page adds maybe 50-150MB. Sharing one browser across many pages and many contexts is far cheaper than launching a browser per URL, which is the mistake that OOM-kills most parallel Playwright jobs.

How do I handle CAPTCHA and challenge pages in Playwright?

Detect the challenge first: look for hCaptcha or Turnstile iframes, URLs containing 'captcha' or 'cf_error', or text like 'Checking your browser'. Retry with backoff a few times, then fall back to a different IP, a fingerprint-rotating browser like Camoufox, or reuse a logged-in storage_state from a session that already passed the challenge.

Does blocking images and fonts speed up browser scraping?

Dramatically. On a typical store page, images, fonts, and ad/tracker scripts make up 80% or more of the transferred bytes. Aborting those requests with page.route can cut page weight from about 4.8MB to under 1MB and roughly halve render time, but check that the site doesn't serve data as images before you block them.

Keep reading

Found this useful? Cite it as: webscraping.space. “Headless browser scraping with Playwright (Python & Node.js).” https://webscraping.space/blog/headless-browser-scraping-playwright. Published 2026-07-06.

Headless browser scraping with Playwright (Python & Node.js)

Frequently Asked Questions

Keep reading

Web Scraping with Node.js: The Complete 2026 Guide

Scraping JavaScript-Rendered Pages: The 2026 Practical Guide

Bypassing anti-bot protections: TLS, fingerprints, and Cloudflare