Node.js Published Jul 6, 2026 · 2 min read · 481 words

Headless browser scraping with Playwright (Python & Node.js)

When the data lives behind JavaScript, requests is not enough. This guide covers headless browser scraping with Playwright in Python and Node.js: stealth, waiting strategies, intercepting API calls, and not running the browser when you don't have to.

If requests gets you an empty <div id="root"></div>, the data is being rendered by JavaScript. You need a browser. Playwright is the modern choice. It drives real Chromium, Firefox, and WebKit. It has bindings for Python and Node.js. This guide covers the parts that bite people: stealth, waiting, and the trick that lets you skip the browser entirely.

Install

# Python
pip install playwright
playwright install chromium

# Node.js
npm i playwright
npx playwright install chromium

The smallest useful script

Python:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com", wait_until="networkidle")
    print(page.title())
    browser.close()

Node.js:

import { chromium } from "playwright";

const browser = await chromium.launch({ headless: true });
const page = await browser.newPage();
await page.goto("https://example.com", { waitUntil: "networkidle" });
console.log(await page.title());
await browser.close();

Waiting: the thing that breaks every scraper

The number-one bug in browser scraping is reading the DOM before it's ready. wait_until controls when goto resolves. You usually want one of:

• "domcontentloaded" is fast, but JS may not have run yet.
• "networkidle" waits until no network for 500ms. Slower but usually safe.

For specific content, wait for the selector instead of guessing:

page.goto(url, wait_until="domcontentloaded")
page.wait_for_selector("article.product", timeout=15000)
cards = page.query_selector_all("article.product")

Never time.sleep(5) and hope. It's flaky and slow.

The API-interception trick (use this first)

Here's the secret that saves 90% of browser-scraping headaches. Most JS-rendered sites still fetch their data from a JSON API. If you intercept that call, you get clean structured data. You barely need the DOM at all.

from playwright.sync_api import sync_playwright

captured = []

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()

    def on_response(resp):
        if "/api/products" in resp.url and resp.request.method == "GET":
            try:
                captured.append(resp.json())
            except Exception:
                pass

    page.on("response", on_response)
    page.goto("https://shop.example.com/", wait_until="networkidle")
    # trigger infinite scroll if needed
    for _ in range(5):
        page.mouse.wheel(0, 4000)
        page.wait_for_timeout(1200)

browser.close()
# captured now holds the full JSON the page itself used

You let the browser do the hard work (running JS, passing bot checks) and read the data from the network tab instead of the DOM. It's faster, more robust, and far less sensitive to redesigns.

Stealth basics

Default headless Chromium leaks navigator.webdriver === true and a distinctive fingerprint. Clean up the obvious signals:

context = browser.new_context(
    user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) ... Chrome/126.0.0.0 Safari/537.36",
    viewport={"width": 1366, "height": 768},
    locale="en-US",
)

context.add_init_script(
    "Object.defineProperty(navigator, 'webdriver', {get: () => undefined})"
)

For more, install playwright-stealth (Python) or playwright-extra with the stealth plugin (Node.js). They patch a bundle of leaky properties at once.

Don't run the browser you don't need

A headless browser uses about 10x the CPU and RAM of an HTTP client. It's about 10x more detectable. Before reaching for one:

1. Reload the page with JavaScript disabled. Is the data in the HTML now? Then you don't need a browser.
2. Open DevTools, go to Network, look at Fetch/XHR. Is there a JSON endpoint? Call it directly with requests.
3. Only if both fail, use Playwright. And prefer the API-interception trick over DOM scraping.

Concurrency

Browsers are expensive. Run several pages in one browser, not several browsers:

context = browser.new_context()
pages = [context.new_page() for _ in range(4)]
# drive them concurrently; share one browser process

Close pages you're done with. Memory grows fast otherwise.

Key takeaways

• Wait for selectors. Never sleep.
• Intercept the site's own JSON API. Parse the DOM only as a last resort.
• Patch navigator.webdriver. Set a real User-Agent and viewport.
• Share one browser across many pages. Close them when done.
• Use a browser only when requests genuinely can't see the data.

Found this useful? Cite it as: webscraping.space. “Headless browser scraping with Playwright (Python & Node.js).” https://webscraping.space/blog/headless-browser-scraping-playwright. Published 2026-07-06.