URL: https://webscraping.spaceBest viewed at 1024×768

JavaScript Published Aug 8, 2026 · 34 min read · 7,498 words

Scraping JavaScript-Rendered Pages: The 2026 Practical Guide

How to scrape JavaScript-rendered websites in 2026: detecting when data is behind JS, finding and calling the site's own JSON APIs, render proxies, headless browsers with Playwright, cost and detection math for each approach, and the decision framework for choosing.

If requests gets you a <div id="app"></div> with nothing inside it, you're looking at a JavaScript-rendered page, and this guide is the decision framework for what to do about it. It's the most common wall in modern scraping: the HTML shell is real, the response is real, and the data just isn't in it. Somewhere between that empty div and your screen, JavaScript ran, made network calls, and built the DOM you see.

The mistake everyone makes on first contact is reaching for a headless browser the moment they see the empty div. A browser is the most expensive and most detectable tool in the box. Before you get there are four cheaper rungs, and in my experience about 70 percent of "JS-rendered" targets are scrapable on rungs one and two with nothing but requests and a DevTools window. This post is the full walkthrough: detecting that content is behind JavaScript, the four rungs and their cost math, how SPAs work under the hood, and the techniques — embedded JSON, API interception, GraphQL, cursor pagination — that let you skip the browser entirely. The deep dive lives in our Playwright guide; here I cover when you're forced onto that rung and what it costs.

The core problem: pages that are empty shells

A server-rendered page contains its content in the HTML: the server queries a database, renders a template, ships finished markup. A client-side-rendered page ships a skeleton — a <div id="root"> or <div id="app">, a JavaScript bundle, nothing else. The content doesn't exist yet. It gets created in the browser: JavaScript runs, reads state, fires network requests, and inserts DOM nodes into that empty div.

Here is what a client-side-rendered page looks like as raw bytes, a real pattern you will see constantly:

<!DOCTYPE html>
<html>
  <head>
    <title>Product Catalog</title>
    <script src="/static/js/main.8f2a3.js"></script>
  </head>
  <body>
    <div id="root"></div>
  </body>
</html>

That is the entire response for a page that shows a thousand products. The products don't exist until main.8f2a3.js runs, and that script only bootstraps a framework, which then issues its own requests for the data. A "page" is really three round trips: HTML, JavaScript, then data. A DOM parser over the raw HTML returns exactly nothing — that's the trap.

Two rendering families explain most of what you'll encounter. Server-side rendering (SSR) ships the content HTML in the response. Client-side rendering (CSR) ships the shell and lets the browser generate the content. Everything else is a hybrid: static site generation pre-renders HTML at build time and serves it like SSR; streaming SSR ships a bit of content plus a bundle that hydrates the rest. For a scraper, the question is not what the framework calls itself; it's whether the data you want is in the HTML a plain HTTP client receives. If it is, you're done. If it isn't, you need the ladder.

An SSR page hands a scraper the content on the first request. A CSR page ships an empty shell, a JavaScript bundle, and the data arrives only after the browser runs the bundle and fires its own API calls.

How to detect that content is JS-rendered, fast

Before choosing a tool, know which world you're in. Five minutes of checking saves you from building a Playwright farm for a page that had __NEXT_DATA__ in the raw HTML all along. Here's the sequence I run on every new target, cheapest first.

Step one: disable JavaScript in your browser. In Chrome, open DevTools, hit Ctrl+Shift+P, select "Disable JavaScript", reload. Content still there? The server rendered it and plain HTTP will work. Empty div? JavaScript is doing the work. Fastest test there is.

Step two: curl the URL and read the raw bytes. The browser lies to you; the raw response doesn't. Run curl -s https://target.example/product/123 | head -c 4000 and look for three things. First, is the data literally in the HTML — titles, prices, descriptions as text? If yes, parse it. Second, is there a JSON blob in a script tag — __NEXT_DATA__, window.__INITIAL_STATE__, __NUXT__, or application/ld+json? If yes, extract it. Third, only a mount point and a bundle URL? Then the data is behind a network call and you're on rung two.

Step three: grep for framework signatures. curl -s URL | grep -oE '__NEXT_DATA__|__NUXT__|__INITIAL_STATE__|application/ld\+json'. Each signature points to its data: __NEXT_DATA__ is Next.js, __NUXT__ is Nuxt/Vue, application/ld+json is structured data — often product data, often a gift.

Step four: compare the server's response to a browser-like agent. Some sites SSR for crawlers and CSR for browsers by sniffing the User-Agent. Compare curl -A "Mozilla/5.0" with your default curl agent; differing HTML means user-agent-based content negotiation.

Step five: watch the network traffic. With the raw HTML empty, reload with the Network tab open and filter by XHR/Fetch. The requests the page makes to fill itself are the data layer. Write down the endpoints — that's your rung-two target.

The detection step as a script, so you never do it by hand again:

import json, re, sys, requests

def detect(url, headers=None):
    headers = headers or {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"}
    resp = requests.get(url, headers=headers, timeout=15)
    html = resp.text

    # 1. Is the data literally in the HTML?
    text_len = len(re.sub(r"<[^>]+>", " ", html).strip())
    print(f"status={resp.status_code} bytes={len(html)} visible_text={text_len}")

    # 2. Is there an embedded JSON blob?
    for key in ("__NEXT_DATA__", "__NUXT__", "__INITIAL_STATE__", "__APOLLO_STATE__"):
        m = re.search(rf'<script[^>]*id=["\']?{re.escape(key)}["\']?[^>]*>(.*?)</script>', html, re.S)
        if m:
            blob = m.group(1).strip()
            try:
                data = json.loads(blob)
                print(f"EMBEDDED_JSON {key}: {len(blob)} bytes, keys={list(data)[:6]}")
                return {"embedded": key, "data": data, "html": html}
            except json.JSONDecodeError:
                print(f"EMBEDDED_JSON {key}: found but not JSON")

    # 3. Structured data?
    ld = re.findall(r'<script type="application/ld\+json">(.*?)</script>', html, re.S)
    if ld:
        print(f"JSON_LD: {len(ld)} blocks")

    # 4. Mount point only?
    if re.search(r'<div id="(root|app)">\s*</div>', html):
        print("SHELL_ONLY: content is client-rendered, look for an XHR/API")
    return {"embedded": None, "html": html}

detect("https://target.example/product/123")

Run it, read the output, and you know which rung to start on. In real projects I run this against every URL pattern before writing extraction code, and it pays for itself in the first hour.

A concrete detection walkthrough

Let's do this end to end on a plausible target so the decision flow is unambiguous. Target: https://retailer.example.com/products?category=audio. First, the raw bytes:

curl -s https://retailer.example.com/products?category=audio -o page.html
wc -c page.html
grep -o '__NEXT_DATA__\|__NUXT__\|__INITIAL_STATE__\|root\|app' page.html | sort | uniq -c

The file is 1.9KB; the grep shows a single id="app" match. Classic shell: a Vue mount point, a bundle script, a title tag. The audio category page contains no audio products.

Next I reload with the Network tab open, filter by Fetch/XHR, and see a request to https://retailer.example.com/api/v3/products?category=audio&page=1&sort=popular. Copy as cURL shows a plain GET with a User-Agent, an Accept: application/json, and a Referer of the products page. I reproduce it in Python:

import requests

resp = requests.get(
    "https://retailer.example.com/api/v3/products",
    params={"category": "audio", "page": 1, "sort": "popular"},
    headers={
        "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/126 Safari/537.36",
        "Accept": "application/json",
        "Referer": "https://retailer.example.com/products?category=audio",
    },
    timeout=15,
)
data = resp.json()
print(data["total"], data["items"][0]["title"])

It returns JSON with a total, a list of items, and a next cursor. Time from first curl to working extraction: about twelve minutes. Compute: a few hundred milliseconds. No browser launched.

The pattern to internalize: detection is not research, it's extraction. Once you know the API contract, the API is your scraper. The browser only discovers that contract — a one-time cost of a couple of DevTools sessions, never a per-page cost.

The decision ladder: four ways to scrape a JS page

Work down the ladder from cheapest and least detectable to most expensive and most detectable. Each rung is a strict superset of the ones above it, so never skip ahead.

The four rungs with ballpark per-1k-page costs and latencies. Rungs one and two handle most JavaScript-rendered sites with zero per-page cost and the lowest detection footprint.

Rung 1: the data is in the HTML or embedded JSON

This is the free win: server-rendered pages, statically generated pages, and — critically — framework pages that embed their data for hydration. Next.js, Nuxt, and Gatsby ship the initial page state in a script tag so the client can hydrate without re-fetching. That tag is a complete JSON dump of the page's data, sitting in the raw HTML.

The blob lives in <script id="__NEXT_DATA__" type="application/json"> (Next.js), __NUXT_DATA__ (Nuxt), window.__INITIAL_STATE__ = {...} (Redux), window.__APOLLO_STATE__ (Apollo). All parse with a regex plus json.loads. The tricky part is nesting: useful data is often several levels down, so you walk the structure. Here is a working extractor:

import json, re, requests

def extract_next_data(url):
    html = requests.get(url, headers={"User-Agent": "Mozilla/5.0"}).text
    m = re.search(
        r'<script id="__NEXT_DATA__" type="application/json"[^>]*>(.*?)</script>',
        html, re.S)
    if not m:
        raise ValueError("no __NEXT_DATA__ on this page")
    return json.loads(m.group(1))

def walk(obj, path=""):
    """Yield (path, value) for every leaf in a nested dict/list."""
    if isinstance(obj, dict):
        for k, v in obj.items():
            yield from walk(v, f"{path}.{k}")
    elif isinstance(obj, list):
        for i, v in enumerate(obj):
            yield from walk(v, f"{path}[{i}]")
    else:
        yield path, obj

data = extract_next_data("https://retailer.example.com/product/123")
for path, value in walk(data):
    if "price" in path.lower() or "title" in path.lower():
        print(path, "=>", value)

The walk function is the workhorse. You don't need the schema in advance — walk every leaf, print the interesting paths, then hard-code the ones that matter. When the site ships data as data.props.pageProps.product, you'll see it in the output immediately.

The failure mode on rung one is a site that embeds only part of the data — the product skeleton, but not reviews, stock, or price history. When embedded JSON is incomplete, the completion lives behind the API, and that's rung two.

Rung 2: find and call the site's JSON API

If the raw HTML has nothing and embedded JSON is absent or incomplete, the page fills itself from network calls. Those calls are your data source — finding them is a DevTools exercise, not an engineering project.

Open DevTools, check "Preserve log", reload, and filter by XHR/Fetch. You'll see the page shell, analytics beacons, and — the ones you care about — JSON endpoints like /api/v3/products, /graphql, /__data/json/.... The response body that contains the on-screen content is your endpoint.

Now reproduce that call outside the browser. Copy as cURL and inspect what it needs: the URL, the query parameters, and the headers. Most JSON APIs need very few headers — often just a User-Agent and Accept: application/json. Some want a Referer, a custom header like X-Requested-With, an API key, or a CSRF token set by an earlier page load. Start with bare requests.get and add headers only until it works; every header you don't send is one you don't have to keep fresh.

For pagination, look at how the app itself paginates. REST APIs usually take ?page=2 or ?offset=20&limit=20; some take an opaque cursor token you carry from one response to the next. Both are trivially loopable, covered below.

Fetch-based SPAs make plain fetch() calls that show up identically in the Network tab. GraphQL SPAs POST to a single /graphql endpoint with a JSON body. Some SPAs push data over a WebSocket — the live-update channel for tickers, chats, and live scores. You can connect to the same URL and subscribe to the same messages. Edge case, but the same discovery process.

The key realization: the app's API is already a perfectly engineered scraping interface — clean JSON, paginated, fast, and exactly what the site itself uses. Your scraper is one HTTP client among many. The only question is whether the site protects it, which I cover later.

Rung 3: use a render proxy when you don't want to run browsers

A render proxy (also called a rendering API, JS rendering API, or headless-browser-as-a-service) is a managed service that runs a headless browser for you and returns the fully rendered HTML. You send it a URL, it loads the page in a real browser in its own infrastructure, waits for the JavaScript to run, and hands you back the finished DOM. Some services return raw HTML, some return JSON with the extracted body, some run a JavaScript snippet you pass in.

When do you choose this over running Playwright yourself? Three situations: when you need rendering for a small slice of a larger crawl and don't want to operate a browser farm; when you're on a deployment where browsers fit poorly — a locked-down serverless function, CI, a laptop; and when the target needs a browser's anti-detection posture that the service already maintains — stealth patching, fingerprint handling, proxy rotation — so you inherit that work for free.

The math is straightforward, and it gets the full treatment in the render-proxy section below: a managed render runs about $1 to $5 per 1,000 pages, while a self-hosted box does roughly 25,000 pages an hour for $30 to $60 a month. The build-versus-buy line lands around a few thousand rendered pages a day.

The detection tradeoff is subtle. A render proxy's fingerprint is closer to a human's than a bare requests client, but its IPs are shared across all customers, and bot-defense vendors fingerprint and mark those ranges. A proxy fails where the target blocks the provider's ASN — common for high-value targets. You can often see it coming: if your own Playwright loads the page fine but the proxy returns a challenge wall, the target is range-blocking the provider, not rendering-detecting you.

Rung 4: run a headless browser when nothing else works

A headless browser executes the page's JavaScript faithfully: it runs the bundle, fires the network calls, waits for the DOM, and returns the rendered document. It is the only tool that handles every case — content locked behind interactions, scroll-dependent infinite scroll, timer-driven state, canvas-rendered charts, login flows, bot checks that need a real browser environment.

It is also the most expensive and detectable tool. A browser process is a real application: megabytes of parsed JavaScript, a layout engine, a compositor, a V8 heap — roughly 10 times the CPU and RAM of a plain HTTP client per unit of work, and roughly 10 times the latency per page. Headless Chromium leaks a small family of signals — navigator.webdriver, absent plugins, a SwiftShader WebGL renderer, a default viewport — and every bot-defense product checks for exactly those. You can patch most of them, but patching is a treadmill that runs forever.

The correct posture on this rung is ruthless minimalism. Share one browser process across many pages — a browser holds dozens of tabs, each a fraction of the cost of a fresh browser. Close tabs when done; memory leaks are the #1 reason browser farms die. Prefer API interception over DOM scraping: let the browser run the page, capture the JSON responses it makes, don't parse the rendered DOM. And if you can, don't run the browser for most pages at all — the full playbook, waiting and stealth included, is in Headless browser scraping with Playwright. Here: when you're on this rung, the browser is a means to discover or capture data, and the moment you've captured it, stop rendering.

How SPAs work, and what it means for you

To scrape client-rendered sites well, you need a mental model of what the browser is doing, because every technique in this guide is an answer to one of its steps.

A single-page application is a JavaScript program that owns a DOM container. On every load: the server returns the shell; the browser fetches and executes the bundle; the bundle reads its initial state (embedded in the HTML or fetched from an API), builds a virtual tree of what the page should look like, and renders that tree into the container. Navigation is not a page load — the bundle calls an API, updates state, re-renders the changed portion. Scroll to the bottom and the bundle fires the next-page API call. From the network's perspective, the SPA is a conversation between the browser and a set of JSON endpoints; the HTML document is a stage, not the show.

Hydration deserves a warning for scrapers. On many Next.js and Nuxt sites, the server pre-renders the HTML, so the raw response does contain content; then the bundle runs and replaces that DOM with a client-rendered copy of the same data. That looks like a gift — the content is in the raw HTML. The catch is that hydration can be partial: below-the-fold, tabbed, modal, and post-load content is fetched later. "The HTML has content" is not "the HTML has everything."

The empty-first-render explains one common failure: reading a CSR page too early. Even in a real browser, the DOM is genuinely empty for the first few hundred milliseconds while the bundle boots and its first fetch returns. Every "why does my scraper see an empty page" bug on a JS site is this race. The fix is never a fixed sleep; it's waiting for the specific content to appear.

The most important practical consequence: the SPA is not a website, it's a client for a data API. The frontend is the reference implementation of that API's contract. Your scraper doesn't need to reproduce the rendering — it needs the API's response, the same data the frontend renders. It's cheaper, faster, more robust, and immune to CSS changes and redesigns.

Finding the API that powers a view

The discovery workflow, made repeatable. When you identify an XHR that carries the content, inspect three things: the request URL and query string, the request headers, and the response shape. Write down the shape — the top-level keys and the array you actually want — because it becomes your extraction contract.

Now reproduce it with requests, starting minimal and adding only what breaks. Strip the copied cURL to its skeleton rather than pasting all twelve headers — every header you paste couples you to the site's current frontend. Here is the full rung-two pattern, including a session cookie and an enforced Referer:

import requests

session = requests.Session()
session.headers.update({
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/126 Safari/537.36",
    "Accept": "application/json",
})

# Some APIs require a Referer or an Origin that matches the page.
# Fetch the page first so cookies and referer-dependent tokens are set.
page_url = "https://retailer.example.com/products?category=audio"
session.get(page_url, timeout=15)

api_url = "https://retailer.example.com/api/v3/products"
params = {"category": "audio", "page": 1, "sort": "popular"}
resp = session.get(api_url, params=params, headers={"Referer": page_url}, timeout=15)
resp.raise_for_status()
payload = resp.json()

print(payload.keys())          # {'items': [...], 'total': 1234, 'next_cursor': '...'}
for item in payload["items"]:
    print(item["id"], item["title"], item["price"])

Note what this script does not do: it doesn't run a browser, it doesn't parse HTML. It reproduces the app's own data request with minimum viable fidelity, and it keeps working through frontend redesigns because the API contract is more stable than the DOM.

The most common rung-two blockers, and their fixes:

The API needs a token from the page. Usually a CSRF token or per-session nonce embedded in the HTML or set as a cookie. Fetch the shell first, regex the token out, send it along — a warm-up request, then the data request.
The API is keyed to a session. Keep a requests.Session, fetch the shell once per session, reuse the cookies.
The API needs a signed URL. Some CDNs sign API URLs with an expiring hash of the path. If it expires in minutes, warm up immediately before each batch; if it's derived from client-side state, you're pushed toward rung four.
The API rate-limits. It sits behind the same limiter as the site: per-domain delay, retry with backoff on 429, honor Retry-After.

GraphQL: the JSON API with a schema

A large and growing share of JavaScript sites — nearly every modern frontend team's default — serves data through GraphQL. For a scraper that changes two things: the endpoint is almost always a single URL, and the query is a POST body rather than URL parameters.

Finding it: in the Network tab, look for a POST to /graphql, /api/graphql, /gql, or any path containing graphql. The body is JSON with a query field and maybe a variables field. That exact query is your contract — copy it verbatim and replay it from your own code.

The gift of GraphQL is introspection. When a server has introspection enabled, you can ask it for its entire schema — every type, every field, every argument — and then construct queries that return precisely the fields you want. One request:

import requests, json

url = "https://target.example.com/graphql"
query = """
query IntrospectionQuery {
  __schema {
    types {
      name
      kind
      fields { name args { name type { name } } }
    }
  }
}
"""
resp = requests.post(url, json={"query": query}, headers={
    "User-Agent": "Mozilla/5.0",
    "Content-Type": "application/json",
}, timeout=15)
schema = resp.json()
# Find the type you care about, e.g. Product, then its fields:
for t in schema["data"]["__schema"]["types"]:
    if t["name"] == "Product":
        print([f["name"] for f in t["fields"]])

When introspection is disabled — increasingly common, since it's a known scraping and security exposure — replay the exact query from the Network tab instead. It includes the field set the frontend wants, more than you need but workable. Watch for two things. First, persisted queries: some sites send a hash instead of a full query (extensions: {persistedQuery: {sha256Hash: "..."}}) and refuse unknown hashes. Replaying a hash works while it lasts; a hash tied to a client version may expire. Second, the variables object: the app passes cursor, filter, and pagination state there, and that's where your pagination loop lives.

GraphQL does not change the anti-bot rules. The endpoint is behind the same defense as the rest of the site, and heavy scraping of one POST-to-a-single-URL is more visible than REST, because a rate limiter can watch it with zero pattern matching. Keep the politeness rules tight.

Infinite scroll and cursor pagination

Infinite scroll is the SPA's answer to pagination: the app fires a new request when you reach the bottom of the viewport. From a scraper's perspective, that's not a rendering problem, it's a pagination problem with a discoverable loop. The loop lives in the API, not in the scrollbar.

Two pagination models dominate. Numeric: ?page=2, ?page=3, with a total_pages or has_more field. Cursor-based: the response carries an opaque token — next_cursor, cursor, after — which you pass back in the next request. Cursor pagination is the modern default because it's stable under concurrent writes; the cursor is opaque by design, you never construct it, you just carry it. Here is a complete loop that pulls every page of a catalog:

import requests

session = requests.Session()
session.headers.update({
    "User-Agent": "Mozilla/5.0",
    "Accept": "application/json",
    "Referer": "https://retailer.example.com/products?category=audio",
})

url = "https://retailer.example.com/api/v3/products"
cursor = None
page = 0
while True:
    params = {"category": "audio", "limit": 100}
    if cursor:
        params["cursor"] = cursor

    resp = session.get(url, params=params, timeout=15)
    resp.raise_for_status()
    payload = resp.json()

    for item in payload["items"]:
        save(item)          # your writer — DB, JSONL, whatever

    page += 1
    cursor = payload.get("next_cursor")
    if not cursor or not payload.get("has_more", True):
        break
    print(f"page {page}: {len(payload['items'])} items, cursor={cursor[:16]}...")

Two details. First, the loop terminates on the absence of a cursor, not on an empty page — a full page with a cursor means keep going. Second, the limit parameter: the app may cap it at 20 or 50 while the API happily accepts 100 or 200. Probing limit upward once is always worth it — fewer requests, smaller detection footprint, faster crawl. And "scroll to load" is nothing but the trigger for exactly this loop; you can reproduce it without ever touching the DOM.

The one case where scrolling genuinely matters: sites that lazy-load images and data by scroll position with no cursor, computing "next" from the current DOM. Those are rare, and they're a rung-four job. Everything else, loop the cursor.

Headless browsers: the brief version

I'm not duplicating the Playwright deep dive — it's here, covering waiting strategies, stealth basics, and API interception in full. What belongs here is the decision math: when this rung is the only option, what it costs, and how to minimize the damage.

The cases that genuinely require a browser are narrower than people think: content behind a click or a sequence of interactions; timer- or animation-derived state; canvas-rendered data; login walls where the credentials are yours and the terms permit it; bot checks that need a real browser environment; data computed client-side from a blob you can't reconstruct. Notice how few of those describe "the product catalog." Most catalogs are rung one or two.

The resource numbers. A plain HTTP client fetches 5 to 20 pages per second per core. A headless browser manages roughly 2 to 6 rendered pages per second per core if you share one browser across tabs; launching a fresh browser per page gets a fraction of that and exhausts memory at a few hundred pages. RAM is the binding constraint: each tab's renderer holds 80 to 300MB. At a million pages, a browser-based crawl takes 3 to 10 times the wall-clock time and 10 times the compute of the equivalent API crawl, and leaves 10 times the fingerprint on the target's logs.

Three habits keep browser scrapes alive: share one browser, wait on selectors never on sleeps, and intercept the site's own API responses instead of scraping the DOM. That last one is the difference between fragile and durable — you render once to get the data layer, the API does the heavy lifting, and the browser retires.

Order-of-magnitude latency per page. The difference between 0.2 seconds and 5 seconds is the difference between 400k pages a day and 17k pages a day on the same hardware.

Render proxies and rendering APIs: the build-versus-buy math

The render-proxy rung gets the full treatment because it's the one people pick without doing the arithmetic. A rendering API is a hosted headless browser: you POST a URL (plus, optionally, a script to run and options like "wait for selector", "block images", "use a residential IP"), and the service returns the rendered HTML or a JSON payload. You pay per render and never touch a browser process.

The service is a rendering layer, not a parsing layer, and that distinction is where people waste money. Sending a plain server-rendered page through a render proxy to "get the content" burns a credit on data you could have parsed with BeautifulSoup. Use a render proxy strictly for genuinely client-rendered pages — the ones where rungs one and two failed. A common architecture: route everything through a fast HTTP client, and only the residual failures — the 5 to 15 percent that genuinely need rendering — through the proxy. That mix keeps average cost per page at pennies.

The cost table, in round numbers that hold in 2026. At the low end, a render credit costs about $1 to $2 per 1,000 renders for basic HTML. Add residential proxies, screenshots, custom JS execution, or higher concurrency and you climb toward $5 to $10 per 1,000. Compare self-hosted Playwright: a $50/month box rendering 25,000 pages an hour costs about $0.002 per 1,000 pages of compute — essentially free — plus your engineering time.

Latency is the other axis. A render proxy adds a network hop to a browser render, so plan on 1.5 to 3 seconds per page plus queueing at peak. Concurrency matters more than per-render latency — a good provider gives you parallel renders, which is what moves throughput. The failure modes are those of any managed service: provider IP ranges that targets block, and quality variance — two identically-priced services can differ 3x on success rate against the same Cloudflare-protected target. Test a few dozen pages before committing to a vendor.

The deeper question is whether to build the renderer at all. Under a few thousand rendered pages a day, buying is cheaper than your own time. On a sustained crawl, the decision is bigger than the renderer — queues, caching, politeness, dedup, anti-detection — and the crawler-API guides at the end of this post cover that arithmetic.

Detection and the anti-bot reality for JS sites

The uncomfortable truth the "just call the API" advice runs into: JavaScript-rendered sites and aggressive bot defense are correlated, because the modern default stack — a SPA behind Cloudflare — includes both the empty-shell problem and a challenge wall in front of it. The JSON API that powers the page is frequently behind the same defense, so rung two can hit a wall rung one would have hit too. Here's what the wall looks like and what it means.

The most common wall on JS-heavy sites is Cloudflare's managed challenge — a 403, or a 200 whose body contains cf-browser-challenge, cf_chl_opt, or a challenge-platform script, or a cf-mitigated: challenge header. Three flavors: the classic JavaScript challenge (a proof-of-work that drops a cf_clearance cookie), the managed challenge (which can escalate to a CAPTCHA), and "Turnstile" (invisible until it decides you're suspicious). For a scraper the practical question is whether a real browser pass solves it: a bare requests client fails all of them; a real browser often passes the JS challenge by just loading; a headless browser sometimes triggers the CAPTCHA; a render proxy with good IP reputation passes most often.

Then there's TLS fingerprinting, which applies even without a visible challenge. The TLS handshake — cipher-suite order, extensions, negotiation style — differs between curl, Python's requests/urllib3, Go's default HTTP client, and a real browser, and bot-defense systems fingerprint it before your request reaches the application. That's why "I added a realistic User-Agent and it still blocked me" is so common: the User-Agent is cosmetic, the TLS fingerprint is what's checked. If you're blocked at the transport layer, changing the User-Agent won't help; you need a client with a browser-like TLS shape (curl_cffi with the right impersonation profile) or a real browser — a rung-four answer.

The JSON API behind the wall deserves a specific test before you commit to a tool: the wall in front of the page does not necessarily protect the API. Sites commonly put SPA HTML behind a challenge while leaving the JSON endpoints open — the frontend calls them with a session cookie from passing the page challenge, but the endpoints don't re-verify every request. The cheapest experiment in scraping: curl the API URL directly. JSON back? You've skipped the wall. A challenge back? You're on the rung-four or render-proxy path.

The realistic success-rate numbers from my own projects, as planning assumptions rather than promises:

Planning numbers: a bare DOM scrape through a browser is the least reliable because of rendering races and selector rot. The resilient architecture combines a fast API path with a browser fallback for residual failures — how you get the rightmost column.

The lesson is structural: the resilient scraper has a fallback path. Run the cheap path, count the failures, route the failures to the expensive path. That's how you get 95 percent overall success while still paying rung-one prices for most of your volume. A scraper with no fallback dies at the first 403.

War stories

Three real jobs to make it concrete. Changed targets, honest numbers.

A Next.js retailer. The job: 400,000 product pages from a fashion retailer, prices and stock included. First curl: __NEXT_DATA__ carried the product object — title, images, description — but price and stock were populated client-side after hydration. DevTools showed /api/catalog/product/{id}?ts=... returning a small JSON with price, stock, and a delivery estimate. The scraper fetched the raw HTML (rung one) for the static data and made one API call per product (rung two) for the dynamic fields. Two requests per product, no browser, ~0.4 seconds per product, and the catalog ran on a single box over a weekend.

A Vue SPA with a hard wall. A classifieds site that was pure Vue: raw HTML was a <div id="app"> and nothing else. No embedded state. But the Network tab showed a GraphQL endpoint at /gql — and introspection was still enabled. I introspected the schema, found the ListingSearch type, and built a query returning exactly id, title, price, and location. The API was rate-limited to 30 requests per minute per IP, so the crawl ran at 30 per minute with exponential backoff on 429s. The city's listing inventory — about 90,000 items — came down in two days of polite crawling. The DOM never mattered.

A React infinite-scroll catalog. A design marketplace with no pagination UI and no visible API — the first Network pass showed only analytics beacons. The trick: the page fetched JSON files from a static CDN path, one per "page," named by a cursor that appeared in the DOM as an attribute on the feed container. The first file contained nextCursor; each subsequent file was at /data/feeds/{cursor}.json. It looked like rendered content because the fetch was instant and local. Once I found the pattern, the scrape was a straight cursor loop over static JSON files — no rendering, no browser, effectively unbannable because the CDN serves the files to everyone. The cursor loop code above handled it unchanged.

Each ended with the same moral: the browser was a diagnostic tool used once in DevTools; extraction ran on rung one or two. The one job that genuinely required a browser was a dashboard computing its charts client-side from a WebGL canvas — no embedded state, no JSON endpoint, nothing but rendered pixels. Playwright captured it, and API interception found the underlying chart data on the second pass. Everything else, the ladder handled.

The full comparison table

Plain requests covers open and lightly-defended JS sites at any volume. Render proxies fit small-to-medium jobs against tougher walls. Self-hosted Playwright earns its cost for sustained volume, hard walls, or interaction-heavy pages.

Here is the comparison in a table, the way I'd hand it to an engineer deciding today:

Approach	Speed per page	Detection risk	Cost per 1k pages	When to use
Parse HTML / embedded JSON	~150ms	Lowest	~$0.00	Data is in the raw response (`__NEXT_DATA__`, JSON-LD, SSR)
Call the site's JSON API	~200ms	Low	~$0.00	The app fetches content from XHR/fetch/GraphQL
Render proxy / rendering API	1-2s	Medium (shared IPs)	~$1-5	Small/medium jobs, no browser ops, residual JS-only pages
Headless browser (self-hosted)	3-8s	Highest (fingerprintable)	~$2-8 in compute	Interactions, canvas, hard walls, sustained large volume
API + browser fallback mix	~0.3s avg	Medium	~$0.05-0.30	Production scraping at any scale — cheap path plus residual fallback

Key takeaways

Detect before you decide: disable JS, curl the raw bytes, and grep for __NEXT_DATA__ / __INITIAL_STATE__ / JSON-LD before you consider a browser.
The ladder, in order: embedded JSON in the HTML, the site's own JSON API, a render proxy, then a headless browser. Each rung is a strict superset of the ones above it, and each costs more.
A client-rendered SPA is a client for a data API. The frontend is the reference implementation of that API's contract. Reverse-engineer the contract, don't reproduce the rendering.
DevTools Network is your API discovery tool. Find the XHR that fills the page, copy it as cURL, strip headers until it breaks, then call it with requests.
GraphQL sites expose a single endpoint and often a full schema via introspection. Replay the app's query, or introspect your own.
Infinite scroll is cursor pagination in disguise. Loop the cursor, don't scroll the DOM.
Browsers cost roughly 10x the CPU, RAM, latency, and detectability of an HTTP client. Share one browser, wait on selectors, and prefer capturing the site's API responses over scraping the DOM.
A render proxy is buying the browser layer. It wins for small and medium jobs; self-hosting wins on sustained volume. Route only the residual JS-only failures through it.
The resilient scraper has a fallback path: run the cheap path, count the failures, send the failures to the expensive path. That's how you get 95 percent success at rung-one prices.

Conclusion

The empty <div id="root"> is not a wall, it's a sign: the site telling you where the data lives — one layer deeper, in the JSON the site itself fetches. Most of the time that JSON is either embedded in the page or served from an endpoint you can find in ten minutes with DevTools, and plain HTTP gets it for a fraction of the cost and detection footprint of a browser. The headless browser remains essential, but it's the last rung, not the first — a fallback for the residual cases where rendering is genuinely required.

Build your scraper as a ladder: the cheap path for the 85 percent, a render proxy for the small jobs you don't want to operate, Playwright for the hard floor. Test the cheap path first, always. And when you reach for the browser, read the Playwright guide first — the waiting and interception patterns there are the difference between a scrape that runs for years and one that dies on Monday.

Frequently Asked Questions

How do I know if a page needs JavaScript rendering to scrape?

Start with curl. Fetch the URL and check whether the raw HTML actually contains the data you want. If it does, you don't need a browser. If you get an empty div, a mount point, or only page chrome, the content is rendered by JavaScript and you need one of the techniques in this guide, starting with the embedded-JSON and API layers before ever considering a browser.

Can I scrape a JavaScript site without a headless browser?

Most of the time, yes. Check the raw HTML for embedded JSON like __NEXT_DATA__ or __INITIAL_STATE__, then look in DevTools Network for the JSON API the site calls to fill the page. Calling that endpoint directly with requests is faster, cheaper, and less detectable than running a browser. A headless browser is the last rung of the ladder, not the first.

What's the difference between client-side and server-side rendering for scraping?

Server-side rendering (SSR) produces HTML with the content already inside it, so plain requests can parse it. Client-side rendering (CSR) ships an empty HTML shell and a JavaScript bundle that fetches data and builds the DOM in the browser. For a scraper, SSR is basically free. CSR is why you get a div with nothing in it and why you need the techniques in this guide.

Is scraping a single-page app different from scraping a normal site?

Yes. A single-page app (SPA) loads one HTML page and then rewrites the view by calling APIs and swapping components without new page loads. That means the data is almost always reachable as JSON somewhere: either embedded in the initial HTML, or behind the API calls the app makes. Infinite scroll is just the app making more API calls as you scroll.

How much slower is a headless browser than plain HTTP?

Roughly 10 to 30 times slower. Plain requests parse a server-rendered page in 100 to 300 milliseconds. A render proxy takes about 1 to 2 seconds per page. A headless browser that runs the full JavaScript bundle, waits for network idle, and scrolls for infinite scroll typically takes 3 to 8 seconds per page. At a million pages, that is the difference between hours and weeks.

What is a render proxy or rendering API, and when should I use one?

A render proxy is a managed service that runs a headless browser for you and returns the fully rendered HTML, either directly or as a JSON payload. Use it when you don't want to operate a browser farm but a page genuinely needs rendering. The tradeoff is cost and latency: you pay per render and you eat a second or two per page, but you skip the infrastructure and detection work entirely.

What is __NEXT_DATA__ and why does it matter for scraping?

It's the JSON blob Next.js embeds in a script tag so the client can hydrate the page without re-fetching. Many Next.js sites put the entire page's data in there, which means you can scrape the content from the raw HTML with a regex or a JSON parse and never run JavaScript at all. Look for __NEXT_DATA__, __NUXT__ (Vue), or __INITIAL_STATE__ (Redux) in the page source.

How do I find the JSON API behind a JavaScript site?

Open DevTools, go to the Network tab, reload, and filter by XHR or Fetch. The API calls that fill the page will show up as JSON responses. Copy one as cURL, strip the headers to the minimum that works, and call it directly from your own code. Then walk the pagination the same way the app does — usually a page or cursor parameter.

Can I scrape a GraphQL endpoint on a JavaScript site?

Yes, and it's often easier than REST. Most GraphQL sites expose a single endpoint, usually /graphql or /api/graphql, that accepts a POST with a query. If introspection is enabled you can enumerate the entire schema. If it's disabled, watch the Network tab and replay the exact query the app sends. The same anti-bot rules apply as for any API.

Why do JavaScript sites trigger infinite scroll instead of pagination?

Because the app treats the page as a scrollable view, not a document. As you reach the bottom it fires another API request for the next page and appends the results. For scraping that's a gift: the pagination is a clean, predictable loop over the same cursor or page parameter. Find the cursor, and you don't need to scroll at all.

Keep reading

Found this useful? Cite it as: webscraping.space. “Scraping JavaScript-Rendered Pages: The 2026 Practical Guide.” https://webscraping.space/blog/scraping-javascript-rendered-pages. Published 2026-08-08.