URL: https://webscraping.spaceBest viewed at 1024×768

Node.js Published Aug 13, 2026 · 42 min read · 9,202 words

Web Scraping with Node.js: The Complete 2026 Guide

Everything for scraping with Node.js in 2026: why Node is a natural fit, the fetch/undici plus cheerio default stack, Playwright for JavaScript pages, p-limit concurrency, retries with backoff, and a complete runnable scraper that respects robots.txt.

Every scraping tutorial in Node shows you one page, one selector, one console.log(), and calls it a day. Then you point it at a real site and it breaks on page two, or gets a 403 on request thirty-seven, or saves the same record fourteen times. That is not a tutorial, that is a screensaver.

This is the guide I wish had existed. It is the complete Node.js pipeline, from "why Node at all" to a production-shaped scraper: choosing a target, reading robots.txt, fetching with the built-in fetch, parsing with cheerio, handling JavaScript pages with Playwright, running concurrent requests without getting yourself blocked, retrying with exponential backoff, and scaling from one process to a worker pool. Every code block is runnable against public sandbox sites, every number is honest, and every choice is explained — because the "why" is what survives when the site changes and your first script dies.

A note on positioning. This site has a complete Python tutorial and a deep dive on the requests library. I am not going to redo those. I am going to make the Node case directly, show you the same discipline in JavaScript, and point at the Python material wherever the lesson is language-agnostic.

Who this guide is for: you already write JavaScript, and you have been told to scrape something — a product catalog, a price list, a directory — and you would rather not stand up a Python environment to do it. By the end you will have a runnable scraper, a mental model of the toolchain, and a set of failure modes you can recognize before they cost you a day. The examples all run against public sandbox sites, so you can reproduce every number here this afternoon.

Why Node.js earns its place in scraping

Let me start with the honest pitch, because I have written scrapers in both worlds and I have opinions.

Async I/O is the native model. Node's entire runtime is built around an event loop that does not block on network. A Python requests.get() occupies a thread for the whole round trip. A Node await fetch() occupies nothing — the loop moves on and the response comes back to the same continuation. That one difference changes the shape of a scraper: in Node, fanning out 100 concurrent requests is a for loop and a Promise.all, not a thread pool. For I/O-bound work, a single Node process with modest memory can saturate a polite per-domain rate limit that would take a multi-threaded Python crawler to match.

The DOM is already your language. Scraping is CSS selectors and a tree of nodes. If you have ever written a frontend, you already know document.querySelector and element.textContent. Cheerio hands you that exact API on the server, and Playwright hands you a real browser DOM. Python developers have to learn a new selector dialect per library; you are just reusing a muscle you already built.

Headless browsers are first-class, not an afterthought. The two dominant headless-browser projects — Puppeteer (Chrome team) and Playwright (Microsoft) — are JavaScript libraries before they are anything else. Python uses ports of the same tools. The API you write in Node is the API the tool was designed around, which means fewer version-mismatch surprises.

The 2026 ecosystem is smaller and sharper than people think. The old "Node is only for web apps" story is stale. Between built-in fetch, node:sqlite for storage, and worker threads for CPU-bound parsing, the standard library now covers most of a scraper with no dependency tree at all. A production Node scraper can be one file with two dependencies.

Here is the trade, so I am not selling a one-sided story. Node's TLS stack is shared by every HTTP client, which makes TLS fingerprint detection a real problem (more on that in the failure-modes section), and the data-science side of the pipeline — cleaning, analyzing, shipping datasets — still belongs to Python. If your project is 20% scraping and 80% analysis, Python is probably the better home. If it is a crawl service, Node is genuinely great.

A word on team size and maintenance, because it shapes the choice in practice. A Node scraper is one language across your whole pipeline: the same developers who own your API can own your crawler, and the deployment story is just another Node service. Python offers deeper scraping libraries (Scrapy's crawl framework has no real Node analogue) but splits your stack. The rule I use: greenfield crawl service, pick Node; existing Python data team, stay in Python and treat the Python tutorial as your starting point.

Key takeaways

Use built-in fetch (undici) plus cheerio as the default stack; it covers most real scraping jobs with zero dependencies.
Check robots.txt before touching a domain, and encode the check into the scraper, not into your memory.
Fan out with p-limit at a polite cap — 5 to 10 concurrent requests per domain is a sane starting number — and never Promise.all a million URLs raw.
Retry transient errors with exponential backoff plus jitter, and honor Retry-After when the server sends it.
Reach for Playwright only when the data is JavaScript-rendered and no JSON API exists; a headless browser costs roughly 50x the resources of an HTTP client and is easier to detect.
CPU-bound cheerio parsing blocks the event loop; offload it to worker threads once pages get big or volume gets high.
Every Node HTTP client shares the same TLS fingerprint. If a WAF blocks you regardless of headers, the fix is Playwright or a scraping API, not a cleverer User-Agent.

The toolchain, mapped

The Node scraping ecosystem is smaller than Python's, and that is a feature. Here is the entire landscape worth knowing, in one grid.

Six tools cover the whole pipeline. The two that matter most are fetch/undici for transport and cheerio for parsing; everything else is a specialized answer to a specific problem.

Two of these need extra color. jsdom builds a real DOM in JavaScript and lets scripts run, but it is one to two orders of magnitude slower than cheerio and it does not fully execute modern pages anyway — if you need real rendering, you need a real browser, and jsdom is the worst of both worlds for scraping. Puppeteer versus Playwright is mostly a coin flip; I default to Playwright because the API is cleaner, the cross-browser support is real, and its auto-waiting saves you from hand-rolled waitForTimeout hacks. The rest of this guide uses that bias.

The rule that drives every choice below: static HTML is cheap, rendered HTML is expensive. Everything in the first half of this guide is about being cheap on purpose, and everything in the middle is about knowing exactly when you have to pay up.

Fetching with fetch and undici

Node 18 shipped global fetch, and by 2026 it is simply the default. It is built on undici, Node's own HTTP/1.1 and HTTP/2 client, which maintains a connection pool per origin and reuses keep-alive sockets aggressively. That pooling is a big deal for scraping: warm sockets mean you are not paying TLS handshake time on every request, and on the same origin the difference between warm and cold connections is often 100 to 300 milliseconds per request.

A first fetch is four lines:

const res = await fetch('https://books.toscrape.com/', {
  headers: {
    'User-Agent': 'webscraping-space-example-bot/1.0 (+https://webscraping.space)',
    'Accept': 'text/html,application/xhtml+xml',
    'Accept-Language': 'en-US,en;q=0.9',
  },
  signal: AbortSignal.timeout(15_000),
});
if (!res.ok) throw new Error(`HTTP ${res.status} for ${res.url}`);
const html = await res.text();

Three things here matter more than beginners expect. First, send a real User-Agent — one that identifies you and a URL a human could visit. The default node-fetch/1.0 or an empty UA is the fastest way to a 403. Second, always set a timeout. A scraper that hangs forever on a dead socket is a scraper that silently stops working. AbortSignal.timeout is the cleanest way to do this and it is built in. Third, check res.ok and throw loudly. A scraper that swallows a 403 and keeps going returns an empty file with a clean exit code, which is the worst failure mode there is.

If you want the undici layer directly — for streaming, for a custom pool, or for more control over redirects — it is the same shape:

import { request } from 'undici';

const { statusCode, body } = await request('https://books.toscrape.com/', {
  headers: { 'User-Agent': 'webscraping-space-example-bot/1.0' },
  maxRedirections: 3,
});
if (statusCode >= 400) throw new Error(`HTTP ${statusCode}`);
let html = '';
for await (const chunk of body) html += chunk;

For the vast majority of scrapers you will never need this direct layer — fetch is enough. But knowing it exists matters because it is the answer to the question "why does fetch reuse connections so well." It is undici doing the work.

Three protocol details that quietly break beginner scrapers. Compression: Node's fetch handles gzip and brotli transparently, so you get compressed responses for free — but if you ever drop to a raw socket layer or an HTTP client without auto-decompression, remember that a 500 KB page can be 40 KB on the wire, and decompress before you measure anything or your "page size" numbers will mislead you. Encoding: res.text() decodes UTF-8 by default. Legacy sites in latin-1 or windows-1252 will come back with mojibake; if you see replacement characters in your data, check the page's charset meta tag and decode with the right encoding, or your scraper will faithfully collect garbage. Redirects: fetch follows redirects by default, up to 20 hops. That is almost always right, but a redirect loop (site sends you to itself) will silently eat your timeout budget, so log the final URL on every page — a scraped URL that differs from the URL you asked for is a signal worth seeing.

There is also the streaming question. For very large responses — a 10 MB JSON export, a paginated API dumping everything — avoid res.text(), which buffers the whole body in memory. Stream it:

const res = await fetch('https://example.com/big-export.json', { signal: AbortSignal.timeout(60_000) });
const out = createWriteStream('./big-export.json');
Readable.fromWeb(res.body).pipe(out);
await finished(out);

One process scraping heavy pages while holding every body in memory at once will hit the heap far sooner than you expect. Stream the big ones; only buffer what you actually parse.

One honest comparison, because the Python material on this site goes deep on requests: the Python requests tutorial and the requests Session are the analogue of undici's pooled agent. The mental model is identical — reuse one client, keep connections warm, send real headers — even though the APIs differ.

Parsing with cheerio

Cheerio is a lean implementation of the jQuery core, tuned for the server: you hand it an HTML string and you get a $ function that selects with CSS selectors. It does not build a full DOM and it does not run scripts, and that is exactly why it is fast. This is the closest Node has to BeautifulSoup, and if you want the language-agnostic parsing theory, the BeautifulSoup guide covers selectors and tree-walking in depth — the same skills transfer directly.

import * as cheerio from 'cheerio';

const $ = cheerio.load(html);

const books = [];
$('article.product_pod').each((_, el) => {
  const title = $('h3 a', el).attr('title');
  const price = $('p.price_color', el).text().trim();
  const inStock = $('p.instock.availability', el).length > 0;
  books.push({ title, price, inStock });
});

A few cheerio habits that save real time in production:

Null-check every attr(). A selector that matches nothing returns undefined, not an error. If you push undefined into your data and only notice three thousand rows later, you have a corrupted dataset. Decide, per field, what a missing value means.
Use text() then strip() your own way. Cheerio's text() gives you the raw text with whitespace intact. Normalize at the edge — trim, collapse newlines — before the value reaches a file.
Scope with $('sel', el) or .find('sel') instead of global selectors. Scoping keeps your selectors honest about structure and survives minor page edits.

For the price field, a real scrape needs a little cleaning pass:

function parsePrice(raw) {
  // "£51.77" -> 51.77 (or null when the element is missing)
  const m = String(raw).match(/([0-9]+(?:\.[0-9]{1,2})?)/);
  return m ? Number(m[1]) : null;
}

Two more parsing techniques you will reach for constantly. Attribute scraping: much of the useful data on a page lives in attributes, not text — image URLs in src, links in href, canonical URLs in link[rel=canonical]. $(el).attr('href') and $(el).attr('src') are as core to scraping as text(). When you pull a relative URL, resolve it against the page with new URL(href, pageUrl).toString(); relative-link bugs are one of the most common silent corrupters of scraped datasets. Table scraping: $('table tr') gives you rows, and the discipline of header mapping — build a map from the th text to the cell index, then read cells by name — survives table layout changes far better than hard-coded column positions.

The most important habit, though, is testing selectors against saved HTML. Fetch a page once, write it to disk, and develop your selectors against that file. It costs zero network, it is reproducible, and it is the difference between a parser you trust and a parser that changes behavior every time the site breathes. This is the same rehearsal pattern the BeautifulSoup guide recommends, and it applies to cheerio unchanged.

The rendering cliff: cheerio vs jsdom vs Playwright

Here is the honest speed table that should drive every architecture decision you make. These are single-core numbers for a typical article page with roughly 350 KB of HTML, measured as pages processed per second:

Switching from cheerio to a headless browser costs you roughly 35x throughput on the same hardware. The rendering cliff is why the first question on every project is always: is this data really JavaScript-rendered?

These numbers are typical, not universal — page size and target speed move them — but the shape is stable across every site I have measured. The gap between "145 pages per second with cheerio" and "4 pages per second with Playwright" is the single biggest performance lever in the Node scraping stack, and it is a lever you pull with the architecture, not with faster code.

JavaScript pages: when and how to use Playwright

The full decision tree for JavaScript-rendered content — view source, then find the JSON API, then the browser — lives in the JavaScript-rendered pages guide, and it applies verbatim here. The short version: open DevTools, look in the Network tab for the XHR that actually delivers your data, and call that endpoint directly with fetch. You will be amazed how often a "single-page app" turns out to be one JSON endpoint and a template.

When the data genuinely only exists in the rendered DOM, Playwright is the right tool, and in Node it is the natural one. The full browser workflow is covered in the Playwright scraping guide, so here is the minimum viable shape:

import { chromium } from 'playwright';

const browser = await chromium.launch({ headless: true });
const page = await browser.newPage();

await page.goto('https://quotes.toscrape.com/js', {
  waitUntil: 'domcontentloaded',
  timeout: 30_000,
});
await page.waitForSelector('.quote'); // the JS has run; the quotes exist

const quotes = await page.$$eval('.quote', (els) =>
  els.map((el) => ({
    text: el.querySelector('.text')?.textContent?.trim(),
    author: el.querySelector('.author')?.textContent?.trim(),
  }))
);

await browser.close();

Three production habits worth adopting immediately. Close the browser — every leaked Chromium process eats hundreds of megabytes. Prefer waitForSelector over fixed sleeps — a sleep races the network and fails the day the site gets slower; a wait-for-selector passes exactly when the DOM is ready. Reuse one browser, not one page per request — create pages on demand from a single browser instance, and if you run concurrent scrapes, cap the browser tabs with p-limit just like you cap requests, because each tab is a real process worth of memory.

A note on detection: Playwright's headless Chromium is far more browser-like on the wire than any Node HTTP client — real TLS, real HTTP/2, real header order — which makes it the pragmatic answer to TLS fingerprinting too. But it is slower and more memory-hungry, which is why it is the last rung, not the first.

Two speed tricks that cut rendered-page crawl time dramatically. Block what you do not need. A product page loads images, analytics scripts, fonts, and ad SDKs you will never parse. Intercept and abort those requests, and a page that took 6 seconds to settle can render in 2:

await page.route('**/*', (route) => {
  const type = route.request().resourceType();
  if (['image', 'font', 'media'].includes(type)) return route.abort();
  return route.continue();
});

Intercept the JSON instead of scraping the DOM. When you know the page loads its data from an XHR, page.waitForResponse((r) => r.url().includes('/api/')) lets you capture the structured payload directly — no selector brittle against markup changes. The JavaScript-rendered pages guide covers this pattern in full; it is the highest-value trick in the rendered-page playbook because it turns a slow browser scrape into a fast structured fetch.

Concurrency with p-limit

Node's async model makes concurrency trivial to write and easy to get wrong. The wrong way is the way most tutorials show:

// dangerous: unbounded concurrency
const pages = await Promise.all(urls.map((u) => fetch(u).then((r) => r.text())));

If urls is a thousand long, this fires a thousand requests in the same instant. You will hammer the target into blocking you, you will trip rate limiters, and you will learn the word "backoff" the hard way. The right way is to cap concurrency with a tiny, battle-tested library:

import pLimit from 'p-limit';

const limit = pLimit(5); // at most 5 in-flight requests
const jobs = urls.map((url) => limit(() => fetchWithRetry(url)));
const pages = await Promise.all(jobs);

p-limit wraps each task so the pool never exceeds your cap, while Promise.all still waits for everything. That is the entire trick: a polite concurrency cap is what lets you be fast without being rude.

One refinement before you ship this. Plain Promise.all rejects the whole batch the instant any single request fails — one 503 and you lose every successful page in that batch. For a crawl, you almost always want per-task error isolation:

const results = await Promise.allSettled(jobs);
const ok = results.filter((r) => r.status === 'fulfilled');
const failed = results.filter((r) => r.status === 'rejected');
console.log(`ok=${ok.length} failed=${failed.length}`);

allSettled lets the batch finish, then you count, log, and decide what to retry — which is exactly the retry-bucket discipline you will need at scale. A scrape that reports "197 of 200 pages, 3 failed with 503s" is a scrape you can trust; one that crashes silently at page 17 is not.

What cap is right? It depends on the target, but a defensible default is 5 to 10 concurrent requests per domain, combined with a jittered delay floor so you are averaging around one request per 150 to 300 milliseconds. A site's robots.txt may specify Crawl-delay, and you should honor it — it is the floor your cap sits on top of. If you are only ever hitting a few pages, none of this matters; the moment you crawl hundreds or thousands, the cap is the difference between finishing and being blocked at page 40.

Here is the shape of the latency win, for a 1,000-page crawl against a target that answers each request in about 250 milliseconds:

Raising concurrency from 1 to 50 turns a 100-second crawl into a 5-second one, because the bottleneck is network latency, not CPU. The practical cap is never the machine — it is politeness and the target's patience.

The takeaway is not "crawl at 50." It is that the cost of concurrency is nearly zero, so the only reason to stay low is politeness. Pick a cap the target can absorb — 5 to 10 per domain is the range I use for real work — and you get 80 percent of the speed with none of the block risk.

Retries, backoff, and Retry-After

Networks are lossy, and targets are flaky. A scraper with no retry logic is one dropped connection away from a corrupted run. The standard answer is exponential backoff with jitter: retry quickly at first, slow down exponentially, and add randomness so your retries do not line up like a metronome.

async function fetchWithRetry(url, {
  fetchImpl = fetch,
  retries = 4,
  baseMs = 250,
  maxMs = 8000,
} = {}) {
  let lastError;
  for (let attempt = 0; attempt <= retries; attempt++) {
    try {
      const res = await fetchImpl(url);
      if (res.status === 429 || res.status >= 500) {
        // honor the server's own clock when it sends one
        const retryAfter = Number(res.headers.get('retry-after'));
        if (retryAfter > 0) await sleep(retryAfter * 1000);
        throw new Error(`HTTP ${res.status}`);
      }
      return res;
    } catch (err) {
      lastError = err;
      if (attempt === retries) break;
      const delay = Math.min(maxMs, baseMs * 2 ** attempt) + Math.random() * 250;
      await sleep(delay);
    }
  }
  throw lastError;
}

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

Three rules baked into that helper. Retry only transient errors — 429, 5xx, and network exceptions — never 4xx like 403 or 404, where retrying is either pointless or actively hostile. Honor Retry-After when the server sends it; that header is the server telling you its clock, and a scraper that ignores it is a scraper that gets banned. Add jitter on top of backoff so a fleet of your own retries does not thundering-herd the target.

The shape of the delays, with base 250 ms, jitter, and an 8-second cap:

Backoff doubles each attempt and the cap stops the waits from getting absurd. Jitter matters as much as the doubling: without it, every retrying client fires at the same instant and the 429 you were trying to escape gets worse.

The first bar in that chart is deliberately tiny: 250 ms means "the network hiccuped, let it breathe and try again." The last bar, 8 seconds, means "the server is genuinely struggling, do not be the reason it falls over."

The complete scraper

Time to assemble everything into one working scraper. The target is books.toscrape.com — the same public sandbox used in the Python tutorial, so you can run both and compare the two stacks side by side. It is a bookstore with categories, pagination, prices, and stock status, and it exists to be scraped.

The full script, runnable on Node 22.5+ (which gives us the built-in node:sqlite), in one piece:

// scrape-books.mjs  —  run with: node scrape-books.mjs
import { DatabaseSync } from 'node:sqlite';
import * as cheerio from 'cheerio';
import pLimit from 'p-limit';
import robotsParser from 'robots-parser';

const BASE = 'https://books.toscrape.com';
const UA = 'webscraping-space-example-bot/1.0 (+https://webscraping.space)';
const CONCURRENCY = 6;

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

async function getRobots() {
  const res = await fetch(`${BASE}/robots.txt`, {
    headers: { 'User-Agent': UA },
    signal: AbortSignal.timeout(10_000),
  });
  // a missing robots.txt is not permission to be rude; it just means no rules were published
  const text = res.ok ? await res.text() : 'User-agent: *\nAllow: /';
  return robotsParser(`${BASE}/robots.txt`, text);
}

async function fetchPage(url) {
  const res = await fetch(url, {
    headers: { 'User-Agent': UA },
    signal: AbortSignal.timeout(15_000),
  });
  if (!res.ok) throw new Error(`HTTP ${res.status} for ${url}`);
  return res.text();
}

async function fetchWithRetry(url) {
  let lastError;
  for (let attempt = 0; attempt <= 4; attempt++) {
    try {
      return await fetchPage(url);
    } catch (err) {
      lastError = err;
      if (attempt === 4) break;
      const delay = Math.min(8000, 250 * 2 ** attempt) + Math.random() * 250;
      await sleep(delay);
    }
  }
  throw lastError;
}

function parseBooks(html) {
  const $ = cheerio.load(html);
  const books = [];
  $('article.product_pod').each((_, el) => {
    const title = $('h3 a', el).attr('title');
    const priceRaw = $('p.price_color', el).text().trim();
    const priceMatch = priceRaw.match(/([0-9]+(?:\.[0-9]{1,2})?)/);
    const inStock = $('p.instock.availability', el).length > 0;
    if (!title) return; // skip malformed cards loudly-free: just log
    books.push({ title, price: priceMatch ? Number(priceMatch[1]) : null, inStock });
  });
  return books;
}

async function main() {
  const robots = await getRobots();
  const seed = `${BASE}/catalogue/page-1.html`;
  if (!robots.isAllowed(seed, UA)) {
    console.error(`robots.txt blocks ${seed} — refusing to scrape.`);
    process.exit(1);
  }

  const db = new DatabaseSync('books.db');
  db.exec(`CREATE TABLE IF NOT EXISTS books (
    title TEXT PRIMARY KEY,
    price REAL,
    in_stock INTEGER,
    scraped_at TEXT
  )`);
  const insert = db.prepare(`INSERT INTO books (title, price, in_stock, scraped_at)
    VALUES (?, ?, ?, ?)
    ON CONFLICT(title) DO UPDATE SET
      price = excluded.price,
      in_stock = excluded.in_stock,
      scraped_at = excluded.scraped_at`);

  // collect category listing pages (this site paginates at /catalogue/page-N.html)
  const limit = pLimit(CONCURRENCY);
  const listingPages = Array.from({ length: 50 }, (_, i) => `${BASE}/catalogue/page-${i + 1}.html`);
  const jobs = listingPages.map((url) => limit(async () => {
    if (!robots.isAllowed(url, UA)) return;
    const html = await fetchWithRetry(url);
    const books = parseBooks(html);
    for (const b of books) insert.run(b.title, b.price, b.inStock ? 1 : 0, new Date().toISOString());
    return books.length;
  }));

  const counts = await Promise.all(jobs);
  const total = counts.reduce((a, b) => a + b, 0);
  console.log(`done: ${total} books in ${db.prepare('SELECT COUNT(*) AS n FROM books').get().n} rows`);
  db.close();
}

main().catch((err) => { console.error(err); process.exit(1); });

Run it once and you have a SQLite file with every book on the site. Run it twice and ON CONFLICT ... DO UPDATE makes the second run idempotent — re-running never duplicates rows. That is the upsert pattern from the Python tutorial applied in Node, and it is the single most useful habit you can steal from that guide.

Notice what the robots gate actually does. It parses robots.txt, checks the seed URL, and refuses to run at all if that URL is disallowed — the politeness rule is encoded in the program, not carried in your head. It also handles the common case where a sandbox site serves no robots.txt at all (a 404): in that case there are no published rules, so we default to allow and keep our own politeness — the cap and the backoff — as the real protection. A robots.txt 404 is not a license to hammer; it is just an absence of instructions. For the full legal and ethical framing behind this posture, the robots.txt and ethics guide on this site is the right next read.

The pagination in this script is honest but deliberately crude. The site paginates at /catalogue/page-N.html, so I hard-coded a range of 50 — that is a stand-in, not a strategy. A real crawler discovers the "next" link from the page itself and stops when it disappears, which handles sites that add or remove pages between runs. That discovery loop, plus deduplication and frontier management, is precisely what the scraping at scale guide builds on.

And the cache is missing. For a one-shot run it does not matter, but during development you will re-run this parser dozens of times — fixing a selector, tweaking a regex — and every re-run re-hits the network. A disk cache keyed by URL turns those re-runs into zero-network affairs that finish in milliseconds. The blocking guide has the full cache pattern, and it applies in Node exactly as written there: check the cache, return if fresh, otherwise fetch and store.

Scaling the Node scraper

The scraper above is correct for a few hundred pages. When the target becomes a few hundred thousand, three bottlenecks show up in order: the event loop, the process, and the memory.

First bottleneck: CPU-bound parsing on the event loop. Cheerio is fast, but parsing a 1 MB HTML document is CPU work, and on the event loop it blocks all concurrent fetches while it runs. At hundreds of pages per second the effect is real. The fix is to move parsing off the event loop onto worker threads:

// parse-worker.mjs
import { parentPort } from 'node:worker_threads';
import * as cheerio from 'cheerio';

parentPort.on('message', (html) => {
  const $ = cheerio.load(html);
  const books = [];
  $('article.product_pod').each((_, el) => {
    books.push({ title: $('h3 a', el).attr('title'), price: $('p.price_color', el).text().trim() });
  });
  parentPort.postMessage(books);
});

The library that makes this painless is piscina, a worker-thread pool with a promise interface: send it a job, get back a result, and the pool keeps n workers busy across as many jobs as you post. The main thread stays free to do what it is good at — I/O — while parsing happens in parallel on every core.

Second bottleneck: one process. piscina scales you to one machine. When you outgrow a single process, the shape is the same as every production crawl system: a queue in the middle, N consumers on each side. BullMQ (Redis-backed) or even an in-memory queue like p-queue give you the producer-consumer pattern, and the whole architecture reduces to one diagram:

The scraper from earlier, redrawn as the production shape: a queue decouples discovery from fetching, capped pools keep politeness even under load, and a retry bucket keeps failures from killing the run. This is the same architecture behind the scraping at scale guide.

Third bottleneck: memory. A headless browser is where memory goes to die — each Chromium tab is hundreds of megabytes. If your scale requires rendering, budget for it explicitly: fewer concurrent tabs than you think, aggressive browser.close(), and a hard cap on the number of browsers per machine. The rendered-page services exist partly because this cost is real and painful to run yourself.

Three disciplines separate a crawl that scales from a crawl that melts. Backpressure: a queue with an unbounded producer and a slow consumer will happily enqueue ten million URLs while your workers fall behind; cap the queue and let the producer wait. In BullMQ this is the job-count limit; with p-queue it is your own loop. Idempotent storage: the upsert from the capstone is what makes a crashed run recoverable — re-run the crawl, and rows already stored simply update instead of duplicating. Without it, every mid-crawl crash corrupts your dataset. Observability: count pages fetched, pages parsed, pages failed, and the bytes moving per minute, and log all of them. The crawls that die at 3 a.m. die silently; the ones you can restart at 9 a.m. have logs that tell you exactly which URLs are still owed.

Common failure modes

The blocking guide on this site — web scraping without getting blocked — covers detection and mitigation thoroughly and language-agnostically. Here are the failure modes that bite Node developers specifically.

TLS fingerprinting. This is the big one, and Node is uniquely exposed. Every Node HTTP client — fetch, undici, axios, node-fetch — shares the same OpenSSL-based TLS stack, which means they all present the same JA3/JA4 fingerprint. A strict WAF like Cloudflare does not need to see your User-Agent; it sees a fingerprint that says "script" and returns a 403 before your request even lands. Python has curl_cffi to impersonate a browser's TLS handshake; Node has no mature equivalent. The honest workarounds are: keep your rate low and behavior human (the fingerprint matters less than the pattern), or use Playwright, whose real Chromium presents a real browser fingerprint. There is no header you can set to fix this, and any library that claims a "browser-like" header fix is selling you a placebo.

Rate limits and the 429. The polite answer is in the retry helper: honor Retry-After, back off with jitter, and — critically — mean it. A scraper that treats 429s as "try again in 400 milliseconds forever" is a scraper that becomes a ban. Back off, and if the site keeps saying no, stop for the day.

The DevTools trap. You open DevTools, see your field in the Elements panel, write a selector, and it finds nothing — because DevTools shows the rendered DOM and fetch gets the source. If a selector works in DevTools but not in your script, view-source first. Nine times out of ten the difference is JavaScript, and the fix is the JSON API, not a harder selector.

Silent partial failures. A scraper that skips a card here and a page there returns "success" with a hole in the middle of your dataset. The discipline is cheap and non-negotiable: log every URL you skip, count pages versus records, and compare totals across runs. The ON CONFLICT upsert plus a row count is how you catch it in Node.

Redirect loops and relative-link rot. A page that redirects to itself, or a scraper that stores href="product.html" without resolving it against the base URL, produces a dataset full of URLs that point nowhere. Log the final URL of every request, and resolve every relative link with new URL(href, pageUrl) at extraction time — the two-line fix that prevents weeks of cleanup.

Encoding mojibake. UTF-8 assumptions silently corrupt latin-1 legacy pages. If your text arrives with replacement characters or wrong accents, decode with the page's declared charset instead of trusting the default. It is a one-line fix that is miserable to discover later, because by then the bad text is already stored.

Session and cookie state. Some targets serve slightly different HTML to a cookie-less first request than to a returning visitor. If your scraper's output differs from what you see in a browser, the difference is often cookies and local storage, not JavaScript. Keep a cookie jar per domain (fetch does this for you when you reuse a client), and if a site sets a consent or region cookie, handle it before you start extracting — or your "clean" run will be quietly missing data.

The economics of rendered pages. Because headless rendering costs real money, it is worth seeing the per-1,000-pages numbers side by side before you commit to an architecture:

Self-hosting a cheerio crawler is essentially free; self-hosting Playwright costs about 4x more on compute. APIs trade money for managed rendering, proxies, and parsing — Keirolabs at $0.25/1k is the notable middle ground, with full markdown output that skips your parser entirely.

Read that chart as a decision rule, not a shopping list. If your pages are static, the gap between self-hosted ($0.02/1k) and any API is two orders of magnitude, and you should self-host. If you genuinely need rendering, the API price starts to look reasonable against the engineering and proxy cost of doing it yourself — and Keirolabs is one honest option among several, not the whole story. The wrong choice is the one you make without looking at the number.

What to build next

You have the full pipeline in one script. The fastest way to make it stick is a second project that stretches exactly one muscle further. Pick one:

A price monitor. Scrape a category on books.toscrape.com into SQLite every day, add a scraped_at column, and write a query that shows which prices moved between runs. This exercises the upsert, the timestamps, and your first data-over-time query — and it is the skeleton of every real price-tracking project.
A JavaScript-rendered target, done right. Take the Playwright example from this guide and run the full decision tree: view source, find the JSON API, call it directly. Practicing API interception on a sandbox means you will recognize the pattern instantly on a real site.
A crawl frontier. Replace the hard-coded page range with real link discovery: follow the "next" link, deduplicate URLs, and stop when the frontier is empty. That is your first taste of crawl logic, and the scraping at scale guide becomes directly relevant.
A politeness report. Add a request log — timestamp, URL, status, bytes — and write a script that summarizes your rate per domain per minute. You will learn more about your own scraper's behavior from this one log than from any chart in this guide.

Whichever you choose, keep the discipline from this guide: encode the robots check, cap concurrency, retry with backoff, and re-run your scraper twice in a row — if the second run hits the network at all, your cache is broken.

FAQ

Is Node.js good for web scraping? Yes, genuinely excellent. Node's event loop makes concurrent HTTP requests almost free, every frontend developer already knows the selectors and DOM that scraping is made of, and the best headless browsers (Playwright, Puppeteer) are JavaScript-native. For I/O-bound crawling, a single Node process handles hundreds of concurrent connections with a fraction of the memory Python needs.

What is the best Node.js web scraping stack? The default that covers most real jobs is global fetch (undici under the hood) plus cheerio. Fetch is built into Node 20+, so there is zero setup; cheerio parses static HTML with a jQuery-style API at hundreds of pages per second. Add p-limit for concurrency and a retry helper with backoff. Reach for Playwright only when the data is rendered by JavaScript and no JSON API exists.

Do I need Puppeteer or Playwright to scrape with Node? No. A headless browser costs roughly 50x the CPU and memory of a plain HTTP client and is far easier for a WAF to detect. First check whether your data is in the initial HTML, then open DevTools and look for the site's own JSON API. Only when both fail should you launch Playwright.

How do I scrape a website that loads content with JavaScript in Node? Three options, in order: find the JSON API the page calls and hit it directly with fetch; use Playwright to load the page and read the rendered DOM; or use a rendered-page scraping API. A real Playwright run costs about 4 pages per second per browser tab, so treat it as the last rung, not the default.

Is Node.js faster than Python for web scraping? For I/O-bound fetching, yes in practice: async concurrency lets one process fan out hundreds of requests, where Python needs threads or asyncio to reach the same parallelism. For parsing, cheerio and BeautifulSoup with lxml are in the same ballpark. Python pulls ahead on the data-science side of the pipeline and on TLS impersonation libraries like curl_cffi; Node has no equivalent yet.

How do I run requests concurrently in Node without getting blocked? Wrap each task with p-limit and cap concurrency at a polite number — 5 to 10 requests per second per domain is a sane starting point. Promise.all on a million URLs with no cap is how people burn their IP on request twenty. Pair the limit with jittered delays and exponential backoff on 429 and 5xx responses.

Should I use axios or fetch for scraping in Node? Either works; fetch is the better default because it is built in, uses undici's fast connection pool, and ships AbortSignal.timeout for free. Axios earns its place when you want interceptors, automatic JSON parsing, or per-request config that survives a codebase growing around it. They both share Node's TLS stack, so neither is more browser-like on the wire.

How do I avoid getting blocked when scraping with Node? Send a realistic User-Agent, keep a per-domain rate cap with jitter, read robots.txt, cache responses to disk, and retry transient errors with backoff. The bigger Node-specific trap is TLS fingerprinting: every Node HTTP client shares the same TLS stack, so a strict WAF can fingerprint you as a bot regardless of headers. For those targets, drop to Playwright or a scraping API.

Frequently Asked Questions

Is Node.js good for web scraping?

Yes, genuinely excellent. Node's event loop makes concurrent HTTP requests almost free, every frontend developer already knows the selectors and DOM that scraping is made of, and the best headless browsers (Playwright, Puppeteer) are JavaScript-native. For I/O-bound crawling, a single Node process handles hundreds of concurrent connections with a fraction of the memory Python needs.

What is the best Node.js web scraping stack?

The default that covers most real jobs is global fetch (undici under the hood) plus cheerio. Fetch is built into Node 20+, so there is zero setup; cheerio parses static HTML with a jQuery-style API at hundreds of pages per second. Add p-limit for concurrency and a retry helper with backoff. Reach for Playwright only when the data is rendered by JavaScript and no JSON API exists.

Do I need Puppeteer or Playwright to scrape with Node?

No. A headless browser costs roughly 50x the CPU and memory of a plain HTTP client and is far easier for a WAF to detect. First check whether your data is in the initial HTML, then open DevTools and look for the site's own JSON API. Only when both fail should you launch Playwright.

How do I scrape a website that loads content with JavaScript in Node?

Three options, in order: find the JSON API the page calls and hit it directly with fetch; use Playwright to load the page and read the rendered DOM; or use a rendered-page scraping API. A real Playwright run costs about 4 pages per second per browser tab, so treat it as the last rung, not the default.

Is Node.js faster than Python for web scraping?

For I/O-bound fetching, yes in practice: async concurrency lets one process fan out hundreds of requests, where Python needs threads or asyncio to reach the same parallelism. For parsing, cheerio and BeautifulSoup with lxml are in the same ballpark. Python pulls ahead on the data-science side of the pipeline and on TLS impersonation libraries like curl_cffi; Node has no equivalent yet.

How do I run requests concurrently in Node without getting blocked?

Wrap each task with p-limit and cap concurrency at a polite number — 5 to 10 requests per second per domain is a sane starting point. Promise.all on a million URLs with no cap is how people burn their IP on request twenty. Pair the limit with jittered delays and exponential backoff on 429 and 5xx responses.

Should I use axios or fetch for scraping in Node?

Either works; fetch is the better default because it is built in, uses undici's fast connection pool, and ships AbortSignal.timeout for free. Axios earns its place when you want interceptors, automatic JSON parsing, or per-request config that survives a codebase growing around it. They both share Node's TLS stack, so neither is more browser-like on the wire.

How do I avoid getting blocked when scraping with Node?

Send a realistic User-Agent, keep a per-domain rate cap with jitter, read robots.txt, cache responses to disk, and retry transient errors with backoff. The bigger Node-specific trap is TLS fingerprinting: every Node HTTP client shares the same TLS stack, so a strict WAF can fingerprint you as a bot regardless of headers. For those targets, drop to Playwright or a scraping API.

Keep reading

Found this useful? Cite it as: webscraping.space. “Web Scraping with Node.js: The Complete 2026 Guide.” https://webscraping.space/blog/web-scraping-nodejs. Published 2026-08-13.