webscraping.space
A field guide to scraping the modern web.
A no-nonsense, hands-on blog about extracting data from websites. From your first requests.get() to scrapers that survive at scale. We cover Python and Node.js, HTML parsing, headless browsers, anti-bot defenses, and how to stay on the right side of robots.txt. New posts whenever the spider finishes crawling.
What is web scraping?
Web scraping is the automated extraction of data from web pages. A scraper is a program that downloads HTML (or intercepts a site API responses), parses the structure, and pulls out the fields you care about. Prices, articles, contact info, product catalogs, research datasets.
In practice a modern scraper combines three pieces: a fetcher (an HTTP client or a headless browser), a parser (CSS or XPath selectors over the DOM), and a pipeline (dedup, storage, scheduling). The hard part is rarely the parsing. It is fetching reliably when the site rate-limits you, fingerprints your TLS handshake, or hides content behind JavaScript and bot detection.
If a human can see it in a browser, a scraper can capture it. The only real questions are how much load you place on the target and whether the site terms let you.
Latest from the log
FEATURED Building search infrastructure for 1M+ queries a month: the stuff nobody tells you
Web scraping ethics and robots.txt: the lines you don't cross
A clear, practical guide to scraping ethically and legally: reading robots.txt, Terms of Service, the CFAA and EU equivalents, login walls, personal data, and the difference between 'public' and 'allowed'.
ethicsrobots-txtlegaltosBuilding search infrastructure for 1M+ queries a month: the stuff nobody tells you
A deep, no-bullshit technical dive into building search infrastructure that handles over 1 million queries per month. Inverted index internals, posting list compression, WAND/MaxScore scoring, Go hot-path code, five-layer cache hierarchies, mmap and SSD reality, tiered segment merging, tail latency, and the non-obvious stuff that actually kills you at scale.
searchinfrastructureinverted-indexgoScraping at scale: queues, caching, and not getting banned
How to take a working scraper to millions of pages without melting the target or getting banned: work queues, bounded concurrency, on-disk caching, dedup, retries with backoff, and polite scheduling.
scalingarchitecturecachingconcurrencyBypassing anti-bot protections: TLS, fingerprints, and Cloudflare
A frank guide to anti-bot defenses and how scrapers get past them: TLS/JA3 fingerprinting, Cloudflare's challenge, PerimeterX, CAPTCHAs, residential proxies, and the libraries (curl_cffi, Camoufox) that actually work in 2026.
anti-botcloudflaretls-fingerprintproxiesHeadless browser scraping with Playwright (Python & Node.js)
When the data lives behind JavaScript, requests is not enough. This guide covers headless browser scraping with Playwright in Python and Node.js: stealth, waiting strategies, intercepting API calls, and not running the browser when you don't have to.
playwrightheadless-browserjavascriptnodejs
Where to start
- New to scraping? Start with Web scraping with Python requests.
- Hitting a Cloudflare wall? Read Bypassing anti-bot protections.
- Need to render JS? See Headless browser scraping with Playwright.
- Going to production? Scraping at scale without getting banned.