The Scraping Log
23 posts. Newest first.
AllAI AgentsAnti-BotEthics & LawFundamentalsJavaScriptNode.jsParsingProxiesPythonScaling
What Actually Gets You Blocked When Web Scraping (2026)
We run a web scraping API and see millions of requests a day. Here's what actually gets you blocked in 2026 — and what doesn't, signal by signal.
Web Scraping for AI Training Data in 2026
How AI companies build training datasets: Common Crawl, web corpora, domain scraping, the crawl-to-JSONL pipeline, what makes good data, the 2026 legal landscape, and how a small team builds its own.
E-commerce Scraping: The Complete 2026 Guide
The complete 2026 guide to e-commerce scraping: what data people scrape, platform anti-bot profiles, price-monitoring pipelines, cost math, working code, and the legal reality.
LLM Extraction: Structured Data from HTML (2026 Guide)
The 2026 guide to LLM extraction: when it beats CSS selectors, schema design, token cost math, validation, retries, and a complete Python pipeline.
GLM-5.3 Review: Coding, Security, Benchmarks (2026)
An engineer's review of GLM-5.3 (Zhipu AI, 2026): post-training wins, Terminal-Bench 28.3, ExploitBench 54.4, the open-weights safety debate, cost, self-hosting.
The Complete Proxy Guide for Web Scraping (2026)
The honest 2026 proxy guide: what a proxy does, the four types compared, rotation, detection, real pricing, cost math, when you need one, and working Python code.
CAPTCHA Solving for Web Scraping: The Honest 2026 Guide
The honest 2026 guide to CAPTCHAs in web scraping: the landscape, why they appear, solving services and real cost per 1k, accuracy, and why avoiding the trigger beats solving it.
The Best Web Scraping Tools in 2026: Full Breakdown
The honest 2026 ranking of web scraping tools — libraries, GUI scrapers, APIs and platforms — with real pricing, honest pros and cons, and a decision guide.
Web Scraping with Node.js: The Complete 2026 Guide
Everything for scraping with Node.js in 2026: why Node is a natural fit, the fetch/undici plus cheerio default stack, Playwright for JavaScript pages, p-limit concurrency, retries with backoff, and a complete runnable scraper that respects robots.txt.
Web Scraping APIs in 2026: The Complete Guide
The honest 2026 guide to web scraping APIs: when to buy vs build, the five stages inside a call, real pricing for every major provider, cost math, code, and verdicts.
Web Scraping Without Getting Blocked: The 2026 Anti-Ban Playbook
How to scrape without getting blocked: the behavioral and operational playbook — politeness and rate shaping, realistic headers and fingerprints, caching, retries, ban detection, IP strategy, and recovery. What actually keeps you unbanned, not proxy marketing.
Web Scraping with Python: The Complete 2026 Tutorial
A complete from-scratch web scraping tutorial in Python: picking a target, reading robots.txt, fetching with requests, parsing with BeautifulSoup and lxml, handling JavaScript pages, pagination, saving clean data to CSV/SQLite, and doing it all politely without getting blocked. Zero-to-working in one guide.
Is Web Scraping Legal? The 2026 Legal Landscape, Explained
A practical, verdict-first explainer of web scraping law in 2026: the CFAA, hiQ v. LinkedIn, Terms of Service, GDPR and personal data, robots.txt, login walls, jurisdiction by jurisdiction, and the difference between public and allowed. Education, not legal advice.
Web Scraping APIs for AI Agents: The 2026 Guide
Why LLM agents can't just fetch URLs, and how to give them clean web content: scraping APIs, markdown conversion, browser rendering, search-and-extract pipelines, structured extraction, and the cost/latency math of feeding an agent the web.
Scraping JavaScript-Rendered Pages: The 2026 Practical Guide
How to scrape JavaScript-rendered websites in 2026: detecting when data is behind JS, finding and calling the site's own JSON APIs, render proxies, headless browsers with Playwright, cost and detection math for each approach, and the decision framework for choosing.
Residential Proxies for Web Scraping: The Honest Guide
What residential proxies actually do for web scraping, when you genuinely need them, and when they're wasted money: how IP rotation works, residential vs datacenter vs mobile, real pricing, integration with requests and Playwright, and why proxies don't fix a bad fingerprint.
Web scraping ethics and robots.txt: the lines you don't cross
A clear, practical guide to scraping ethically and legally: reading robots.txt, Terms of Service, the CFAA and EU equivalents, login walls, personal data, and the difference between 'public' and 'allowed'.
Building search infrastructure for 1M+ queries a month: the stuff nobody tells you
A deep, no-bullshit technical dive into building search infrastructure that handles over 1 million queries per month. Inverted index internals, posting list compression, WAND/MaxScore scoring, Go hot-path code, five-layer cache hierarchies, mmap and SSD reality, tiered segment merging, tail latency, and the non-obvious stuff that actually kills you at scale.
Scraping at scale: queues, caching, and not getting banned
How to take a working scraper to millions of pages without melting the target or getting banned: work queues, bounded concurrency, on-disk caching, dedup, retries with backoff, and polite scheduling.
Bypassing anti-bot protections: TLS, fingerprints, and Cloudflare
A frank guide to anti-bot defenses and how scrapers get past them: TLS/JA3 fingerprinting, Cloudflare's challenge, PerimeterX, CAPTCHAs, residential proxies, and the libraries (curl_cffi, Camoufox) that actually work in 2026.
Headless browser scraping with Playwright (Python & Node.js)
When the data lives behind JavaScript, requests is not enough. This guide covers headless browser scraping with Playwright in Python and Node.js: stealth, waiting strategies, intercepting API calls, and not running the browser when you don't have to.
Parsing HTML with BeautifulSoup: selectors, speed, and survival
A deep look at parsing HTML with BeautifulSoup and lxml in Python: CSS vs XPath, handling malformed markup, the speed difference between parsers, and writing selectors that survive small redesigns.
Web scraping with Python requests: a practical starter
A hands-on guide to web scraping with Python's requests library: downloading pages, setting headers, handling redirects, parsing with BeautifulSoup, and avoiding the most common beginner mistakes.