Ethics & Law Published Aug 14, 2026 · 37 min read · 8,190 words
Web Scraping for AI Training Data in 2026
How AI companies build training datasets: Common Crawl, web corpora, domain scraping, the crawl-to-JSONL pipeline, what makes good data, the 2026 legal landscape, and how a small team builds its own.
Web scraping for AI training is the most contested scraping question of 2026, and the one I get asked about most. Not because the engineering is hard — a crawl-to-JSONL pipeline is a weekend project, and I'll show you one below — but because everything around it is unsettled. The law is being written in real time by courts and regulators. The ethics are being negotiated in robots.txt files and licensing deals. And the data itself, the thing everyone is fighting over, is mostly just the web, scraped.
Let me be honest about my position before I go further, because this is a contested area and you should know where I sit. I run scrapers in production for a living. I believe public, non-personal data scraped politely is a legitimate raw material, and I believe the web's open crawl is one of the great public goods of the last two decades. I also believe the current free-for-all is not sustainable, that ignoring a site owner's stated opt-out is wrong even when it is technically possible, and that the people who will still be scraping in 2030 are the ones who built provenance and consent into their pipelines now. This post is the honest version of that position: the engineering, the numbers, the law, and the ethics, with the tradeoffs stated plainly.
Here is the one-paragraph map of 2026. Every frontier model you have used was trained, in significant part, on text that started as a web crawl. The largest share comes from Common Crawl, a nonprofit that has been crawling the web monthly since 2008 and giving the data away. On top of that raw crawl, the AI labs and open-source community have built cleaned corpora — C4, RefinedWeb, FineWeb, Dolma, The Pile — each the product of the same pipeline: crawl, filter, dedupe, clean, format. Around the edges, companies scrape domain-specific data directly: code, medical text, legal opinions, financial filings. And around all of it, a legal war is being fought over whether training on scraped copyrighted content is fair use, whether robots.txt is a binding opt-out, and whether the whole arrangement survives contact with the courts. That is the landscape. Now let me show you the terrain.
Key takeaways
- Almost all training data starts as a web crawl. Common Crawl is the base layer; C4, RefinedWeb, FineWeb, and Dolma are cleaned versions of it.
- The pipeline is the same everywhere: crawl, filter, dedupe, clean, format. The differences between corpora are differences in how hard each stage is applied.
- Deduplication is the most underrated stage. Exact hashes catch copies; MinHash catches the same article syndicated across fifty sites. Skip it and your model memorizes duplicates.
- The law is genuinely unsettled. US courts rejected the "training is fair use" assumption in 2025; the EU has made robots.txt a statutory opt-out for training data. There is no global answer.
- robots.txt is now a legal signal, not just a courtesy. In the EU it is a rights-reservation mechanism with regulatory teeth. Ignoring it is the fastest way to make every other argument you have weaker.
- A small team can build a domain dataset. A few thousand pages, politely crawled, cleaned, and deduped, beats a huge dirty crawl for most domain-specific models.
- Keep provenance. The question "was this data obtained with consent?" gets asked retroactively, and the only way to answer it is records.
How AI companies actually build training datasets
The dirty secret of the AI boom is that the most valuable training data in the world is mostly free. Not free as in "no one owns it" — free as in "a nonprofit crawls the web and hands it to you." The entire frontier-model supply chain starts with Common Crawl, and understanding that one fact explains more about the industry than any model architecture.
Common Crawl is a California nonprofit that has run a monthly web crawl since 2008 and published the raw results for anyone to download. The numbers are the first thing that surprises people. The full archive holds roughly 250 billion pages. Each monthly crawl adds 3 to 5 billion pages and lands at 200 to 400 terabytes of compressed data. The whole archive is on the order of 10 to 15 petabytes uncompressed. It is the largest open web corpus on the planet, and it is free.
But raw Common Crawl is not training data. It is a firehose of HTML, much of it boilerplate, spam, machine-generated junk, and content that no one would want a model to learn from. That is where the corpora come in. Starting around 2019, researchers began building cleaned datasets on top of Common Crawl snapshots, and the pattern they established is now the industry standard.
The first famous one was C4, the Colossal Clean Crawled Corpus, built by Google from the April 2019 Common Crawl snapshot. C4 runs a long list of filters — remove pages with placeholder text, remove pages with too many exclamation marks, remove pages whose text fails a simple profanity check, deduplicate, and so on — and the result is roughly 800 billion tokens of reasonably clean English web text. C4 became the default training corpus for a generation of models, and its filter list is still the template most pipelines copy.
The open-source community then scaled the idea up. RefinedWeb, from Hugging Face, applied a similar recipe to a larger snapshot and produced about 5 trillion tokens. FineWeb, also from Hugging Face, pushed it further to roughly 15 trillion tokens across 1.3 trillion documents. Dolma, from the Allen Institute for AI, sits around 3 trillion tokens. RedPajama, an open reproduction of the LLaMA recipe, is about 1.2 trillion. The Pile, from EleutherAI, is smaller — around 800 gigabytes — but deliberately mixes 22 curated sources, from arXiv and PubMed to GitHub and Stack Exchange, to get diversity that a single web crawl cannot provide.
The scale difference between these corpora is the second thing that surprises people, so let me show it. A domain dataset that a small team would build — say, every page of a documentation site — is measured in millions of tokens. The Pile is hundreds of billions. FineWeb is trillions. Common Crawl's full archive is tens of trillions. That is a five-order-of-magnitude spread, and it changes what each dataset is for.
The third thing worth understanding is what a frontier training corpus is actually made of, because "the web" is doing a lot of work in that sentence. If you could slice open a typical frontier model's training mix, the composition looks roughly like this: the majority is Common Crawl-derived text, filtered and deduped. A second chunk is other web text — curated crawls, blog networks, forums — that did not come through Common Crawl. Then books, then code, then academic text, then a small but disproportionately important slice of domain-specific data that the labs scraped or licensed directly.
Domain-specific scraping is where the pipeline stops being generic. Code is the clearest example: GitHub and its mirrors are the source for most code-training data, and the crawl is a different discipline from web text — you want the raw files, the license headers, and the commit history, and you want to filter out generated code and vendored dependencies. Medical text comes from PubMed and the clinical trial registries, which are structured enough that the crawl is almost an API call. Legal text comes from PACER, CourtListener, and the public-domain opinions that jurisdictions publish; the value is in the cleaning, because the formats are a mess. Financial text comes from SEC EDGAR, which is a public dataset with a documented API and a rate limit that is generous if you are polite. In every case the pattern is the same: the crawl is the easy part, the domain knowledge is the moat, and the licensing is the thing that separates a dataset you can ship from a dataset that gets you sued.
That last slice is where the scraping actually gets interesting, and it is where a small team can compete. Generic web text is a commodity: everyone has the same Common Crawl, and the corpora built on it are public. But a clean, well-documented dataset of, say, every public court opinion in a jurisdiction, or every product manual in an industry, or every page of a technical documentation ecosystem — that is not a commodity. That is a moat, and it is built with the exact skills this site teaches: polite crawling, careful extraction, honest deduplication, and provenance you can prove.
The pipeline, stage by stage
Every web-derived corpus, from C4 to FineWeb to a two-person startup's domain dataset, runs the same five stages. The differences are in how hard each stage is applied, and the numbers are the part nobody writes down. Let me walk each stage with the actual math.
Stage 1: Crawl
The crawl is the easiest stage to get right and the easiest to get wrong. The goal is not to fetch everything — it is to fetch the right things, once, without breaking the site or the law. The discipline is the same one this site teaches everywhere: read robots.txt, honor it, set a crawl-delay, identify yourself with a real User-Agent, and never crawl behind a login or a paywall.
The practical starting point is the sitemap, not the homepage. Most sites publish a sitemap.xml that lists the URLs the owner considers canonical, and crawling it is both more polite and more complete than discovering links by BFS. A typical crawl then deduplicates URLs at the frontier — the same page reachable through tracking parameters, print views, and pagination is one page, not five — and caps concurrency per domain at a handful of connections.
The volume math matters. A polite single-threaded crawler at one request every two seconds fetches about 43,000 pages a day. That is a real dataset for a domain-specific model, and it costs almost nothing. A serious multi-threaded crawl of a large site does a few pages per second per worker and needs to think about backoff, retries, and 429 handling. The politeness rules from the ethics post on this site are not optional here, because the crawl is the stage where you either build a reputation or destroy one.
Stage 2: Filter
The filter stage is where raw HTML becomes candidate text. Three filters dominate.
Language identification removes everything that is not in your target language. FastText's language-identification model is the standard tool, and it is accurate enough that a single pass is usually sufficient. Boilerplate removal strips navigation, footers, cookie banners, and comment sections — the parts of a page that are identical across thousands of URLs and teach a model nothing except how to emit cookie banners. Content extraction, the step this site's parsing guides cover in depth, turns the remaining HTML into clean text or markdown.
The third filter is the quality heuristic, and this is where C4's filter list became famous. C4 removes pages with placeholder text like "lorem ipsum" or "this page was generated automatically." It removes pages with too many exclamation marks or too many capital letters — signals of spam and clickbait. It removes pages whose text contains a word from a profanity list, which is crude but effective at cutting adult content. It removes pages that are too short or too long. None of these rules is sophisticated; together they remove a shocking amount of the web.
Stage 3: Dedupe
Deduplication is the most underrated stage in the entire pipeline, and the one most small teams skip. It matters for two reasons. First, a model trained on duplicated data spends its capacity memorizing copies instead of learning — if the same article appears fifty times in your corpus, the model treats it as fifty times more important than it is. Second, duplicated data inflates evaluation scores in misleading ways: a benchmark question whose answer appears verbatim in the training set is not a test of the model, it is a test of the model's memory.
There are two levels of deduplication. Exact deduplication hashes the text and drops identical copies — cheap, obvious, and necessary. Near-duplicate deduplication catches the same article syndicated across fifty sites with different headlines, or the same documentation page with different navigation. The standard tool is MinHash, which converts each document into a set of shingles, hashes them, and estimates similarity from the overlap. At a threshold of 0.8, two documents that share 80% of their shingles are treated as duplicates and one is dropped. The numbers are dramatic: near-duplicate dedupe typically removes 20 to 40% of a raw web corpus, and it is the single biggest quality lever after the crawl itself.
Stage 4: Clean
Cleaning is where the corpus becomes something you would actually train on. The big items are PII removal, toxicity filtering, and formatting normalization.
PII removal is the one with legal teeth. If your corpus contains personal data — names, emails, phone numbers, addresses — you have a GDPR problem in the EU regardless of whether the data was public. The practical approach is a mix of regex-based redaction for emails and phone numbers, and classifier-based detection for the harder cases. This is also the stage where the site's ethics stance bites hardest: if you do not need the personal data, do not collect it in the first place, because removing it later is harder than never having it.
Toxicity filtering uses a classifier — the standard one is a RoBERTa model fine-tuned on the Jigsaw toxicity dataset — to drop hateful or abusive text. It is blunt, it over-removes, and it is still worth doing, because the alternative is a model that reproduces the worst of the web. Formatting normalization is the boring but essential part: consistent newlines, consistent whitespace, consistent handling of code blocks and lists, so the tokenizer sees clean input.
Stage 5: Format
The final stage turns the cleaned text into the format the training code consumes. The de facto standard is JSONL — one JSON object per line, with fields for the text, the source URL, and any metadata. The metadata is not optional: it is your provenance record, and it is the only thing that lets you answer "where did this come from and did we have permission?" when someone asks later.
The other formatting decision is tokenization. Most pipelines tokenize once and store the token IDs, because tokenizing during training is wasted compute. The token count is also the number that matters for cost: training compute scales with tokens, so the corpus size in tokens is the number that determines your training bill.
What makes good training data
The pipeline removes noise, but it does not tell you what to keep. That is the quality question, and it has three axes: quality, diversity, and licensing. Getting the first two right is engineering; getting the third right is the difference between a dataset and a lawsuit.
Quality. The crude version is the C4 filter list — length, profanity, placeholder text. The sophisticated version is perplexity filtering: train a small language model on a sample of your corpus, then score every document by how surprised the model is by it. Documents with very low perplexity are often boilerplate or duplicated text; documents with very high perplexity are often spam or machine-generated garbage. Keep the middle. It sounds circular — using a model to clean data for a model — and it is, but it works, and it is how most serious corpora are built.
Diversity. A corpus can be clean and still be useless because it is all one thing. Diversity operates at several levels: domain diversity (a model trained only on news writes like a newsroom), language diversity (most of the web is English, so most corpora are English, which is a real bias), and source diversity (fifty sites are not the same as fifty thousand). Deduplication is a diversity tool as much as a quality tool, because it prevents a handful of prolific sites from dominating the corpus.
Licensing. This is the axis that changed most between 2023 and 2026. Five years ago, "it's public" was the whole licensing story. Now the question is whether the source opted out, whether the content is copyrighted, and whether your jurisdiction's text-and-data-mining exception applies. The practical rule this site applies: prefer sources that are explicitly permissive, honor opt-outs, and keep the provenance records that let you prove both.
The honest tradeoff, stated plainly: for most small teams, a clean domain dataset of a few million tokens beats a huge dirty crawl, because the model you can actually train on a small budget will learn more from clean, diverse, well-licensed data than from a firehose of junk. The frontier labs need trillions of tokens because they are training trillion-parameter models. You are probably not, and the data strategy should match the model.
How do you know your dataset is good before you spend the training compute? The honest answer is that you do not fully know until you train, but there are cheap proxies. Run your quality filters and spot-check a random sample — a hundred documents read by a human tells you more than any metric. Check the deduplication rate: if near-duplicate removal dropped less than 10% of your corpus, your threshold is too loose or your corpus was already clean. Check the token-to-document ratio: a corpus that averages under 500 tokens per document is mostly noise, and one that averages over 5,000 is mostly long-form content that may be over-represented. And keep a held-out evaluation set that never touches the training pipeline, because the fastest way to fool yourself is to evaluate on data that leaked into training.
The legal landscape in 2026
This is the part where I have to be careful, because the law is genuinely unsettled and anyone who tells you otherwise is selling something. This is education, not legal advice, and the full treatment lives in the legal guide on this site. Here is the honest map.
The central fight is copyright, and it is being fought in US courts. The New York Times sued OpenAI and Microsoft in December 2023, alleging that its content was scraped into training data without permission. A wave of author and publisher suits followed. The case that changed the analysis was Thomson Reuters v. Ross Intelligence, decided in February 2025: a federal court ruled that training on copyrighted legal headnotes was not fair use. That decision landed like a brick, because "training is fair use" was the AI industry's core legal theory, and a court had just said no. The fair-use fight is not over — the NYT case may still rewrite the rules — but the assumption that training on scraped content was safe collapsed in 2025.
The second front is opt-outs, and this is where robots.txt became a legal instrument. In the EU, the DSM Directive's text-and-data-mining exception lets rightsholders reserve their rights in a machine-readable way, and the machine-readable way the entire web already uses is robots.txt. The AI Act then made compliance with those opt-outs a statutory obligation for general-purpose AI providers, with training-data transparency duties that started applying in August 2025. The practical consequence: in the EU, a publisher that blocks GPTBot or ClaudeBot in robots.txt has not just expressed a preference — it has exercised a legal right, and scraping the content anyway is a statutory violation, not just a rude move.
The third front is the market that grew up between the courts and the crawlers. Starting in 2024, the major labs began signing licensing deals with publishers — the OpenAI and News Corp deal being the most visible — and a small industry of licensing brokers and opt-out registries appeared to sit between "blocked crawler" and "settled case." Cloudflare's AI Audit, launched in September 2024, gave site owners a dashboard of which AI bots were hitting them and a one-click block. By 2026, roughly a third of the largest sites block at least one major AI crawler, and the number is still climbing.
What does this mean for you, practically? Three rules. First, if you are building a training dataset, treat robots.txt as a binding opt-out — it is in the EU, and US courts are increasingly treating ignored opt-outs as evidence in fair-use analysis. Second, keep provenance: log what you crawled, what robots.txt said at the time, and what you did with it, because the question gets asked retroactively. Third, prefer licensed and permissive sources when they exist, and be honest that the law is unsettled enough that no one can give you a clean bill of health.
The practical posture for a small team is more conservative than the frontier labs can afford to be. A lab with a legal department can litigate the fair-use question; you cannot. So the rules that make sense for a small team are the conservative ones: crawl only public, non-personal data; honor robots.txt as if it were a statute, because in the EU it effectively is; keep provenance records for every document; and when a source offers a license or an API, take it. The cost of this conservatism is real — you will have less data than the labs — but the cost of the alternative is existential, and a dataset you can defend is worth more than a dataset you cannot.
The ethics: what we actually do
The legal landscape tells you what you can get away with. The ethics tell you what you should do, and on this site the two are not the same thing. The ethics stance here is simple and I will state it plainly: prefer APIs and official datasets when they exist, respect robots.txt, rate-limit politely, and do not scrape personal data you do not need. Applied to training data, that stance produces a concrete checklist.
Respect robots.txt, including the AI crawler blocks. The dedicated AI crawlers — GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot — publish their user-agents and say they honor robots.txt. If a site blocks your crawler, the block is the owner speaking. Treat it as a stop, not a puzzle. This is the single most important ethical rule in this post, and it is now a legal rule in the EU.
Do not scrape personal data you do not need. Training data is a place where this rule gets violated at scale, because a web crawl is indiscriminate. The GDPR does not care that the data was public. If your corpus contains personal data and you are in the EU, or touching EU residents, you need a lawful basis and a deletion path. The engineering corollary is the one this site repeats everywhere: if you cannot delete someone's data when they ask, you should not have collected it.
Prefer the official channel. If the domain you care about has an API, a data dump, or a research program, use it. Common Crawl exists precisely so people do not have to crawl the web themselves. The SEC has EDGAR, the courts have PACER and CourtListener, PubMed is open, arXiv is open, GitHub is open. The list of domains where scraping is the only option is much shorter than people think, and it is shrinking.
Go gently. The politeness rules from the ethics post apply in full: crawl-delay, jitter, honor 429s, identify yourself with a real User-Agent and a contact address. A training-data crawl is a long-running crawl, and a long-running crawl that is rude is a distributed denial-of-service attack with good intentions.
How a small team can build a domain dataset
Now the part you actually came for. You want to train a model on a specific domain — your product's documentation, a niche industry's public filings, a technical ecosystem's forums — and you want to build the dataset yourself. Here is the honest playbook, including the build-versus-buy math and a working pipeline.
Start from what exists
Before you crawl anything, check what already exists. For generic text, the answer is always "download a corpus" — Common Crawl subsets, FineWeb, The Pile, Dolma are free and already cleaned. For domain-specific text, check whether the domain publishes a dataset or an API: EDGAR for SEC filings, PubMed for biomedical, arXiv for physics and math, CourtListener for US case law, GitHub Archive for code. The list of domains where scraping is genuinely the only option is short, and starting from an official source converts the whole question from "can I scrape this?" to "what does the license allow?"
The build-versus-buy math
If you do need to crawl, the cost question is real, and the answer depends on volume. Let me give you the honest numbers for a 10-million-page crawl, because that is the size where the decision actually matters.
Crawling it yourself, politely, costs bandwidth and compute: roughly a few hundred dollars of egress and a modest server, plus your time. The catch is that "your time" is not free, and a 10-million-page crawl is a real infrastructure project — proxies, retries, backoff, storage, and the maintenance that never ends. Renting the crawl from a scraping API converts that into a per-page price, and the price varies by an order of magnitude depending on the provider.
The honest recommendation: for a one-off dataset under a few hundred thousand pages, crawl it yourself with the pipeline below — it is free, it is a weekend, and you keep full control of provenance. For a large or ongoing crawl, rent the fetch layer from a scraping API and spend your engineering time on the stages that actually differentiate your dataset: filtering, deduplication, and cleaning. The providers worth comparing are the ones this site covers in the APIs and tools guides — ScraperAPI, ScrapingBee, Firecrawl, ZenRows, Keirolabs, and the rest. Keirolabs is worth a specific mention because it is the outlier on price at $0.25 per 1,000 pages, it returns full markdown content which is exactly what a training pipeline wants, and its extraction model scores at the top of factuality benchmarks like FinanceBench and SimpleQA — relevant if your dataset feeds a RAG system rather than a from-scratch training run. But it is one honest option in a comparison, not the whole story, and you should test any provider against your actual pages before committing.
The working pipeline
Here is the pipeline I actually use for a small domain dataset, in full. It is deliberately simple — standard library plus requests and BeautifulSoup — because the point is to show the shape, not to hide it behind a framework. It crawls politely, honors robots.txt, extracts clean text, deduplicates exactly, and writes JSONL. The near-duplicate dedupe is a second snippet, because it is the stage most people skip and the one that matters most.
# build_domain_dataset.py — crawl -> extract -> dedupe -> JSONL
import hashlib, json, random, re, time
from urllib.parse import urljoin, urlparse
from urllib.robotparser import RobotFileParser
import requests
from bs4 import BeautifulSoup
BASE = "https://docs.example.com" # the domain you have permission to crawl
UA = "DomainDatasetBot/1.0 (+https://example.com/contact)"
OUT = "dataset.jsonl"
rp = RobotFileParser()
rp.set_url(urljoin(BASE, "/robots.txt"))
rp.read()
def polite_get(url):
if not rp.can_fetch(UA, url):
return None
delay = (rp.crawl_delay(UA) or 2.0) + random.uniform(0, 1.0)
time.sleep(delay)
return requests.get(url, headers={"User-Agent": UA}, timeout=15)
def extract_text(html):
soup = BeautifulSoup(html, "html.parser")
for tag in soup(["script", "style", "nav", "footer", "noscript"]):
tag.decompose()
return re.sub(r"\n{3,}", "\n\n", soup.get_text("\n")).strip()
def page_links(html, base):
soup = BeautifulSoup(html, "html.parser")
for a in soup.find_all("a", href=True):
href = urljoin(base, a["href"])
if urlparse(href).netloc == urlparse(base).netloc:
yield href
def main():
seen_urls, seen_hashes = set(), set()
queue = [BASE]
with open(OUT, "w") as fh:
while queue and len(seen_urls) < 500:
url = queue.pop(0)
if url in seen_urls:
continue
seen_urls.add(url)
resp = polite_get(url)
if resp is None or resp.status_code != 200:
continue
text = extract_text(resp.text)
if len(text) < 200: # thin pages are noise
continue
doc_hash = hashlib.sha256(text.encode()).hexdigest()
if doc_hash in seen_hashes: # exact duplicate
continue
seen_hashes.add(doc_hash)
fh.write(json.dumps({
"url": url,
"text": text,
"hash": doc_hash,
"fetched": time.time(),
}) + "\n")
queue.extend(page_links(resp.text, url))
if __name__ == "__main__":
main()
Run that against a documentation site and you have a JSONL file with one clean document per line, each carrying its source URL and a content hash. The URL is your provenance record; the hash is your exact-dedupe key. Two things are deliberately encoded in the code. First, the robots gate: can_fetch is checked before every request, and a disallowed URL is skipped, not worked around. Second, the politeness delay: crawl_delay from robots.txt, or a two-second default, plus jitter, so the request pattern looks like a person rather than a metronome.
The exact-dedupe in that script catches identical copies, but it misses the case that actually dominates web data: the same article syndicated across fifty sites with different headlines and navigation. That is what near-duplicate deduplication is for, and MinHash is the standard tool. Here is the second snippet, run over the JSONL you just wrote:
# near-duplicate dedupe with MinHash (pip install datasketch)
import json, re
from datasketch import MinHash, MinHashLSH
def shingles(text, k=5):
text = re.sub(r"\s+", " ", text).lower()
return {text[i:i + k] for i in range(len(text) - k + 1)}
rows = [json.loads(line) for line in open("dataset.jsonl")]
lsh = MinHashLSH(threshold=0.8, num_perm=128)
kept = []
for i, row in enumerate(rows):
m = MinHash(num_perm=128)
for s in shingles(row["text"]):
m.update(s.encode())
if lsh.query(m): # too similar to a doc we already kept
continue
lsh.insert(f"doc-{i}", m)
kept.append(row)
with open("dataset_deduped.jsonl", "w") as fh:
for row in kept:
fh.write(json.dumps(row) + "\n")
The threshold=0.8 means two documents that share 80% of their five-word shingles are treated as duplicates and one is dropped. On a typical web crawl this removes 20 to 40% of the corpus, and it is the single biggest quality lever after the crawl itself. If you skip this step, your model will spend a meaningful fraction of its capacity memorizing the same article in fifty different fonts.
The honest small-team playbook
Put it together and the playbook is short. Start from public corpora for generic text. Crawl the handful of authoritative domain sources yourself, politely, with the pipeline above. Deduplicate hard — exact and near-duplicate. Keep the source URL on every record. Document what robots.txt said when you crawled, because the question gets asked later. And be honest about scale: a few million clean tokens of domain data, added to a generic base, is a real and defensible dataset. A hundred million pages of junk is not.
The honest bottom line
Web scraping for AI training is the most contested scraping question of 2026 because it is the one where the stakes are highest and the rules are least settled. The engineering is not the hard part — the pipeline is five stages, the code is a weekend, and the corpora are free. The hard parts are the ones that do not have a library: deciding what is yours to take, keeping the records that prove you took it honestly, and building the dataset in a way that survives contact with the law and with the people whose content you used.
My position, stated plainly one more time. Public, non-personal data, scraped politely, with robots.txt honored and provenance kept, is a legitimate raw material, and the open crawl is a public good worth defending. The free-for-all is not sustainable, and the people who will still be building datasets in 2030 are the ones who treated consent and provenance as engineering requirements rather than afterthoughts. Build the pipeline, keep the records, and ask before you take. That is the whole discipline, and it is the one this site has been teaching all along.
Further reading
- Web scraping legal: the 2026 landscape — the case law behind the fair-use fight, the CFAA, and the GDPR, in depth.
- Web scraping ethics and robots.txt — the politeness rules and the public-versus-allowed framework this post builds on.
- Web scraping APIs — the build-versus-buy decision, applied to hosted fetching and rendering.
- Web scraping tools — the tooling landscape, from HTTP clients to browser automation.
- Web scraping with Python — the language most training-data pipelines are written in, from the ground up.
Frequently Asked Questions
Is it legal to scrape data for AI training?
There is no global answer. In the US it is being fought out in copyright courts — NYT v. OpenAI and Thomson Reuters v. Ross Intelligence, where a court rejected fair use for training in 2025. In the EU, the DSM Directive's text-and-data-mining exception lets rightsholders opt out via machine-readable means like robots.txt, and the AI Act now requires training-data transparency. The honest answer: it depends on jurisdiction, what you scrape, and whether the owner opted out. This post is education, not legal advice.
What is Common Crawl?
Common Crawl is a nonprofit that has crawled the web monthly since 2008 and made the raw data freely available. The archive holds roughly 250 billion pages and each monthly crawl adds 3 to 5 billion pages. It is the starting point for most large web corpora — C4, RefinedWeb, FineWeb, and Dolma are all built on top of Common Crawl snapshots.
How do AI companies get training data?
Mostly by downloading Common Crawl and other public web corpora, then running a pipeline that filters, deduplicates, cleans, and formats the text. Some data is licensed from publishers, some is scraped directly, and some comes from curated sources like GitHub, arXiv, and PubMed. The pipeline is the same everywhere: crawl, filter, dedupe, clean, format.
Can I build my own training dataset?
Yes, and for a domain-specific model it is often the right call. A small team can crawl a few thousand pages politely, extract clean text, dedupe, and write JSONL in a weekend. The honest caveat: a 10-million-page crawl is a real infrastructure project, and for generic text you are usually better off starting from Common Crawl or a public corpus than from scratch.
What is the best source of training data for a small team?
For generic text, download a Common Crawl subset or a public corpus like FineWeb or The Pile — it is free and already cleaned. For domain-specific data, crawl the handful of authoritative sites in your niche yourself, politely, with a documented pipeline. Mixing a small clean domain dataset with a large generic corpus is the pattern most small teams end up with.
Do AI crawlers respect robots.txt?
The major ones — GPTBot, ClaudeBot, Google-Extended, CCBot — say they do, and most honor it. But robots.txt is a request, not a technical block, and compliance is voluntary. In the EU, the AI Act and the DSM Directive have made robots.txt a legally meaningful opt-out signal for training data, so ignoring it is now a statutory problem there, not just an ethical one.
How much does it cost to build a training dataset?
A small domain dataset of a few thousand pages costs almost nothing — a weekend of compute and bandwidth. A 10-million-page crawl via a scraping API runs roughly $2,500 to $32,000 depending on the provider, versus a few hundred dollars of bandwidth if you crawl politely yourself or download a Common Crawl subset. The expensive part is usually cleaning and deduplication compute, not the crawling.
What is deduplication and why does it matter for training data?
Deduplication removes repeated content so the model does not memorize or over-weight it. Exact duplicates are caught with hashes; near-duplicates — the same article syndicated across fifty sites — need MinHash or SimHash. It matters because a model trained on duplicated data spends its capacity memorizing copies instead of learning, and it inflates benchmark scores in misleading ways.
Keep reading
Web scraping ethics and robots.txt: the lines you don't cross
A clear, practical guide to scraping ethically and legally: reading robots.txt, Terms of Service, the CFAA and EU equivalents, login walls, personal data, and the difference between 'public' and 'allowed'.
Is Web Scraping Legal? The 2026 Legal Landscape, Explained
A practical, verdict-first explainer of web scraping law in 2026: the CFAA, hiQ v. LinkedIn, Terms of Service, GDPR and personal data, robots.txt, login walls, jurisdiction by jurisdiction, and the difference between public and allowed. Education, not legal advice.
Web Scraping APIs for AI Agents: The 2026 Guide
Why LLM agents can't just fetch URLs, and how to give them clean web content: scraping APIs, markdown conversion, browser rendering, search-and-extract pipelines, structured extraction, and the cost/latency math of feeding an agent the web.
Found this useful? Cite it as: webscraping.space. “Web Scraping for AI Training Data in 2026.” https://webscraping.space/blog/scraping-for-ai-training-data. Published 2026-08-14.