Ethics & Law Published Aug 8, 2026 · 46 min read · 10,031 words
Is Web Scraping Legal? The 2026 Legal Landscape, Explained
A practical, verdict-first explainer of web scraping law in 2026: the CFAA, hiQ v. LinkedIn, Terms of Service, GDPR and personal data, robots.txt, login walls, jurisdiction by jurisdiction, and the difference between public and allowed. Education, not legal advice.
Let me answer the question you actually typed into a search bar, because everything else is details: in most jurisdictions, scraping public, non-personal data is generally lawful. But "generally" hides a lot. It hides the difference between public and allowed, the difference between a contract breach and a crime, and the difference between scraping HTML you could paste into a browser and scraping data behind a login. Get any of those wrong and "generally lawful" stops applying to you.
A word before the deep dive, because this matters and it's not a disclaimer buried in a footer: I am a production engineer, not your lawyer. This post is education, not legal advice. It reflects how the law looks from where I sit as someone who has run scrapers for years, watched companies get sued, and had to make real calls about what to crawl and what to leave alone. Laws change, courts disagree, and one bad-fact case can rewrite the landscape. Use this to know which questions to ask your own counsel, not as a substitute for them.
Here's the honest one-paragraph map of 2026. The United States has no statute that says "web scraping is illegal." The closest thing, the Computer Fraud and Abuse Act, has been narrowed twice in the last six years and now mostly reaches access-control circumvention, not polite crawling of public pages. The EU doesn't ban scraping either, but it regulates the two things scrapers most want to collect — personal data and copyrighted works — harder than any other jurisdiction. The UK and the Commonwealth sit somewhere in between, with older computer-misuse statutes that have never fully been tested against modern scraping. And everywhere, the Terms of Service you didn't read is quietly the most common thing that gets people sued. That's the landscape. Now let me show you the actual terrain, case by case, statute by statute, jurisdiction by jurisdiction.
The short verdict
If you want the verdict before the analysis, here it is, stated as precisely as I can state it:
- Scraping public, non-personal data, politely, without bypassing anything, is lawful in most jurisdictions — and US courts have repeatedly declined to let platforms stop it under the CFAA.
- Scraping personal data — even public personal data — is regulated by GDPR and its cousins, and in the EU/UK it requires a lawful basis and a deletion path. "It was public" is not a defense.
- Scraping behind a login or bypassing any access control — a CAPTCHA, an IP block, a paywall — is the single biggest legal escalation you can make. It is where the CFAA, the Computer Misuse Act, and breach of contract all converge against you.
- Scraping in violation of a ToS is a contract problem, not a crime — but contracts get you banned, sued, and enjoined, and post-Van Buren, contract claims are precisely where platforms are winning.
- Scraping for AI training is the open wound. US courts are deciding it under copyright law right now; the EU has built a machine-readable opt-out system; and nobody has a global answer yet.
Everything below is those five sentences unpacked with the cases, the statutes, the numbers, and the practical rules. We'll spend most of our time on the US framework, because that's where the case law lives, then Europe, because that's where the enforcement money is.
The US framework: the CFAA and "exceeds authorized access"
The center of gravity of US scraping law is 18 U.S.C. § 1030, the Computer Fraud and Abuse Act, passed in 1986 and amended several times since. It's the statute that makes it a federal crime to access a computer "without authorization" or to "exceed authorized access" and thereby obtain information from a protected computer. It has criminal and civil prongs — the government can prosecute, and private parties can sue for damages — and for two decades it was the weapon of choice against scrapers.
The phrase that does all the work is "exceeds authorized access." The statute defines it as accessing a computer with authorization and then using that access to obtain information you're not entitled to obtain. For a long time, plaintiffs stretched that phrase to mean almost anything: you're authorized to browse a site, therefore you're authorized to fetch pages, therefore if you fetch them faster than a human, with a bot, in violation of the ToS, you've "exceeded" your authorization. Under that reading, violating the ToS was a federal crime. That was the theory platforms pushed, and for years courts were all over the place on it.
Then in 2021, the Supreme Court cut the theory down.
Van Buren narrowed the CFAA
Van Buren v. United States, decided June 3, 2021, wasn't a scraping case at all. It was a police officer who used his law-enforcement credentials to look up a license plate for cash — misuse of authorized access, classic "gates up, gates down" fact pattern. The government argued the officer exceeded his authorization because he used the database for an improper purpose. The Court said no, in an opinion that matters enormously for scraping.
The holding: "exceeds authorized access" covers only those who obtain information to which their access does not extend, not those who misuse access they otherwise have. The Court used a gates-up/gates-down metaphor. If a gate is up, you're authorized to pass; whether you pass for a good reason or a bad reason is beside the point under the CFAA. If the gate is down, you can't pass, and forcing your way through is a violation.
Translate that to scraping and it's a structural shift. Under Van Buren, the CFAA question is not "did the ToS forbid this?" It's "was there a gate, and did you cross it?" Public web pages have no gate. A login wall has a gate. An IP block has a gate. A CAPTCHA has a gate. That reframing is why the CFAA stopped being a reliable weapon against scrapers of public data — and why bypassing an access control became the fact that actually matters.
Two consequences followed. First, criminal and civil CFAA exposure for scraping public data collapsed; you can't "exceed" access you were never granted to data that was openly served. Second, every platform that had built its litigation strategy around the CFAA had to go find a different cause of action. They found breach of contract — more on that below. The important thing to internalize: after Van Buren, a ToS violation is not a federal crime, and a platform that wants to stop you usually has to sue you for something else.
hiQ v. LinkedIn: the saga, in order
No scraping case tells the story of the last decade better than hiQ Labs v. LinkedIn, so let me walk it in order. hiQ scraped publicly visible LinkedIn profiles — pages you could view without logging in — and sold workforce analytics to employers. In 2017 LinkedIn sent a cease-and-desist and started blocking hiQ's IP addresses. hiQ sued, and the case ran for five years through three courts.
The beats matter, so here they are in sequence.
June 2017 — the TRO. After LinkedIn blocked hiQ, hiQ sued and asked for a temporary restraining order. The district court granted it, ordering LinkedIn to restore access. This was the first big "public data wins" signal, and it set the tone: a federal court told LinkedIn it could not use technical blocks to cut off a scraper of data that was openly available.
2019 — the Ninth Circuit's public-data ruling. On appeal, the Ninth Circuit affirmed, holding that accessing publicly available information does not violate the CFAA's "without authorization" prong. The court went further and said that LinkedIn's robots.txt directive and its ToS didn't change the analysis for the CFAA claim — the data was public, full stop. This became the most-cited sentence in scraping law: public data is not "without authorization" under the CFAA. Every scraper-friendly briefing since has leaned on it.
June 2021 — the Supreme Court steps in. The Supreme Court granted certiorari, and then, in the same order, vacated the Ninth Circuit's judgment and remanded in light of Van Buren. That's the "GVR" maneuver: grant, vacate, remand. The Court wasn't saying the Ninth Circuit was wrong; it was saying "reconsider this in light of our new CFAA opinion." The upshot was that the celebrated public-data ruling was now provisional.
April 2022 — the retreat. On remand, the Ninth Circuit issued an amended opinion that withdrew its earlier public-data reasoning. It stopped short of saying the CFAA barred hiQ's scraping, but it no longer blessed it either. It kept the injunction alive on balance-of-hardship grounds — hiQ would go out of business without access — while leaving the actual CFAA question open. In practical terms, the strongest pro-scraper precedent of the decade had been walked back to "maybe."
2022 — settlement. The parties settled, and the case was dismissed. LinkedIn's breach-of-contract counterclaims were still pending; hiQ's core CFAA claim was still undecided. The case that was going to settle scraping law instead settled out of court, and the definitive answer everyone wanted never arrived.
What should you take from hiQ? Three things. First, the "public data" argument is real and it has carried real cases — including post-Van Buren ones like X v. Bright Data, which we'll get to. Second, it is not a guarantee; the same circuit that blessed it in 2019 withdrew it in 2022. Third, a scraper that wins on the law can still lose on time and money: hiQ spent five years and, by all appearances, most of its resources fighting before the case just evaporated. Even when you're right, litigation is the punishment.
State computer-crime laws
The CFAA is the famous one, but it's not the only one. Every US state has its own computer-tampering statute, and several are broader than the federal law. California Penal Code § 502 is the one you're most likely to run into, because so much of the web's infrastructure lives in California and California courts have been willing to hear scraping claims under it. Texas, New York, Massachusetts, and Washington have similar statutes with their own quirks.
The key difference from the CFAA: state statutes vary wildly in how they define "without authorization" and "exceeds authorized access." Some use language that's essentially a copy of the CFAA, which means Van Buren's reasoning — if a court chooses to follow it — applies there too. Others define access crimes more loosely, and a handful explicitly criminalize accessing a computer in violation of a "restriction" that the owner has stated, which can sweep in ToS and robots.txt in ways the federal law no longer does.
Two practical points. First, when you're assessing risk for a big scrape, check the law of the state where the server lives, not just your own jurisdiction — a scrape that's clean under federal law can still face a state claim. Second, state statutes are where "you violated the ToS" can still become "you committed a computer crime," depending on the state. That's a genuinely different risk profile from the federal one, and it's one of the reasons platforms file in California courts when they can.
ToS claims after Van Buren
Here's the shift that matters more than any single case: post-Van Buren, the CFAA is no longer a reliable way to sue a scraper of public data, so platforms now sue for breach of contract instead. Every ToS says something about automated access; most say "no scraping" in some form; and by using the site, the argument goes, you agreed to it. That converts a scraping dispute from a federal computer-crime fight into an ordinary contract dispute, and contract disputes are much friendlier to the platform.
The leading example is X Corp v. Bright Data, decided in 2024. X sued Bright Data for scraping public tweets — the same activity that would have been a headline CFAA case a decade earlier — and its claims were dismissed. The court held, among other things, that Bright Data wasn't bound by X's ToS because Bright Data never had an account and never agreed to it. The case is a win for scrapers, but read it carefully: the win was that the contract didn't apply, not that contract claims can never apply. If you do have an account, or you do sign up and then scrape, the contract analysis is different.
That's the practical posture in the US in 2026. Public data scraping is hard to stop under the CFAA, but it's still stoppable under contract, under tort theories like trespass and tortious interference, and under state computer-crime laws. The platforms have adapted. The question "can they sue me?" is almost always yes — the real question is "can they win, and what does it cost me if they try?"
Here's the post-Van Buren scorecard, with each case placed by how much clarity it gave scrapers, not by who "won" in a headline sense — because several of these cases never produced a merits ruling at all.
Europe: GDPR, and the regulators who actually enforce
If the US is where scraping law gets litigated, Europe is where it gets enforced. The difference is institutional. In the US, a scraper's worst case is usually a private lawsuit brought by a well-funded platform. In Europe, the worst case is a data-protection authority with the power to fine you up to 4% of global turnover, and they've shown they will use it. You cannot understand scraping law in 2026 without understanding the GDPR.
Public personal data is still personal data
Here is the single most counterintuitive fact in this entire post: under the GDPR, "public" and "personal" are orthogonal. Data is personal if it relates to an identified or identifiable natural person — a name, an email, a username, an IP address, a photo, location data. Whether that data was published on a public website is irrelevant to the classification. The GDPR's Article 4 definition doesn't have a "but only if it's private" clause. Scraping a public directory of names and job titles is, under the GDPR, processing personal data, full stop.
The Danish regulator made this exact point explicitly. In January 2023, the Danish Data Protection Authority (Datatilsynet) issued an opinion on scraping personal data from publicly accessible sources, and its position was unambiguous: publicly accessible does not mean freely processable. "Accessible" is a fact about the server; "lawful to collect" is a legal conclusion, and the conclusion depends on your basis and your purpose, not on the server's configuration. That opinion has become a touchstone for every European scraping discussion since.
The consequences are practical, not abstract. If you're in the EU, processing EU residents' personal data, or targeting EU residents — and the GDPR's territorial scope reaches all three — you need a lawful basis for collecting it, you need to honor data-subject rights including deletion, and you need to be able to show your work. The engineering corollary, which I'll come back to: if you can't delete someone's data when they ask, you shouldn't have collected it.
The lawful-basis test
The GDPR gives you a short list of lawful bases for processing personal data: consent, contract, legal obligation, vital interests, public task, and legitimate interests. For scrapers, legitimate interests is the one that does the work, and it's the one most people get wrong.
Legitimate interests isn't a checkbox; it's a balancing test. You have to weigh your interest in collecting and using the data against the rights and freedoms of the people the data is about. Scraping a million email addresses to build a cold-outreach list fails that test so hard it's not close — the individual's interest in not being spammed outweighs your interest in a mailing list, and every regulator in Europe would say so. Scraping company names, titles, and office locations from a public business directory to verify B2B contact data is a much easier case, because the data is less sensitive, the expectations are different, and the purpose is more defensible. But even the easy case requires you to have actually done the balancing and documented it. "We thought it was fine" is not a lawful basis.
The minimization principle adds a second constraint: you may only collect what you actually need for the purpose you've declared. Scraping a page that happens to contain phone numbers and birthdays when you only needed names is a minimization failure. The GDPR doesn't require you to be an angel; it requires you to be deliberate. Collect the minimum, for a documented purpose, with a documented basis, and build the deletion machinery before you start.
Enforcement against AI scrapers, 2023-2026
The GDPR's scraping teeth have been bared most visibly against the AI-training scrapers, because that's where personal data meets the world's largest crawls. The pattern started with Clearview AI, whose entire business was scraping publicly available facial images from the internet. Multiple European regulators looked at that model and each one, independently, concluded it violated the GDPR. The fines tell the story better than any paragraph of analysis.
Walk the bars left to right. The Italian Garante fined Clearview €20 million in 2022. The French CNIL fined it another €20 million in 2022. Greece's DPA added a third €20 million in 2022. In 2023, the Garante fined OpenAI €15 million over ChatGPT's use of personal data in training — the same underlying activity as scraping, just downstream of it. In 2024, the Netherlands DPA hit Clearview with €30.5 million. And Denmark, rather than a fine, published the formal position that publicly accessible data is not free for the taking.
Read the lesson straight: the European risk for scraping personal data is not theoretical, and it is not one-shot. These were not obscure regulators reaching for publicity; they were four separate national authorities, applying the same regulation, and arriving at the same result. If your scrape touches EU residents' personal data and you don't have a defensible basis, you are not facing one fine. You are facing one fine per member state that notices you.
The EU AI Act and training data
The AI Act, in force since August 2024, doesn't criminalize scraping, but it adds a layer of obligations on top of the GDPR for exactly the kind of scraping AI companies do. For general-purpose AI models, the AI Act requires providers to publish a sufficiently detailed summary of the training data they used, and the transparency obligations for general-purpose AI started applying in August 2025. If your model was trained on scraped web data, you now have to be able to say what was in it, and you have to be able to produce that documentation to regulators.
More importantly for the copyright side, the AI Act explicitly requires providers to comply with the rights-reservation mechanisms that publishers have set up. That's a legal recognition, at EU regulation level, of the machine-readable opt-outs we'll cover in the AI training section: if a publisher said "no" via robots.txt or a similar signal, and you scraped anyway, you're not just in copyright trouble — you're in breach of a direct statutory obligation. The AI Act is the first major law in the world to put training-data scraping on a statutory footing rather than leaving it entirely to courts.
Copyright: the DSM Directive and text-and-data-mining opt-outs
Underneath the personal-data layer sits copyright, and for AI training that's where the real EU fight is. The 2019 Copyright Directive (the DSM Directive) created a text-and-data-mining (TDM) exception: you can reproduce and extract works from datasets for TDM purposes without infringing copyright. That sounds like a free pass for AI training, and for a while AI companies read it that way.
But the DSM Directive built in a catch. For TDM on data accessed lawfully — the Article 4 commercial exception — rightsholders can reserve their rights and opt out, and the directive says the reservation must be made "in an appropriate manner, such as machine-readable means." Which machine-readable means do you think the entire web already runs? Exactly: robots.txt. A publisher that adds a TDM-related robots.txt rule — or a "no training" clause in a header, or a terms-of-use reservation — has, in the EU's construction, opted out of the TDM exception for their content. Scrape it for training anyway, and the copyright analysis flips from "exempted" to "infringing."
The AI Act then made the opt-outs enforceable as a matter of regulatory law rather than just copyright, which is why the entire AI-crawler ecosystem — GPTBot, ClaudeBot, Google-Extended, PerplexityBot and the rest — exists in the form it does. Those user-agents exist, in significant part, so that publishers can express their TDM opt-out in a machine-readable way that both the DSM Directive and the AI Act will honor. The robots.txt you thought was a polite convention is now, in Europe, a statutory rights-reservation mechanism with regulatory teeth.
The UK and Commonwealth: the Computer Misuse Act
The UK took a different path from both the US and the EU. Its primary statute is the Computer Misuse Act 1990, and it predates the modern web. Three sections matter. Section 1 criminalizes unauthorized access to computer material. Section 2 covers unauthorized access with intent to commit further offenses. Section 3 covers unauthorized acts that impair the operation of a computer. For scrapers, the question under all three is the same one the CFAA fights over: what makes access "unauthorized"?
The CMA's answer is different from Van Buren's in one important respect. Section 17(5) defines authorization in terms of what the person who controls the computer has agreed to. UK courts and commentators have read this to mean that authorization can be defined by the site owner's stated terms — including a ToS or robots.txt — which makes the UK statute, on paper, more favorable to platforms than the post-Van Buren CFAA. If the ToS says "no automated access" and you scrape anyway, a UK court has more room to say you exceeded your authorization than a US court does.
The catch, and it's a real one: the CMA has never been cleanly tested against modern scraping in a high-profile case. There's no UK hiQ. The statute was written for hackers, not crawlers, and courts have spent far more time on the hacking end of it. So you're left with a statute that looks like it could reach scraping, an academic literature that mostly says it probably does when access controls are bypassed, and very little case law confirming it. The honest description is: in the UK, scraping public data is probably not a CMA offense, scraping behind authentication probably is, and ToS-plus-robots.txt sits in an unsettled middle where a court could go either way.
The Commonwealth follows roughly the same pattern. Canada, Australia, and New Zealand have computer-crime statutes modeled on or inspired by the CMA, and their privacy statutes — PIPEDA in Canada, the Privacy Act in Australia — apply the same "public personal data is still personal data" logic. Australia's Privacy Act is in the middle of its biggest reform in decades, and the reform's scope explicitly targets data scraping and the data-broker trade. The direction of travel across the Commonwealth is the same as everywhere else: public data is scrapable, personal data is regulated, and access-control circumvention is where the criminal exposure lives.
robots.txt and the law
I'm not going to re-explain robots.txt mechanics here — the ethics post on this site covers how to read it and why it exists. What belongs in a legal explainer is what robots.txt does legally, and the answer is three distinct things that compound:
First, it's evidence of authorization. Under the CFAA, the CMA, and their analogues, the key question is often whether you accessed a system without authorization. A robots.txt that says Disallow: / for your user-agent is the site owner's written statement that they did not authorize you to access those paths. It's not conclusive — the whole hiQ fight was about how much weight it carries — but it's evidence, and it's evidence the site owner can hand to a court.
Second, it's a ToS hook. Most ToS incorporate robots.txt by reference or ban automated access outright. When the ToS says "no scraping" and robots.txt says "no access," you've been told twice, in two different formats, and a court sees both as the owner's consistent expression of intent. That compound signal is what makes the breach analysis easy for the plaintiff.
Third, and most consequential post-Van Buren, it's the line between "public data" and "gated data." After Van Buren, the CFAA's teeth depend on whether there's a gate. Some courts — including the Ninth Circuit in 2022 — have treated a robots.txt block as part of an "effective" access-control story. The strongest reading for platforms: a site that disallows your bot in robots.txt and enforces it with technical blocks has turned "please don't" into "you can't," and the latter is what the CFAA cares about.
There's a scene every scraping engineer has seen: robots.txt says no, the site also runs Cloudflare, and you're weighing whether a residential proxy is worth it. From a pure law standpoint, that decision is a step-function change in risk. Ignoring robots.txt is a weak legal signal on its own. Bypassing Cloudflare's bot defense is circumvention of an access control — it's the fact that, on every theory in every jurisdiction, makes your "it's public" argument evaporate. This is exactly why the operating checklist at the end of this post treats "don't bypass anything" as a hard rule rather than a suggestion.
Terms of Service: contract, not crime
Let me be precise about the ToS question, because it's the source of endless confusion. Violating a ToS is a breach of contract, not a crime. There is no statute that says "scraping in violation of a ToS is illegal." What there is, in every jurisdiction, is the law of contracts: you agreed to the ToS — courts generally say you did the moment you used the site, under the browse-wrap and click-wrap doctrines — and breaching a contract has civil consequences.
Here's what those consequences actually look like, in the order they hit:
- A ban. The site revokes your access. Your account, IP, or fingerprint goes on a block list. This is immediate, it's automatic, and it's the outcome in 99% of ToS violations.
- A cease-and-desist. If the site noticed you and you're big enough to matter, a lawyer sends a letter. Most scrapers stop here, which is the point of the letter.
- A lawsuit. If the site has a claim it wants to press, it sues — and the cause of action is breach of contract, often bundled with trespass to chattels, tortious interference with contract, or unjust enrichment. Post-Van Buren, this is the default; the CFAA no longer does the job for public data.
- An injunction. The dangerous one. Courts can order you to stop scraping and order you to delete the data you already collected. Injunctions are equitable orders against you personally — violating one is contempt of court, which is a far more serious category than breach of contract, up to and including criminal contempt.
Why do platforms sue for contract breach rather than the CFAA now? Because they win more. A contract claim doesn't require proving a gate or an authorization boundary; it requires proving you agreed to a term and broke it. And it has a second benefit for the platform: it's easy to win on the merits, which means the platform doesn't have to rely on novel legal theories that a court might reject. The practical posture to internalize: the ToS you didn't read is the most likely source of your lawsuit, and it's a lawsuit you'll probably lose.
Login walls and authenticated data
Every serious legal exposure in scraping funnels to the same place: content behind a login is not public, and scraping it is a categorically different activity. Not "a bit riskier" — a different category, the difference between reading a newspaper in the lobby and going through the locked side door.
Here's why the category flips. On the CFAA analysis, a login wall is the textbook gate that Van Buren was talking about. You authenticated, which means you crossed a gate you weren't authorized to cross, which means "exceeds authorized access" now fits — the access control isn't a polite request, it's a technical barrier, and the courts are much more comfortable treating a technical barrier as the line. On the contract analysis, the ToS you agreed to at signup almost certainly says the account is for personal use and bans automated access, so you're in breach from the moment your scraper logs in. And on the data analysis, the stuff behind the login is frequently other people's data — private messages, account details, dashboards — which is a personal-data problem on top of the access problem.
The same logic extends to every access-control bypass: CAPTCHAs, IP blocks, fingerprinting evasion, residential-proxy rotation to dodge bans. All of it is circumvention, and circumvention is the single fact that weakens every defensive argument you might have. A site that's actively defending against bots is telling you "no automated access" in machine language; the defensible move is to hear it.
Does that mean scraping behind auth is never legitimate? No. Some of the most valuable scraping in the world happens against your own accounts, your own data, with the site's blessing — think of a company extracting its own analytics. And some sites explicitly permit API access with authentication. The dividing line is permission, and the permission has to come from whoever controls the data. The safe default, the one I apply: never scrape behind a login without written permission from the party who can grant it.
Personal data, emails, phone numbers
This section is short because the ethics post covers the lawful-basis test in detail, but the legal stakes deserve their own emphasis. The GDPR/UK-GDPR framework, applied to scraping, comes down to three rules:
- The lawful-basis rule. Collecting personal data requires a basis — consent, contract, legal obligation, legitimate interests, and the rest — and the burden is on you to have one and to be able to show it. "It was public" is not on the list.
- The minimization rule. You may only collect what your documented purpose requires. If a page contains more than you need, you collect less, not more.
- The deletion rule. You must honor access and deletion requests, and that's an engineering obligation. If you never built the delete pipeline, you shouldn't have built the crawler.
Emails and phone numbers get a fourth rule, because they sit under two regimes at once. They're personal data under GDPR, and they're communication channels under anti-spam law: CAN-SPAM in the US, the ePrivacy Directive and GDPR marketing rules in the EU. The US has an opt-out regime — you can email, but you must honor unsubscribes. The EU has an opt-in regime — unsolicited commercial email is illegal without prior consent, in most cases, no unsubscribe link can save you. If your business model is "scrape email addresses, then send everyone offers," the scraping is the smaller problem; the outreach is where the fines are. Every serious scraper I know has a hard rule about this: contact details are collected only with a basis, and outreach only happens through channels that permit it.
Scraping for AI training: the 2024-2026 fights
No subject has reshaped scraping law faster than AI training. This is where the "generally lawful" verdict breaks down into a full-scale legal war, and it's the reason this post exists in 2026 rather than 2020. Let me give you the map.
The crawlers arrived, then the lawsuits arrived. Starting in mid-2023, the major AI labs shipped dedicated crawlers — GPTBot and OAI-SearchBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot — each with its own user-agent and a claim that they honor robots.txt. Publishers reacted fast, blocking the AI crawlers in robots.txt at a pace the web had never seen. Cloudflare launched its AI Audit dashboard in 2024, giving site owners visibility into which AI bots were hitting them and a one-click block. By 2026, roughly a third of the largest sites block at least one major AI crawler, and the number is still climbing.
The litigation timeline tells the rest of the story.
The cases break into two buckets. The copyright bucket: The New York Times v. OpenAI and Microsoft, the Authors Guild suits, Getty Images v. Stability AI, the Daily News and other publishers against Microsoft and OpenAI, and the sleeper that changed the analysis — Thomson Reuters v. Ross Intelligence, where a US court ruled in 2025 that training on copyrighted headnotes was not fair use. That decision landed like a brick; "training = fair use" was the AI industry's core legal theory, and a federal court just said no. The fair-use fight is far from over, and the NYT case may still rewrite the rules, but the assumption that training on scraped content was safe collapsed in 2025.
The rights bucket is where scrapers of any size actually live. Cloudflare's AI Audit turned robots.txt blocks into a dashboard metric. The major AI labs built opt-out processes, most notably via their dedicated crawler user-agents. The EU AI Act made training-data summaries and rights-reservation compliance statutory. And a small industry of opt-out registries, licensing brokers, and content-licensing deals grew up in 2025 and 2026 to sit between "blocked crawler" and "settled case." The practical upshot for anyone who scrapes data for a model: robots.txt is now a legally meaningful opt-out signal for training in the EU, and US courts are increasingly treating ignored opt-outs as evidence in fair-use analysis. "But it was public" is not the defense it was in 2020, because the question is no longer just access — it's what you did with the content.
Jurisdiction by jurisdiction
The global picture, one line per jurisdiction. This is a summary for orientation, not a substitute for local advice; the enforcement reality on the ground varies more than the statutes suggest.
| Jurisdiction | Primary laws | Public non-personal data | Personal data | Behind a login or access control |
|---|---|---|---|---|
| United States | CFAA + state computer-crime laws + contract + tort | Generally lawful | Regulated, lightly enforced | Risky |
| European Union | GDPR + DSM Directive + AI Act | Lawful | Needs lawful basis; heavily enforced | Very risky |
| United Kingdom | Computer Misuse Act 1990 + UK GDPR | Probably lawful | Needs lawful basis | Risky; untested statute |
| Canada | PIPEDA + Criminal Code | Generally lawful | Regulated | Risky |
| Australia | Privacy Act (reforming) + state computer laws | Generally lawful | Regulated; reform targets scraping | Risky |
| China | PIPL + anti-unfair-competition law | Platform data protected; courts side with platforms | Heavily regulated | Very risky |
| India | IT Act + DPDP Act 2023 | Largely unregulated | DPDP Act applies | Risky |
| Japan | Unfair Competition Prevention Act + APPI | Generally lawful | Regulated | Restricted-data rules bite |
A few notes that don't fit in a table. China is the outlier in the other direction: Chinese courts have repeatedly sided with platforms over scrapers, treating large-scale scraping of platform data as unfair competition even when the data was technically public, and the Personal Information Protection Law applies its own rigorous regime to personal data. If your business touches Chinese platform data, assume the platform's view wins in court. India is the opposite extreme: scraping law is thin, the IT Act's hacking provisions are mostly aimed at malware, and the 2023 DPDP Act is still ramping up. Japan sits in the interesting middle: since the 2018 amendments to the Unfair Competition Prevention Act, scraping "limited provision data" — data the owner provides to specific people under restricted terms — is itself an unfair-competition act, which gives Japanese platforms a statutory handle that most jurisdictions lack.
A risk matrix you can keep on the wall
Let me compress the whole post into the two artifacts I actually use. The first is a heatmap of risk by data type and jurisdiction. The second is the "what's okay / what's risky" table. Neither is a substitute for judgment; both are what judgment looks like before you start.
| Scenario | Legal posture | What can go wrong | Verdict |
|---|---|---|---|
| Plain public HTML, politely, no rules against it | CFAA likely inapplicable post-Van Buren | Ban; a ToS clause you missed | Fine, politely |
| Public API, no auth, documented | Use the API; honor its limits | Rate-limit ban; ToS API terms | Fine, gently |
| Disallowed by robots.txt | Weak signal alone, but evidence of non-authorization | Ban; strengthens a CFAA or contract claim | Don't, without a reason |
| ToS forbids automated access | Breach of contract; enforceable | Suit, damages, injunction to stop and delete | No |
| Login-gated page | CFAA + contract + possibly other data | Criminal exposure, account termination, suit | No, without written permission |
| Bypassing CAPTCHA / IP block / bot defense | Circumvention of an access control | CFAA exposure, ban, and it kills every "public data" defense | No |
| Public personal data | GDPR/UK-GDPR requires a basis | Fines up to 4% of global turnover; deletion orders | No without a documented basis |
| Emails / phone numbers for outreach | Personal data + anti-spam law | ePrivacy fines; CAN-SPAM penalties; GDPR fines | No in the EU without consent |
The operating checklist: how to stay on the right side
This is the list I wish someone had handed me before my first scrape. It's the operational version of everything above, in the order you should apply it:
- Public, non-personal data only. If you can't point at a page and say "any human with a browser can see this," you're not in "public data" territory anymore.
- No auth walls, no paywalls, no bypassing. If there's a login, a CAPTCHA, an IP block, or a bot-defense system, the scrape is in a different legal category. Don't route around it. This rule is the one that keeps you out of CFAA and Computer Misuse Act territory.
- Read robots.txt and honor it. It's evidence of authorization, a ToS hook, and part of the access-control story. If the answer is no for your user-agent, treat that as the owner's written position and go ask or go elsewhere. (Mechanics live in the ethics post.)
- Read the ToS before you write a line of code. Thirty seconds now saves a lawyer later. If it says "no automated access," you now know your scrape is a contract breach, and you should be able to justify that consciously rather than discovering it in a lawsuit.
- Minimize personal data to near zero. If your dataset could identify a person, treat it as regulated. Collect only what your documented purpose requires — not the whole page because it was easy.
- Document your lawful basis. If you're in the EU/UK, or touching EU/UK residents' data, write down your purpose, your basis, and your balancing-test reasoning before you start. A documented legitimate-interest analysis is a defense; a memory of having thought about it is not.
- Build the deletion path before the crawler. If you hold personal data, you must be able to honor access and deletion requests. If you can't, you shouldn't have collected it. That's a systems design constraint, not an abstraction.
- Go gently. Rate-limit harder than the site requires, cache so you never re-request, honor 429s, and identify yourself with a real User-Agent when scraping permitted targets. Politeness is the cheapest legal and anti-ban insurance there is.
- Prefer the official channel. If the site offers an API or a data dump, use it. It converts the whole question from "can I scrape this?" to "what does the contract allow?" — and it's usually faster and cheaper too.
- Have a stop-when-asked plan. If you get a cease-and-desist, the answer is to stop and talk to a lawyer — not to rotate proxies. The scrapers who get destroyed by litigation are the ones who treated a legal letter as a technical challenge.
What happens when you get it wrong
Every legal risk in this post translates into a concrete sequence of events, and it's worth walking through the actual escalation path, because the shape of it is the same everywhere. It starts long before any courtroom.
Step one: the ban. Your IP or account gets blocked. This is automatic, it happens fast, and for most scrapers it's the whole story. The site noticed you, decided you weren't worth the bandwidth, and shut the door. No lawyer, no letter, no drama. Just a 403 and a blocklist entry.
Step two: the cease-and-desist. If you're big enough to matter, or the site is sophisticated, a lawyer's letter arrives. It will cite the ToS, possibly robots.txt, and threaten the CFAA, the GDPR, or whatever else fits. The letter is a negotiation, not a verdict, but the correct response is to take it seriously — this is the point where the cost of continuing starts to compound.
Step three: the lawsuit. If the site has a commercial interest in stopping you — and it usually does — the letter becomes a complaint. The causes of action are the ones we've covered: breach of contract, trespass to chattels, tortious interference, unfair competition, the CFAA where it fits, and in Europe, a complaint to a data-protection authority. The point of the lawsuit is not always money; often it's an injunction to make you stop and delete the data.
Step four: the fine or the injunction. Europe is where the fine arrives. Clearview AI is the canonical case — scraped public personal data, was hit with a sequence of multimillion-euro fines across member states, each one for the same underlying conduct. In the US, the equivalent is the injunction: a court order that you stop and delete, enforceable by contempt. The Thomson Reuters v. Ross Intelligence case shows the damages side of it — a judgment for the copyright owner over training on scraped content.
Here's the honest bottom line on consequences. The realistic, most-common outcome for a run-of-the-mill scrape that crosses a line is: ban, then letter, then you stop. The catastrophic outcomes — fines in the tens of millions, injunctions, contempt — are reserved for the people who ignored the escalations, who scraped personal data at scale in Europe, who trained models on opted-out content, or who treated a cease-and-desist as a performance review. The pattern in every case that went badly is the same: the scrapers were told no, they knew they were being told no, and they kept going. Don't be that person. The web has enough room for people who ask.
Key takeaways
- Verdict: scraping public, non-personal data, politely, without bypassing anything, is generally lawful in most jurisdictions. "Generally" is doing a lot of work.
- Public is not allowed. Public is a fact about the server; allowed is a statement about intent. Most legal trouble comes from confusing them.
- The CFAA was narrowed by Van Buren and is no longer the weapon platforms want for public data; they now sue for breach of contract, and contract claims are winnable.
- hiQ v. LinkedIn settled with the core question unanswered — the 2019 "public data is fine" ruling was withdrawn on remand in 2022. Don't build a business on a court case that was never resolved.
- In the EU/UK, public personal data is still personal data. It needs a lawful basis, minimization, and a deletion path. The Clearview fines prove the regulators enforce it.
- Behind a login, a paywall, or an access control is a different category. Circumvention is the fact that makes every other argument fail.
- For AI training, the ground shifted. US courts rejected the "training is fair use" assumption in 2025; the EU AI Act and DSM Directive made robots.txt opt-outs statutorily meaningful.
- The ToS you didn't read is the most likely source of your lawsuit. Read it before you scrape.
Further reading
For the ethics side and the robots.txt mechanics that this post deliberately skipped, see Web scraping ethics and robots.txt (internal). For the practical side of actually building a scraper — and the build-vs-buy decision that should come before you write any crawler code — see Website Content Extraction API: The 2026 Guide and Best Web Crawler APIs in 2026: Build vs Buy.
Frequently Asked Questions
Is web scraping legal?
In most jurisdictions, scraping public, non-personal data is generally lawful — but public is not the same as allowed. US case law has favored scraping public data (hiQ v. LinkedIn, X v. Bright Data), yet the picture flips once you cross into personal data, authenticated areas, or a site's Terms of Service. Whether scraping is legal depends on what you scrape, how you scrape it, and where you are. This post is education, not legal advice.
Is scraping public data legal in the US?
Generally yes for plain, public, non-personal data. The Ninth Circuit ruled in hiQ v. LinkedIn that scraping publicly visible data does not violate the CFAA, and X v. Bright Data dismissed contract claims over public-tweet scraping in 2024. But a ToS that bans automated access, a robots.txt block, or bypassing an access control can change the answer, and the Supreme Court never settled the CFAA question definitively.
Does violating a website's Terms of Service make scraping illegal?
It makes it a breach of contract, which is a civil wrong, not a crime. A ToS breach can still get you banned, sued for damages, and hit with an injunction. After Van Buren, platforms increasingly sue scrapers for breach of contract rather than under the CFAA, because the CFAA claim is harder to win now.
Is scraping personal data illegal under GDPR?
Scraping personal data in the EU or UK requires a lawful basis, even when the data is public. 'It was on a public website' is not a defense. Regulators have fined companies like Clearview AI tens of millions of euros for scraping publicly available facial images, and the Danish DPA has said publicly accessible data is not a free-for-all.
Is web scraping for AI training legal?
There is no global answer. In the US it is being fought out in copyright courts (NYT v. OpenAI, Thomson Reuters v. Ross Intelligence). In the EU, the DSM Directive's text-and-data-mining exception lets rightsholders opt out via machine-readable means, and the EU AI Act now requires transparency about training data. The law is genuinely unsettled and moving.
Can I get sued for scraping?
Yes. Platforms can sue for breach of contract, trespass, tortious interference, or the CFAA, and regulators can fine you under GDPR or similar laws. The realistic sequence is ban, then cease-and-desist, then lawsuit. The scrapers who get sued are almost always the ones who ignored a clear 'no' or caused measurable harm to the target.
Does robots.txt have the force of law?
No statute makes ignoring robots.txt illegal by itself — it is a convention, not a law. But it serves as evidence of authorization under computer-misuse statutes, gives a ToS breach more teeth, and is the fastest way to get IP-banned. Disobeying it makes every other legal argument you have weaker.
Is scraping behind a login ever legal?
Only with explicit permission from whoever controls the data. Authenticated data is categorically different from public data under the CFAA, the UK Computer Misuse Act, and GDPR. Scraping behind a login stacks contract breach on top of access-control violations, and it often involves other people's data as well.
What is the difference between public data and allowed data?
Public is a fact about the server: anyone with a browser and the URL can see it. Allowed is a statement about intent: the site owner consents to automated access. Most legal trouble comes from confusing the two. Public data is not automatically free for bulk automated collection.
Can I scrape email addresses and send marketing emails?
Collecting email addresses is personal-data processing that needs a lawful basis, and contacting those people is separately regulated by anti-spam law — CAN-SPAM in the US, the ePrivacy Directive and GDPR marketing rules in the EU. In most of the world, unsolicited commercial email to scraped addresses is illegal. The scraping is often the smaller problem; the outreach is where you get fined.
Keep reading
Web scraping ethics and robots.txt: the lines you don't cross
A clear, practical guide to scraping ethically and legally: reading robots.txt, Terms of Service, the CFAA and EU equivalents, login walls, personal data, and the difference between 'public' and 'allowed'.
Web Scraping for AI Training Data in 2026
How AI companies build training datasets: Common Crawl, web corpora, domain scraping, the crawl-to-JSONL pipeline, what makes good data, the 2026 legal landscape, and how a small team builds its own.
What Actually Gets You Blocked When Web Scraping (2026)
We run a web scraping API and see millions of requests a day. Here's what actually gets you blocked in 2026 — and what doesn't, signal by signal.
Found this useful? Cite it as: webscraping.space. “Is Web Scraping Legal? The 2026 Legal Landscape, Explained.” https://webscraping.space/blog/web-scraping-legal. Published 2026-08-08.