Robots.txt and Web Scraping: What It Means for Your Scraper

Robots.txt is a plain-text file at the root of a website that tells crawlers which paths the site owner doesn't want them to visit. For web scraping it's the first thing to read: it shows which parts of a site are off limits to bots, where the sitemaps are, and sometimes how the owner feels about automated access in general. It is a request, not a lock: the standard itself says its rules "are not a form of access authorization."

This guide explains how the rules work under RFC 9309, the standard published in 2022, walks through a real robots.txt file, shows how to check URLs in Python (and a case where Python's built-in parser gives a different answer from the standard), covers what robots.txt does and doesn't decide legally, and shows how to respect it in a Scrape.ly crawl.

How robots.txt works

A crawler fetches https://example.com/robots.txt before it crawls the site. The file is a list of groups. Each group starts with one or more User-agent lines and holds Allow and Disallow rules for those crawlers:

User-agent: *
Disallow: /checkout/
Disallow: /search
Allow: /search/help

User-agent: ExampleBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

The rules that matter, from RFC 9309, the Robots Exclusion Protocol standard:

• Pick the group for your crawler. A crawler uses the group whose User-agent matches its name, and falls back to *. ExampleBot above may crawl nothing; everyone else follows the first group.
• The longest match wins. "The most specific match found MUST be used. The most specific match is the match that has the most octets." So /search/help is allowed even though /search is disallowed. If an allow and a disallow rule are equally specific, the RFC says the allow rule should win.
• Paths are prefixes, with two wildcards. Disallow: /search blocks /search, /search?q=x and /searching. * matches any characters and $ anchors the end, so Disallow: /*.pdf$ blocks PDFs.
• Matching is case sensitive and starts at the beginning of the path.
• Sitemaps are listed here too. Sitemap: lines point to the site's URL lists, often the easiest way to find every product or article page.

What happens when there's no robots.txt

The RFC is specific about errors. If the file returns a 4xx status such as 404, "the crawler MAY access any resources on the server". If the server errors out with a 5xx, the file is undefined "and the crawler MUST assume complete disallow." Crawlers "SHOULD NOT use the cached version for more than 24 hours", so re-read the file daily on long jobs.

Crawl-delay isn't in the standard

Many files contain Crawl-delay: 10. It was never part of RFC 9309, and Google's robots.txt documentation says it supports user-agent, allow, disallow and sitemap, and that "other fields such as crawl-delay aren't supported". Treat a crawl delay as the site owner's stated wish and keep your request rate well below it.

Reading a real robots.txt

Target's robots.txt, fetched on 27 October 2026, starts with six Sitemap: lines and then has a single group for every crawler, User-agent: *, with a long list of Disallow rules. Line 117 is:

Disallow: /s?

That blocks Target's search result pages (/s?searchTerm=...) for all crawlers, while product pages (/p/...) and category pages (/c/...) stay allowed. For a scraper, the file says: get products from categories, sitemaps or product URLs, not by crawling search results. Our Target tutorial shows that in practice.

Patterns you'll meet in the wild

A few shapes come up again and again:

• The near-empty file. User-agent: * with a blank Disallow: allows everything. Many small sites have exactly that, or no file at all.
• Private areas only. Checkout, cart, account, admin and internal search paths disallowed, everything else allowed. That's the Target pattern above, and the most common one on retail sites.
• Named AI crawlers. Extra groups for specific bots, such as OpenAI's GPTBot or Common Crawl's CCBot, with Disallow: /. They don't affect other crawlers, but they're a clear statement of how the owner feels about AI training use.
• Everything closed. User-agent: * with Disallow: /. The owner doesn't want any crawler; treat that as a strong signal and read the terms before going further.

How to check robots.txt in Python

Python's standard library has urllib.robotparser. Tested on 27 October 2026 with Python 3.14:

from urllib.robotparser import RobotFileParser

rp = RobotFileParser("https://www.target.com/robots.txt")
rp.read()
for path in ["/p/coffee-maker/-/A-91863923", "/s?searchTerm=coffee+maker", "/c/coffee-makers/-/N-4ysy0"]:
    print(rp.can_fetch("*", "https://www.target.com" + path), path)
print("crawl-delay:", rp.crawl_delay("*"))

True /p/coffee-maker/-/A-91863923
False /s?searchTerm=coffee+maker
True /c/coffee-makers/-/N-4ysy0
crawl-delay: None

It agrees with the file here, but it has two behaviors worth knowing before you rely on it:

• It applies the first matching rule, not the longest. With the rules Disallow: /shop/ then Allow: /shop/public/, RFC 9309 allows /shop/public/item. urllib.robotparser blocks it. Here is the test, against Protego 0.7.0, the RFC-style parser Scrapy uses:

from urllib.robotparser import RobotFileParser

from protego import Protego

rules = "User-agent: *\nDisallow: /shop/\nAllow: /shop/public/\n"
std = RobotFileParser(); std.parse(rules.splitlines())
rfc = Protego.parse(rules)
url = "https://example.com/shop/public/item"
print("urllib.robotparser:", std.can_fetch("*", url))
print("Protego (RFC 9309):", rfc.can_fetch(url, "*"))

urllib.robotparser: False
Protego (RFC 9309): True

• A 401 or 403 means "disallow everything". In CPython's source, read() sets disallow_all when fetching the file returns 401 or 403, and allows everything on other 4xx codes. Sites that block Python's default user agent therefore look fully closed. Fetch the file yourself with a real user agent and pass the text to parse().

For anything beyond a quick check, use Protego. Fetch the file with your crawler's user agent, parse it, and ask per URL:

import urllib.request

from protego import Protego

robots_url = "https://www.target.com/robots.txt"
req = urllib.request.Request(robots_url, headers={"User-Agent": "my-research-bot/1.0"})
rp = Protego.parse(urllib.request.urlopen(req).read().decode())
for path in ["/p/coffee-maker/-/A-91863923", "/s?searchTerm=coffee+maker", "/c/coffee-makers/-/N-4ysy0"]:
    print(rp.can_fetch("https://www.target.com" + path, "my-research-bot"), path)
print("sitemaps:", len(list(rp.sitemaps)))

True /p/coffee-maker/-/A-91863923
False /s?searchTerm=coffee+maker
True /c/coffee-makers/-/N-4ysy0
sitemaps: 6

What robots.txt means legally

Robots.txt isn't a law, and following it doesn't make a scrape lawful by itself. But it increasingly matters in three places:

• Terms of service point to it. Google's Terms of Service (effective July 30, 2026) prohibit "using automated means to access content from any of our services in violation of the machine-readable instructions on our web pages (for example, robots.txt files that disallow crawling, training, or other activities)".
• EU copyright uses it as an opt-out. The text and data mining exception in Article 4 of the EU Copyright Directive applies unless rightholders have reserved their rights "in an appropriate manner, such as machine-readable means in the case of content made publicly available online". A robots.txt rule aimed at AI crawlers is a widely used form of that reservation, although European courts are still settling exactly what counts.
• Data protection regulators weigh it. France's CNIL, in its June 2025 focus sheet on web scraping, lists among the required measures excluding "websites that clearly oppose the scraping of their content for the purpose of creating training databases by using robots.txt exclusion protocols". Our GDPR and web scraping guide covers the rest of that analysis.

The RFC is also blunt about what robots.txt is not: "The Robots Exclusion Protocol is not a substitute for valid content security measures." A path listed in robots.txt is public knowledge, not protected. For the wider legal picture, see is web scraping legal? This is not legal advice; for a commercial project, ask a lawyer about your case.

How to respect robots.txt with Scrape.ly

The Website Content Crawler and the Web Scraper both have a Respect the robots.txt file option (respectRobotsTxtFile). With it on, the crawler consults the site's robots.txt before each page and skips disallowed URLs:

{
  "startUrls": [{ "url": "https://docs.python.org/3/tutorial/" }],
  "maxCrawlPages": 50,
  "respectRobotsTxtFile": true,
  "proxyConfiguration": { "useScrapelyProxy": true }
}

Two details from the input's own documentation: the crawler "does not use any specific user agent identifier", so it follows the * group, and "The crawl-delay directive is also not supported yet", so set maxConcurrency low yourself on small sites. The option is pre-filled on in the console form.

Ready-made Store Actors for specific sites usually read a site's data feeds rather than crawling page by page. Before a commercial project, read the target's robots.txt and terms yourself, as our tutorials do in their "Is it legal" sections.

A robots.txt checklist for scrapers

1. Fetch /robots.txt with the user agent your crawler will use, and keep the text with your run.
2. Find your group, or *, and apply the longest-match rule.
3. Use the Sitemap: lines to find pages instead of crawling everything.
4. Respect a 5xx as "disallow all" and re-read the file at least daily.
5. Keep request rates gentle, crawl-delay or not; our web scraping best practices cover rate limits.
6. Read the terms of service too. Robots.txt is one signal, not the whole answer.

FAQ

Is robots.txt legally binding?

Not by itself. It's a technical convention, standardized in RFC 9309, which says its rules "are not a form of access authorization". It does matter legally where terms of service, copyright opt-outs or data protection rules refer to it.

Does robots.txt apply to scrapers or only to search engines?

The protocol applies to any crawler. The User-agent: * group covers every bot that isn't named elsewhere in the file.

What does "Disallow: /" mean?

It asks the matching crawlers not to crawl anything on the site. Disallow: with an empty value means nothing is disallowed.

What if a site has no robots.txt?

Under RFC 9309, a 4xx response such as 404 means the crawler may access any resources. A 5xx error means it must assume everything is disallowed until the file can be read.

Does Scrape.ly respect robots.txt?

The Website Content Crawler and the Web Scraper do when you turn on Respect the robots.txt file. They follow the * group and don't support crawl-delay yet.