--- title: "Robots.txt and Web Scraping Explained | Scrape.ly" description: "Robots.txt and web scraping: how the rules work under RFC 9309, how to check URLs in Python, what robots.txt means legally, and how to respect it." url: https://scrape.ly/blog/robots-txt-web-scraping markdown: https://scrape.ly/blog/robots-txt-web-scraping.md --- [Guides](https://scrape.ly/blog/category/guides.md) # Robots.txt and Web Scraping: What It Means for Your Scraper What robots.txt means for web scraping: how the rules work under RFC 9309, how to read a real file, how to check URLs in Python (and where the standard library gets it wrong), what robots.txt does and doesn't decide legally, and how to respect it in a Scrape.ly crawl. [Scrape.ly Team The team behind Scrape.ly](https://scrape.ly/blog/author/scrapely-team.md) - Published Oct 27, 2026 - Reading time 7 min read ![Pixel-art robot with an orange >\_ on its terminal head walks away from a robots.txt file with a robot face on its plate and rule lines ticked green or crossed red, through a raised red-and-white barrier towards a web page](https://scrape.ly/media/robots-txt-web-scraping.png?w=1080) Robots.txt is a plain-text file at the root of a website that tells crawlers which paths the site owner doesn't want them to visit. For web scraping it's the first thing to read: it shows which parts of a site are off limits to bots, where the sitemaps are, and sometimes how the owner feels about automated access in general. It is a request, not a lock: the standard itself says its rules "are not a form of access authorization." This guide explains how the rules work under RFC 9309, the standard published in 2022, walks through a real robots.txt file, shows how to check URLs in Python (and a case where Python's built-in parser gives a different answer from the standard), covers what robots.txt does and doesn't decide legally, and shows how to respect it in a Scrape.ly crawl. ## How robots.txt works A crawler fetches `https://example.com/robots.txt` before it crawls the site. The file is a list of groups. Each group starts with one or more `User-agent` lines and holds `Allow` and `Disallow` rules for those crawlers: ```plaintext Useragent: * Disallow: checkout Disallow: search Allow: searchhelp Useragent: ExampleBot Disallow: Sitemap: https://example.com/sitemap.xml ``` The rules that matter, from [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html), the Robots Exclusion Protocol standard: - **Pick the group for your crawler.** A crawler uses the group whose `User-agent` matches its name, and falls back to `*`. `ExampleBot` above may crawl nothing; everyone else follows the first group. - **The longest match wins.** "The most specific match found MUST be used. The most specific match is the match that has the most octets." So `/search/help` is allowed even though `/search` is disallowed. If an allow and a disallow rule are equally specific, the RFC says the allow rule should win. - **Paths are prefixes, with two wildcards.** `Disallow: /search` blocks `/search`, `/search?q=x` and `/searching`. `*` matches any characters and `$` anchors the end, so `Disallow: /*.pdf$` blocks PDFs. - **Matching is case sensitive** and starts at the beginning of the path. - **Sitemaps are listed here too.** `Sitemap:` lines point to the site's URL lists, often the easiest way to find every product or article page. ### What happens when there's no robots.txt The RFC is specific about errors. If the file returns a 4xx status such as 404, "the crawler MAY access any resources on the server". If the server errors out with a 5xx, the file is undefined "and the crawler MUST assume complete disallow." Crawlers "SHOULD NOT use the cached version for more than 24 hours", so re-read the file daily on long jobs. ### Crawl-delay isn't in the standard Many files contain `Crawl-delay: 10`. It was never part of RFC 9309, and [Google's robots.txt documentation](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt) says it supports user-agent, allow, disallow and sitemap, and that "other fields such as crawl-delay aren't supported". Treat a crawl delay as the site owner's stated wish and keep your request rate well below it. ## Reading a real robots.txt Target's [robots.txt](https://www.target.com/robots.txt), fetched on 27 October 2026, starts with six `Sitemap:` lines and then has a single group for every crawler, `User-agent: *`, with a long list of `Disallow` rules. Line 117 is: ```plaintext Disallow: s? ``` That blocks Target's search result pages (`/s?searchTerm=...`) for all crawlers, while product pages (`/p/...`) and category pages (`/c/...`) stay allowed. For a scraper, the file says: get products from categories, sitemaps or product URLs, not by crawling search results. Our [Target tutorial](https://scrape.ly/blog/scrape-target.md) shows that in practice. ## Patterns you'll meet in the wild A few shapes come up again and again: - **The near-empty file.** `User-agent: *` with a blank `Disallow:` allows everything. Many small sites have exactly that, or no file at all. - **Private areas only.** Checkout, cart, account, admin and internal search paths disallowed, everything else allowed. That's the Target pattern above, and the most common one on retail sites. - **Named AI crawlers.** Extra groups for specific bots, such as OpenAI's `GPTBot` or Common Crawl's `CCBot`, with `Disallow: /`. They don't affect other crawlers, but they're a clear statement of how the owner feels about AI training use. - **Everything closed.** `User-agent: *` with `Disallow: /`. The owner doesn't want any crawler; treat that as a strong signal and read the terms before going further. ## How to check robots.txt in Python Python's standard library has `urllib.robotparser`. Tested on 27 October 2026 with Python 3.14: ```python from urllib.robotparser import RobotFileParser rp = RobotFileParser("https://www.target.com/robots.txt") rp.read() for path in ["/p/coffee-maker/-/A-91863923", "/s?searchTerm=coffee+maker", "/c/coffee-makers/-/N-4ysy0"]: print(rp.can_fetch("*", "https://www.target.com" + path), path) print("crawl-delay:", rp.crawl_delay("*")) ``` ```plaintext True pcoffeemakerA91863923 False s?searchTerm=coffee+maker True ccoffeemakersN4ysy0 crawldelay: None ``` It agrees with the file here, but it has two behaviors worth knowing before you rely on it: - **It applies the first matching rule, not the longest.** With the rules `Disallow: /shop/` then `Allow: /shop/public/`, RFC 9309 allows `/shop/public/item`. `urllib.robotparser` blocks it. Here is the test, against [Protego](https://pypi.org/project/Protego/) 0.7.0, the RFC-style parser Scrapy uses: ```python from urllib.robotparser import RobotFileParser from protego import Protego rules = "User-agent: *\nDisallow: /shop/\nAllow: /shop/public/\n" std = RobotFileParser(); std.parse(rules.splitlines()) rfc = Protego.parse(rules) url = "https://example.com/shop/public/item" print("urllib.robotparser:", std.can_fetch("*", url)) print("Protego (RFC 9309):", rfc.can_fetch(url, "*")) ``` ```plaintext urllib.robotparser: False Protego (RFC 9309): True ``` - **A 401 or 403 means "disallow everything".** In CPython's source, `read()` sets `disallow_all` when fetching the file returns 401 or 403, and allows everything on other 4xx codes. Sites that block Python's default user agent therefore look fully closed. Fetch the file yourself with a real user agent and pass the text to `parse()`. For anything beyond a quick check, use Protego. Fetch the file with your crawler's user agent, parse it, and ask per URL: ```python import urllib.request from protego import Protego robots_url = "https://www.target.com/robots.txt" req = urllib.request.Request(robots_url, headers={"User-Agent": "my-research-bot/1.0"}) rp = Protego.parse(urllib.request.urlopen(req).read().decode()) for path in ["/p/coffee-maker/-/A-91863923", "/s?searchTerm=coffee+maker", "/c/coffee-makers/-/N-4ysy0"]: print(rp.can_fetch("https://www.target.com" + path, "my-research-bot"), path) print("sitemaps:", len(list(rp.sitemaps))) ``` ```plaintext True pcoffeemakerA91863923 False s?searchTerm=coffee+maker True ccoffeemakersN4ysy0 sitemaps: 6 ``` ## What robots.txt means legally Robots.txt isn't a law, and following it doesn't make a scrape lawful by itself. But it increasingly matters in three places: - **Terms of service point to it.** Google's [Terms of Service](https://policies.google.com/terms) (effective July 30, 2026) prohibit "using automated means to access content from any of our services in violation of the machine-readable instructions on our web pages (for example, robots.txt files that disallow crawling, training, or other activities)". - **EU copyright uses it as an opt-out.** The text and data mining exception in Article 4 of the [EU Copyright Directive](https://eur-lex.europa.eu/eli/dir/2019/790/oj) applies unless rightholders have reserved their rights "in an appropriate manner, such as machine-readable means in the case of content made publicly available online". A robots.txt rule aimed at AI crawlers is a widely used form of that reservation, although European courts are still settling exactly what counts. - **Data protection regulators weigh it.** France's CNIL, in its [June 2025 focus sheet on web scraping](https://www.cnil.fr/en/legal-basis-legitimate-interest-focus-sheet-measures-implement-case-data-collection-web-scraping), lists among the required measures excluding "websites that clearly oppose the scraping of their content for the purpose of creating training databases by using robots.txt exclusion protocols". Our [GDPR and web scraping](https://scrape.ly/blog/gdpr-web-scraping.md) guide covers the rest of that analysis. The RFC is also blunt about what robots.txt is not: "The Robots Exclusion Protocol is not a substitute for valid content security measures." A path listed in robots.txt is public knowledge, not protected. For the wider legal picture, see [is web scraping legal?](https://scrape.ly/blog/is-web-scraping-legal.md) This is not legal advice; for a commercial project, ask a lawyer about your case. ## How to respect robots.txt with Scrape.ly The [Website Content Crawler](https://scrape.ly/store/querystone/website-content-crawler.md) and the [Web Scraper](https://scrape.ly/store/querystone/web-scraper.md) both have a **Respect the robots.txt file** option (`respectRobotsTxtFile`). With it on, the crawler consults the site's robots.txt before each page and skips disallowed URLs: ```json { "startUrls": [{ "url": "https://docs.python.org/3/tutorial/" }], "maxCrawlPages": 50, "respectRobotsTxtFile": true, "proxyConfiguration": { "useScrapelyProxy": true } } ``` Two details from the input's own documentation: the crawler "does not use any specific user agent identifier", so it follows the `*` group, and "The crawl-delay directive is also not supported yet", so set `maxConcurrency` low yourself on small sites. The option is pre-filled on in the console form. Ready-made Store Actors for specific sites usually read a site's data feeds rather than crawling page by page. Before a commercial project, read the target's robots.txt and terms yourself, as our tutorials do in their "Is it legal" sections. Scrape.ly Store Website Content Crawler Crawl a site into clean text and Markdown with robots.txt respected: set Start URLs, switch on Respect the robots.txt file, and export JSON, CSV or Markdown. [Try it free](https://scrape.ly/store/querystone/website-content-crawler.md) [Browse the Store](https://scrape.ly/store.md) ## A robots.txt checklist for scrapers 1. Fetch `/robots.txt` with the user agent your crawler will use, and keep the text with your run. 2. Find your group, or `*`, and apply the longest-match rule. 3. Use the `Sitemap:` lines to find pages instead of crawling everything. 4. Respect a 5xx as "disallow all" and re-read the file at least daily. 5. Keep request rates gentle, crawl-delay or not; our [web scraping best practices](https://scrape.ly/blog/web-scraping-best-practices.md) cover rate limits. 6. Read the terms of service too. Robots.txt is one signal, not the whole answer. ## FAQ ### Is robots.txt legally binding? Not by itself. It's a technical convention, standardized in RFC 9309, which says its rules "are not a form of access authorization". It does matter legally where terms of service, copyright opt-outs or data protection rules refer to it. ### Does robots.txt apply to scrapers or only to search engines? The protocol applies to any crawler. The `User-agent: *` group covers every bot that isn't named elsewhere in the file. ### What does "Disallow: /" mean? It asks the matching crawlers not to crawl anything on the site. `Disallow:` with an empty value means nothing is disallowed. ### What if a site has no robots.txt? Under RFC 9309, a 4xx response such as 404 means the crawler may access any resources. A 5xx error means it must assume everything is disallowed until the file can be read. ### Does Scrape.ly respect robots.txt? The Website Content Crawler and the Web Scraper do when you turn on **Respect the robots.txt file**. They follow the `*` group and don't support crawl-delay yet. Scrape.ly Store Web Scraper with robots.txt checks Crawl any site with your own page function and robots.txt respected. Over HTTP or in a hosted Chrome, $0.715 per 1,000 records, platform usage included. [Try it free](https://scrape.ly/store/querystone/web-scraper.md) [Browse the Store](https://scrape.ly/store.md) About the author [Scrape.ly Team](https://scrape.ly/blog/author/scrapely-team.md) The team behind Scrape.ly Scrape.ly is the web data platform from TeraShift GmbH in Switzerland, powered by Evomi: ready-made scrapers in the Scrape.ly Store, a cloud runtime, an API with JavaScript and Python SDKs, proxies, a hosted anti-bot browser and an MCP server for AI agents. - [Website](https://scrape.ly/index.md) Keep reading ## More in Guides - [![Pixel-art robot with an orange >\_ on its terminal head holds up a personal-data card whose name is blacked out, beside a ticked checklist on an easel and the blue EU flag with its ring of gold stars, on its way to a crate locked with an amber padlock](https://scrape.ly/media/gdpr-web-scraping.png?w=750)](https://scrape.ly/blog/gdpr-web-scraping.md) ### [Web Scraping and GDPR: What You Can Collect and How](https://scrape.ly/blog/gdpr-web-scraping.md) [How the GDPR applies to web scraping: what counts as personal data, why public data still needs a legal basis, what regulators in the Netherlands and France expect, the safeguards that make a scrape defensible, and a practical checklist, with sources quoted and dated.](https://scrape.ly/blog/gdpr-web-scraping.md) - [![Pixel-art robot with an orange >\_ on its terminal head pulls a data card out of a browser window whose page is still rendering, its top drawn and the rest grey placeholder blocks around an orange loading spinner, and drops it into a crate](https://scrape.ly/media/scrape-dynamic-websites.png?w=750)](https://scrape.ly/blog/scrape-dynamic-websites.md) ### [How to Scrape Dynamic Websites: JavaScript Pages, Hidden APIs and Headless Browsers](https://scrape.ly/blog/scrape-dynamic-websites.md) [How to scrape dynamic, JavaScript-rendered websites: tell whether a page needs a browser, read the data from embedded JSON or the site's own API, and render the rest in a headless browser. Tested Python code and three real runs from 27 October 2026 with their exact cost.](https://scrape.ly/blog/scrape-dynamic-websites.md) - [![Pixel-art robot with an orange >\_ on its terminal head carries a data card towards a crate at the end of a cable from a server, above which a grey cloud holds a browser window showing a web page](https://scrape.ly/media/what-is-a-cloud-browser.png?w=750)](https://scrape.ly/blog/what-is-a-cloud-browser.md) ### [What Is a Cloud Browser? How Hosted Browsers Work and Why Scrapers Use Them](https://scrape.ly/blog/what-is-a-cloud-browser.md) [A cloud browser is a web browser that runs on a provider's servers while you control it remotely. The two kinds, how a hosted browser works over CDP (with tested code), local headless vs cloud browser, use cases, real run costs and how providers compare.](https://scrape.ly/blog/what-is-a-cloud-browser.md) ## The internet is full of data. Start owning yours. Scrape.ly gives you the infrastructure, the tooling, and the team to extract, process, and operationalize web data at any scale. Setup takes minutes. [Get Started Free](https://my.scrape.ly/register) [Chat with Support](mailto:hello@scrape.ly?subject=Question%20about%20Scrape.ly\&body=Hi%2C%20I%20have%20a%20question%20about%20Scrape.ly.) No credit card. No setup calls.