Fetch any site's robots.txt and audit it against the spec. Catches Disallow: /, missing sitemaps, broken syntax, oversized files, and unknown directives in one pass.
robots.txt is the first file Google fetches when it visits your site. A single stray "Disallow: /" can wipe your entire site off Google in under 24 hours. A missing Sitemap directive slows discovery of new pages. An oversized file (over 500 KB) is silently truncated. This tool catches every one of those mistakes before they cost you traffic.
We issue a single HTTPS GET against `/robots.txt` on the apex of the domain you enter, following redirects up to a 10 KB body cap and an 8-second timeout. The response is parsed line-by-line using the original Martijn Koster (1994) convention plus Google's extensions: `User-agent`, `Disallow`, `Allow`, `Sitemap`, `Crawl-delay`, and inline `#` comments. We do not execute regex or wildcard matching against your URLs — that is the job of a crawler simulator. What we check is whether the file itself is valid and whether the directives present make sense. Findings are scored: fail = -30, warn = -10, capped at 0–100.
Use a noindex meta tag or X-Robots-Tag header, not robots.txt. A robots.txt disallow stops the crawl but the URL can still be indexed from external links — and because the crawler never fetches the page, it never sees your noindex.
Enter the URL above. The tester matches it against every user-agent group in the live robots.txt and reports which rule wins, so you can see whether a broad Disallow is quietly taking out pages you want crawled.
A common cause is "Disallow: /" in robots.txt blocking Googlebot from crawling content. Google can still show the URL in site: results from external links, but cannot index the content. Our tester catches this specific pattern.
No. robots.txt blocks crawling, not indexing. A URL blocked by robots.txt can still appear in search results if other sites link to it — Google just shows the URL without a snippet. To remove a page, use noindex meta and ALLOW crawling.
Google enforces a 500 KB ceiling — anything past that is silently ignored. Most other major crawlers respect a similar limit. If yours is near 500 KB, you are probably overengineering it; consider splitting by subdomain or simplifying rules.
No. If you have no file, every crawler assumes everything is allowed — usually fine. You only need one if you want to block specific paths, point to a sitemap, or restrict specific bots (AI crawlers, archive crawlers).
Googlebot ignores Crawl-delay entirely. To slow it down, use the crawl-rate setting in Google Search Console instead. Bingbot, Yandex, and most scrapers do respect Crawl-delay.
Each subdomain (host) reads its own robots.txt — so staging.example.com/robots.txt can ship "Disallow: /" while example.com/robots.txt stays open. Use this to prevent staging being indexed.
Use a noindex meta tag or X-Robots-Tag header, not robots.txt. A robots.txt disallow stops the crawl but the URL can still be indexed from external links — and because the crawler never fetches the page, it never sees your noindex.
Enter the URL above. The tester matches it against every user-agent group in the live robots.txt and reports which rule wins, so you can see whether a broad Disallow is quietly taking out pages you want crawled.