--- title: "Free robots.txt Tester — Spec-Compliant Validator" description: "Free robots.txt tester. Fetches and validates any site's robots.txt — flags missing sitemaps, accidental site-wide blocks, oversized files, syntax errors. No…" canonical: "https://lemwatch.com/tools/robots-txt-tester" source: "https://lemwatch.com" --- # robots.txt tester > Fetch any site's robots.txt and audit it against the spec. Catches Disallow: /, missing sitemaps, broken syntax, oversized files, and unknown directives in one pass. ## What this tool checks - File reachable at /robots.txt with HTTP 200 - Correct Content-Type (text/plain) - Total file size under the 500 KB Google limit - At least one User-agent block, including a wildcard fallback - No accidental "Disallow: /" blocking the whole site - Sitemap directive present and resolves to a valid absolute URL - Crawl-delay (if set) is within sane bounds - Unknown / non-standard directives flagged ## Why it matters robots.txt is the first file Google fetches when it visits your site. A single stray "Disallow: /" can wipe your entire site off Google in under 24 hours. A missing Sitemap directive slows discovery of new pages. An oversized file (over 500 KB) is silently truncated. This tool catches every one of those mistakes before they cost you traffic. ## How to fix what it finds - If the file 404s — create one at the root of your domain, even if empty - If "Disallow: /" appears — remove it unless you intentionally want the site de-indexed - If no Sitemap directive — add the absolute URL of your sitemap.xml - If size is over 500 KB — split into per-subdomain files; Google ignores everything past the cap - If unknown directives appear — most crawlers ignore them; remove for cleanliness - Re-test after each change; the result updates instantly ## Methodology We issue a single HTTPS GET against `/robots.txt` on the apex of the domain you enter, following redirects up to a 10 KB body cap and an 8-second timeout. The response is parsed line-by-line using the original Martijn Koster (1994) convention plus Google's extensions: `User-agent`, `Disallow`, `Allow`, `Sitemap`, `Crawl-delay`, and inline `#` comments. We do not execute regex or wildcard matching against your URLs — that is the job of a crawler simulator. What we check is whether the file itself is valid and whether the directives present make sense. Findings are scored: fail = -30, warn = -10, capped at 0–100. ## This tool also answers ### How do I block a page from Google? Use a noindex meta tag or X-Robots-Tag header, not robots.txt. A robots.txt disallow stops the crawl but the URL can still be indexed from external links — and because the crawler never fetches the page, it never sees your noindex. ### How do I check if robots.txt is blocking a URL? Enter the URL above. The tester matches it against every user-agent group in the live robots.txt and reports which rule wins, so you can see whether a broad Disallow is quietly taking out pages you want crawled. ## Frequently asked questions ### Why does my site rank for "site:example.com" but not real queries? A common cause is "Disallow: /" in robots.txt blocking Googlebot from crawling content. Google can still show the URL in site: results from external links, but cannot index the content. Our tester catches this specific pattern. ### Does robots.txt actually remove pages from Google? No. robots.txt blocks crawling, not indexing. A URL blocked by robots.txt can still appear in search results if other sites link to it — Google just shows the URL without a snippet. To remove a page, use noindex meta and ALLOW crawling. ### What is the maximum robots.txt file size? Google enforces a 500 KB ceiling — anything past that is silently ignored. Most other major crawlers respect a similar limit. If yours is near 500 KB, you are probably overengineering it; consider splitting by subdomain or simplifying rules. ### Do I need a robots.txt file at all? No. If you have no file, every crawler assumes everything is allowed — usually fine. You only need one if you want to block specific paths, point to a sitemap, or restrict specific bots (AI crawlers, archive crawlers). ### Why does Crawl-delay not work on Google? Googlebot ignores Crawl-delay entirely. To slow it down, use the crawl-rate setting in Google Search Console instead. Bingbot, Yandex, and most scrapers do respect Crawl-delay. ### Can I have a different robots.txt for staging? Each subdomain (host) reads its own robots.txt — so staging.example.com/robots.txt can ship "Disallow: /" while example.com/robots.txt stays open. Use this to prevent staging being indexed. ### How do I block a page from Google? Use a noindex meta tag or X-Robots-Tag header, not robots.txt. A robots.txt disallow stops the crawl but the URL can still be indexed from external links — and because the crawler never fetches the page, it never sees your noindex. ### How do I check if robots.txt is blocking a URL? Enter the URL above. The tester matches it against every user-agent group in the live robots.txt and reports which rule wins, so you can see whether a broad Disallow is quietly taking out pages you want crawled. ## Related - [All free tools](https://lemwatch.com/tools) - [robots.txt Generator](https://lemwatch.com/tools/robots-txt-generator) - [Sitemap Checker](https://lemwatch.com/tools/sitemap-checker) - [noindex checker](https://lemwatch.com/tools/indexability-checker) - [hreflang checker](https://lemwatch.com/tools/hreflang-checker) - [canonical tag checker](https://lemwatch.com/tools/canonical-checker) - [Lemwatch pricing](https://lemwatch.com/pricing) - [All monitoring checks](https://lemwatch.com/features)