robots.txt tester
Fetch any site's robots.txt and audit it against the spec. Catches Disallow: /, missing sitemaps, broken syntax, oversized files, and unknown directives in one pass.
By Deepak Shukla — Founder & CEO, Pearl Lemon
Part of Sitemap monitoring — the full answer, the facts and the alternatives in one place.
What This Tool Checks
- File reachable at /robots.txt with HTTP 200
- Correct Content-Type (text/plain)
- Total file size under the 500 KB Google limit
- At least one User-agent block, including a wildcard fallback
- No accidental "Disallow: /" blocking the whole site
- Sitemap directive present and resolves to a valid absolute URL
- Crawl-delay (if set) is within sane bounds
- Unknown / non-standard directives flagged
Why This Matters
robots.txt is the first file Google fetches when it visits your site. A single stray "Disallow: /" can wipe your entire site off Google in under 24 hours. A missing Sitemap directive slows discovery of new pages. An oversized file (over 500 KB) is silently truncated. This tool catches every one of those mistakes before they cost you traffic.
How to Fix Issues
- 1.If the file 404s — create one at the root of your domain, even if empty
- 2.If "Disallow: /" appears — remove it unless you intentionally want the site de-indexed
- 3.If no Sitemap directive — add the absolute URL of your sitemap.xml
- 4.If size is over 500 KB — split into per-subdomain files; Google ignores everything past the cap
- 5.If unknown directives appear — most crawlers ignore them; remove for cleanliness
- 6.Re-test after each change; the result updates instantly
robots.txt is the first file a crawler fetches and the cheapest way to remove a website from search results by accident. This test fetches the live file from the domain you enter and checks it against what crawlers actually do with it, rather than what the file appears to say.
The single line that does the damage
Disallow: / under a wildcard user-agent blocks the entire site. It belongs on staging, and it reaches production more often than anyone would like, usually because staging was copied forward at launch. The site keeps working for visitors, so nothing looks wrong; rankings simply decay over the following days as pages drop out. Any site that has recently launched, migrated or changed hosting should have this checked before anything else.
Blocking is not removing
A URL blocked in robots.txt can still appear in search results if other pages link to it, because the block prevents fetching rather than listing — the result shows as a bare URL with no description. Worse, blocking a page that carries a noindex tag prevents the crawler from ever reading the noindex, so the page stays listed indefinitely. To remove a page, allow crawling and serve noindex. To keep something private, use authentication; robots.txt is a public file and a list of the paths you would rather people did not visit.
The quieter mistakes
Blocking directories that hold CSS and JavaScript, which stops pages rendering properly for the crawler and can make a good page look broken. A missing Sitemap directive, which slows discovery of new URLs. A relative sitemap URL, which is ignored — it must be absolute. A file that returns HTML instead of plain text, usually because a 404 page is being served, which some crawlers treat as no restrictions and others treat unpredictably. A file over the size limit, where everything past the cap is silently discarded.
Per host, not per site
Every subdomain reads its own file, and so does every protocol and port. A correct file on the apex domain says nothing about the staging subdomain, the help subdomain or the shop. Check each host you publish, and re-check after any platform migration, because hosting platforms frequently generate their own robots.txt and overwrite whatever you uploaded.
Methodology
We issue a single HTTPS GET against /robots.txt on the apex of the domain you enter, following redirects up to a 10 KB body cap and an 8-second timeout. The response is parsed line-by-line using the original Martijn Koster (1994) convention plus Google's extensions: User-agent, Disallow, Allow, Sitemap, Crawl-delay, and inline # comments. We do not execute regex or wildcard matching against your URLs — that is the job of a crawler simulator. What we check is whether the file itself is valid and whether the directives present make sense. Findings are scored: fail = -30, warn = -10, capped at 0–100.
Frequently Asked Questions
This tool also answers
How do I block a page from Google?
Use a noindex meta tag or X-Robots-Tag header, not robots.txt. A robots.txt disallow stops the crawl but the URL can still be indexed from external links — and because the crawler never fetches the page, it never sees your noindex.
How do I check if robots.txt is blocking a URL?
Enter the URL above. The tester matches it against every user-agent group in the live robots.txt and reports which rule wins, so you can see whether a broad Disallow is quietly taking out pages you want crawled.
Related Free Tools
robots.txt Generator
Build a spec-compliant robots.txt in 30 seconds. Block crawlers, allow specific paths, point search engines to your sitemap, and decide which AI crawlers can train on your site.
Sitemap Checker
Inspect your XML sitemap. Find it, check its basic sitemap structure, count URLs, detect sitemap indexes, and flag suspicious or broken sub-sitemaps.
Indexability
Check whether Google can actually index your page. Detects noindex, canonical conflicts, X-Robots-Tag headers, soft 404s, and crawl blockers.
HTTP Status Code Reference
Understanding HTTP status codes helps you diagnose issues found by this tool: