robots.txt Generator
Build a spec-compliant robots.txt in 30 seconds. Block crawlers, allow specific paths, point search engines to your sitemap, and decide which AI crawlers can train on your site.
By Deepak Shukla — Founder & CEO, Pearl Lemon
Generate your robots.txt
# Default rules — apply to every crawler unless overridden below. User-agent: * Disallow: /admin Disallow: /cart Disallow: /checkout
Upload to https://yourdomain.com/robots.txt at the root of your site.
What This Tool Checks
- Includes a default `User-agent: *` block
- Disallow rules for the paths you do not want indexed
- Allow overrides that take precedence over Disallow
- Optional Crawl-delay for misbehaving crawlers
- Sitemap directive so Google and Bing find your URLs faster
- Optional block of every known AI training crawler
Why This Matters
robots.txt is the first file most crawlers fetch — get it wrong and Google can de-index your entire site overnight. The right file keeps admin pages and duplicate URLs out of the index, points crawlers at your sitemap, and decides whether AI systems (GPTBot, ClaudeBot, Google-Extended) can train on your content.
How to Fix Issues
- 1.Pick which paths to disallow (admin, cart, checkout, internal search)
- 2.Add the absolute URL to your sitemap.xml so it is discovered immediately
- 3.Decide your stance on AI training crawlers — block or allow
- 4.Download the generated file and upload to `https://yourdomain.com/robots.txt`
- 5.Verify with Google Search Console robots.txt tester before publishing
robots.txt is a plain text file at the root of your domain that tells crawlers where they may go. It is the first thing most of them fetch, it takes thirty seconds to write, and a single wrong line in it can remove a site from search results. This generator builds a valid file from choices rather than from memory.
What the file can and cannot do
It controls crawling, not indexing. A disallowed URL can still appear in search results if other sites link to it — the crawler simply cannot fetch the page to describe it, so it shows as a bare URL with no snippet. To keep a page out of the index you must allow crawling and serve a noindex directive, which is the exact opposite of what most people expect. It also has no enforcement: well-behaved crawlers respect it, scrapers ignore it entirely, and nothing in the file protects private content. Anything that must stay private needs authentication, not a Disallow line.
What belongs in it
A wildcard user-agent block that applies to everyone. Disallow rules for administrative paths, internal search result pages, faceted URL parameters and anything else that generates near-infinite low-value URLs. A Sitemap directive with the absolute URL of your sitemap, which speeds up discovery of new pages. Optionally, blocks for AI training crawlers if you would rather your content was not used that way — a decision worth making deliberately rather than by default in either direction.
The mistakes this generator avoids
Blocking CSS and JavaScript directories, which stops Google rendering your pages and makes them look broken to the crawler. Leaving a staging Disallow: / in place after launch, which is the single most expensive typo in SEO. Using robots.txt to hide a page that also carries a noindex tag, which prevents the crawler from ever seeing the noindex. Relative sitemap URLs, which are ignored. Trailing-slash mistakes that block far more than intended.
After you publish it
Upload the file to the root of the domain — it must be at /robots.txt on the exact host, and each subdomain needs its own. Then test what you published rather than what you meant to publish, and re-test after any platform migration, because hosting platforms frequently generate their own file and overwrite yours.
Methodology
A pure client-side generator. We build the file deterministically from your inputs — no edge call, no data leaves your browser. The output follows the original robots.txt convention (Martijn Koster, 1994) plus the de facto extensions every modern crawler recognises: Allow, Sitemap, and per-user-agent blocks. When you opt to block AI training crawlers, we emit explicit User-agent: ... + Disallow: / blocks for GPTBot, ChatGPT-User, CCBot, anthropic-ai, ClaudeBot, Google-Extended, PerplexityBot, and Bytespider — the eight bots that publicly identify themselves and respect robots.txt. The file is offered as both copy-to-clipboard and a downloadable robots.txt file.
Frequently Asked Questions
Related Free Tools
llms.txt Generator
Generate a concise llms.txt-style Markdown index that maps a site to its canonical pages. Review it before publishing.
Sitemap Checker
Inspect your XML sitemap. Find it, check its basic sitemap structure, count URLs, detect sitemap indexes, and flag suspicious or broken sub-sitemaps.
Indexability
Check whether Google can actually index your page. Detects noindex, canonical conflicts, X-Robots-Tag headers, soft 404s, and crawl blockers.
HTTP Status Code Reference
Understanding HTTP status codes helps you diagnose issues found by this tool: