Build a spec-compliant robots.txt in 30 seconds. Block crawlers, allow specific paths, point search engines to your sitemap, and decide which AI crawlers can train on your site.
robots.txt is the first file most crawlers fetch — get it wrong and Google can de-index your entire site overnight. The right file keeps admin pages and duplicate URLs out of the index, points crawlers at your sitemap, and decides whether AI systems (GPTBot, ClaudeBot, Google-Extended) can train on your content.
A pure client-side generator. We build the file deterministically from your inputs — no edge call, no data leaves your browser. The output follows the original robots.txt convention (Martijn Koster, 1994) plus the de facto extensions every modern crawler recognises: `Allow`, `Sitemap`, and per-user-agent blocks. When you opt to block AI training crawlers, we emit explicit `User-agent: ...` + `Disallow: /` blocks for GPTBot, ChatGPT-User, CCBot, anthropic-ai, ClaudeBot, Google-Extended, PerplexityBot, and Bytespider — the eight bots that publicly identify themselves and respect robots.txt. The file is offered as both copy-to-clipboard and a downloadable `robots.txt` file.
At the root of your domain, served as plain text at `https://yourdomain.com/robots.txt`. Subdirectories and subdomains each need their own file — `blog.example.com/robots.txt` is separate from `example.com/robots.txt`.
No. robots.txt blocks crawling, not indexing. A blocked URL can still appear in search results if other sites link to it — Google just shows the URL without a snippet. To remove a page from the index, use a `noindex` meta tag and let crawlers fetch it (do not block it in robots.txt).
Only if your server is overwhelmed by aggressive crawlers. Googlebot ignores Crawl-delay — set it in Google Search Console instead. Bingbot, Yandex, and some scrapers do respect it.
No, but it is strongly recommended. It lets Google, Bing, and other crawlers discover your sitemap without you submitting it through each search-engine console.
It depends on your content strategy. Blocking them stops your content being used to train new models — at the cost of being cited less often in their answers. Most content sites now allow them; security-sensitive or paywalled sites typically block.
No — only the file at the exact path `/robots.txt` on each host is consulted. Subdomains each need their own; subfolders cannot have their own.