---
title: "Free robots.txt Generator — Spec-Compliant in 30 Seconds"
description: "Free robots.txt generator. Block crawlers, allow paths, point to your sitemap, and choose whether AI bots (GPTBot, ClaudeBot) can crawl. No signup."
canonical: "https://lemwatch.com/tools/robots-txt-generator"
source: "https://lemwatch.com"
---

# robots.txt Generator

> Build a spec-compliant robots.txt in 30 seconds. Block crawlers, allow specific paths, point search engines to your sitemap, and decide which AI crawlers can train on your site.

## What this tool checks

- Includes a default `User-agent: *` block
- Disallow rules for the paths you do not want indexed
- Allow overrides that take precedence over Disallow
- Optional Crawl-delay for misbehaving crawlers
- Sitemap directive so Google and Bing find your URLs faster
- Optional block of every known AI training crawler

## Why it matters

 robots.txt is the first file most crawlers fetch — get it wrong and Google can de-index your entire site overnight. The right file keeps admin pages and duplicate URLs out of the index, points crawlers at your sitemap, and decides whether AI systems (GPTBot, ClaudeBot, Google-Extended) can train on your content.

## How to fix what it finds

- Pick which paths to disallow (admin, cart, checkout, internal search)
- Add the absolute URL to your sitemap.xml so it is discovered immediately
- Decide your stance on AI training crawlers — block or allow
- Download the generated file and upload to `https://yourdomain.com/robots.txt`
- Verify with Google Search Console robots.txt tester before publishing

## Methodology

 A pure client-side generator. We build the file deterministically from your inputs — no edge call, no data leaves your browser. The output follows the original robots.txt convention (Martijn Koster, 1994) plus the de facto extensions every modern crawler recognises: `Allow`, `Sitemap`, and per-user-agent blocks. When you opt to block AI training crawlers, we emit explicit `User-agent: ...` + `Disallow: /` blocks for GPTBot, ChatGPT-User, CCBot, anthropic-ai, ClaudeBot, Google-Extended, PerplexityBot, and Bytespider — the eight bots that publicly identify themselves and respect robots.txt. The file is offered as both copy-to-clipboard and a downloadable `robots.txt` file.

## Frequently asked questions

### Where do I put robots.txt?

 At the root of your domain, served as plain text at `https://yourdomain.com/robots.txt`. Subdirectories and subdomains each need their own file — `blog.example.com/robots.txt` is separate from `example.com/robots.txt`.

### Does blocking a page in robots.txt remove it from Google?

 No. robots.txt blocks crawling, not indexing. A blocked URL can still appear in search results if other sites link to it — Google just shows the URL without a snippet. To remove a page from the index, use a `noindex` meta tag and let crawlers fetch it (do not block it in robots.txt).

### Should I add Crawl-delay?

 Only if your server is overwhelmed by aggressive crawlers. Googlebot ignores Crawl-delay — set it in Google Search Console instead. Bingbot, Yandex, and some scrapers do respect it.

### Is Sitemap: in robots.txt required?

 No, but it is strongly recommended. It lets Google, Bing, and other crawlers discover your sitemap without you submitting it through each search-engine console.

### Should I block GPTBot and ClaudeBot?

 It depends on your content strategy. Blocking them stops your content being used to train new models — at the cost of being cited less often in their answers. Most content sites now allow them; security-sensitive or paywalled sites typically block.

### Can I have multiple robots.txt files?

 No — only the file at the exact path `/robots.txt` on each host is consulted. Subdomains each need their own; subfolders cannot have their own.


## Related

- [All free tools](https://lemwatch.com/tools)
- [llms txt generator](https://lemwatch.com/tools/llms-txt-generator)
- [Sitemap Checker](https://lemwatch.com/tools/sitemap-checker)
- [noindex checker](https://lemwatch.com/tools/indexability-checker)
- [SPF Record Generator](https://lemwatch.com/tools/spf-generator)
- [DMARC Record Generator](https://lemwatch.com/tools/dmarc-generator)
- [Lemwatch pricing](https://lemwatch.com/pricing)
- [All monitoring checks](https://lemwatch.com/features)