robots.txt is a plain-text file at the root of a website that tells well-behaved web crawlers which URLs they can and cannot access. It is the first file any reputable bot fetches when visiting a domain.
Format follows the Robots Exclusion Protocol: per-user-agent `Allow` and `Disallow` rules, plus a `Sitemap:` directive pointing crawlers at the XML sitemap. It is a request, not enforcement — malicious crawlers ignore it.
Common mistakes: blocking CSS/JS files (which prevents Google from rendering and ranking the page), accidentally disallowing the entire site after a refactor, and confusing `noindex` (a per-page directive) with `Disallow` (a crawl directive).
A misconfigured robots.txt can de-index an entire site overnight. Monitoring its content for unexpected changes — and verifying that the production version matches the intended source — is one of the highest-value SEO checks available.
See it in the product: SEO monitoring.