SEO
robots.txt
robots.txt is a plain-text file at the root of a website that tells well-behaved web crawlers which URLs they can and cannot access. It is the first file any reputable bot fetches when visiting a domain.
In depth
Format follows the Robots Exclusion Protocol: per-user-agent `Allow` and `Disallow` rules, plus a `Sitemap:` directive pointing crawlers at the XML sitemap. It is a request, not enforcement — malicious crawlers ignore it.
Common mistakes: blocking CSS/JS files (which prevents Google from rendering and ranking the page), accidentally disallowing the entire site after a refactor, and confusing `noindex` (a per-page directive) with `Disallow` (a crawl directive).
Why it matters
A misconfigured robots.txt can de-index an entire site overnight. Monitoring its content for unexpected changes — and verifying that the production version matches the intended source — is one of the highest-value SEO checks available.
How Lemwatch helps with robots.txt
Lemwatch monitors robots.txt continuously and alerts you the moment something changes.
SEO monitoring →Related terms
SEO
Search Engine Optimization (SEO) is the practice of structuring a website's content, code, and reputation so t…
Sitemap
A sitemap is an XML file that lists every URL on a website intended for search engines to index, along with me…
Indexing
Indexing is the process by which a search engine adds a page to its searchable database after crawling it. A p…
Canonical Tag
A canonical tag is an HTML link element (`<link rel="canonical" href="...">`) that tells search engines which …
IndexNow
IndexNow is an open protocol jointly developed by Microsoft and Yandex that lets websites instantly notify sea…
Browse the full Lemwatch monitoring glossary (47 terms).