---
title: "robots.txt — monitoring glossary"
description: "robots.txt is a plain-text file at the root of a website that tells well-behaved web crawlers which URLs they can and cannot access. It is the first file any…"
canonical: "https://lemwatch.com/glossary/robots-txt"
source: "https://lemwatch.com"
---

# robots.txt

> robots.txt is a plain-text file at the root of a website that tells well-behaved web crawlers which URLs they can and cannot access. It is the first file any reputable bot fetches when visiting a domain.

## Definition

 Format follows the Robots Exclusion Protocol: per-user-agent `Allow` and `Disallow` rules, plus a `Sitemap:` directive pointing crawlers at the XML sitemap. It is a request, not enforcement — malicious crawlers ignore it.

 Common mistakes: blocking CSS/JS files (which prevents Google from rendering and ranking the page), accidentally disallowing the entire site after a refactor, and confusing `noindex` (a per-page directive) with `Disallow` (a crawl directive).

## Why it matters

 A misconfigured robots.txt can de-index an entire site overnight. Monitoring its content for unexpected changes — and verifying that the production version matches the intended source — is one of the highest-value SEO checks available.

 See it in the product: [SEO monitoring](https://lemwatch.com/features).


## Related

- [Monitoring glossary](https://lemwatch.com/glossary)
- [SEO](https://lemwatch.com/glossary/seo)
- [Sitemap](https://lemwatch.com/glossary/sitemap)
- [Indexing](https://lemwatch.com/glossary/indexing)
- [Canonical Tag](https://lemwatch.com/glossary/canonical)
- [All monitoring checks](https://lemwatch.com/features)
- [Recovery Point Objective (RPO)](https://lemwatch.com/glossary/rpo)
- [RDAP (Registration Data Access Protocol)](https://lemwatch.com/glossary/rdap)