How does Lemwatch track robots.txt changes?

Lemwatch snapshots robots.txt on every check and alerts when disallow rules change, sitemaps disappear, or the file returns non-200.

Quick answer

We snapshot /robots.txt on every scheduled check, diff against the last version, and alert when disallow rules change, sitemap declarations disappear, or the file returns anything other than 200 OK.

What we flag

Where to find it in Lemwatch

Site → SEO → Robots.txt — see current file plus a change log.

FAQ

Do you check per user-agent?

Yes — Googlebot, Bingbot, and generic (*) blocks are tracked separately.

Related