Lemwatch snapshots robots.txt on every check and alerts when disallow rules change, sitemaps disappear, or the file returns non-200.
We snapshot /robots.txt on every scheduled check, diff against the last version, and alert when disallow rules change, sitemap declarations disappear, or the file returns anything other than 200 OK.
Disallow: / (catastrophic — blocks all crawlers)Disallow: rules on important pathsSitemap: declarationsSite → SEO → Robots.txt — see current file plus a change log.
Yes — Googlebot, Bingbot, and generic (*) blocks are tracked separately.