Robots.txt check
Fetches robots.txt hourly and alerts on syntax errors, disallowed indexable paths, or blocked search-engine bots.
Quick answer
We fetch /robots.txt, parse every User-agent / Disallow / Sitemap directive, and diff against yesterday. Any Disallow: / raises severity: critical.
What it checks
- 200 response on robots.txt
- No accidental
Disallow: /
- Sitemap: directive present and points to a live sitemap
- Non-standard directives flagged
- Diff vs previous fetch
Where to find it in Lemwatch
Site details → SEO → Robots.
How to fix
- If Disallow: / snuck in from a staging deploy, remove it and redeploy.
- Add Sitemap: <full-url> so Google finds your sitemap.
- Don't block CSS/JS — Google needs them to render.
- Re-run.
FAQ
Is robots.txt enforced?
By honest crawlers only. It doesn't stop scrapers.
Can I have multiple sitemap directives?
Yes — list one per line.
Related