Robots.txt check

Fetches robots.txt hourly and alerts on syntax errors, disallowed indexable paths, or blocked search-engine bots.

Quick answer

We fetch /robots.txt, parse every User-agent / Disallow / Sitemap directive, and diff against yesterday. Any Disallow: / raises severity: critical.

What it checks

Where to find it in Lemwatch

Site details → SEO → Robots.

How to fix

  1. If Disallow: / snuck in from a staging deploy, remove it and redeploy.
  2. Add Sitemap: <full-url> so Google finds your sitemap.
  3. Don't block CSS/JS — Google needs them to render.
  4. Re-run.

FAQ

Is robots.txt enforced?

By honest crawlers only. It doesn't stop scrapers.

Can I have multiple sitemap directives?

Yes — list one per line.

Related