Security & SEO

    Robots.txt check

    Fetches robots.txt hourly and alerts on syntax errors, disallowed indexable paths, or blocked search-engine bots.

    Updated 2026-07-10
    1 min read
    robots
    seo
    crawl

    Quick answer#

    We fetch /robots.txt, parse every User-agent / Disallow / Sitemap directive, and diff against yesterday. Any Disallow: / raises severity: critical.

    What it checks#

  1. 200 response on robots.txt
  2. No accidental Disallow: /
  3. Sitemap: directive present and points to a live sitemap
  4. Non-standard directives flagged
  5. Diff vs previous fetch

    6. Where to find it in Lemwatch#

    Site details → SEO → Robots.

    How to fix#

  6. If Disallow: / snuck in from a staging deploy, remove it and redeploy.
  7. Add Sitemap: <full-url> so Google finds your sitemap.
  8. Don't block CSS/JS — Google needs them to render.
  9. Re-run.

    10. FAQ#

    Is robots.txt enforced?#

    By honest crawlers only. It doesn't stop scrapers.

    Can I have multiple sitemap directives?#

    Yes — list one per line.

    Related#

  10. Canonical tag check
  11. Hreflang tag check
  12. Structured data check
    13. Was this article helpful?

    Previous

    Canonical tag check

    Next

    Hreflang tag check

    Related Articles

    Canonical tag check

    Crawls your top pages and flags missing, self-referential loops, cross-domain, or conflicting canonical tags — the top-3 SEO indexation bug.

    Hreflang tag check

    Validates hreflang tags across your language variants — return-tag mismatches are the #1 cause of Google demoting international sites.

    Structured data (JSON-LD) check

    Validates schema.org JSON-LD on every crawled page against Google's rich-results eligibility rules.

    X-Frame-Options / frame-ancestors check

    Confirms clickjacking protection is enabled via X-Frame-Options or the modern CSP frame-ancestors directive.

    Back to Help Center