---
title: "Robots.txt check — Lemwatch Help"
description: "Fetches robots.txt hourly and alerts on syntax errors, disallowed indexable paths, or blocked search-engine bots."
canonical: "https://lemwatch.com/help/robots-txt-check"
source: "https://lemwatch.com"
---

# Robots.txt check

> Fetches robots.txt hourly and alerts on syntax errors, disallowed indexable paths, or blocked search-engine bots.

## Quick answer

 We fetch /robots.txt , parse every User-agent / Disallow / Sitemap directive, and diff against yesterday. Any Disallow: / raises **severity: critical**.

## What it checks

- 200 response on robots.txt

- No accidental Disallow: /

- Sitemap: directive present and points to a live sitemap

- Non-standard directives flagged

- Diff vs previous fetch

## Where to find it in Lemwatch

 **Site details → SEO → Robots.**

## How to fix

- If Disallow: / snuck in from a staging deploy, remove it and redeploy.

- Add Sitemap: <full-url> so Google finds your sitemap.

- Don't block CSS/JS — Google needs them to render.

- Re-run.

## FAQ

### Is robots.txt enforced?

 By honest crawlers only. It doesn't stop scrapers.

### Can I have multiple sitemap directives?

 Yes — list one per line.

## Related

- Canonical tag check

- Hreflang tag check

- Structured data check


## Related

- [Help centre](https://lemwatch.com/help)
- [Canonical tag check](https://lemwatch.com/help/canonical-tag-check)
- [Hreflang tag check](https://lemwatch.com/help/hreflang-tag-check)
- [Structured data (JSON-LD) check](https://lemwatch.com/help/structured-data-check)
- [Lemwatch pricing](https://lemwatch.com/pricing)
- [How does Lemwatch track robots.txt changes?](https://lemwatch.com/help/robots-txt-monitoring)
- [Response Time Threshold Alerts](https://lemwatch.com/help/response-time-thresholds)