Playbook
    seo
    easy

    Fix robots.txt blocking your sitemap

    Your robots.txt is blocking Googlebot from crawling pages listed in your sitemap. Here is how to spot and fix the conflict.

    10 minutesLast reviewed 2026-06-09

    What is happening

    You have URLs listed in sitemap.xml that robots.txt tells search engines not to crawl. Google Search Console reports these as "Indexed, though blocked by robots.txt" or "Discovered — currently not indexed".

    Fix it

  1. Open `yourdomain.com/robots.txt` and look at every Disallow: line.
  2. Cross-check each disallow path against the URLs in your sitemap. Any overlap is a conflict.
  3. Decide intent. If the page should be indexed, remove the Disallow line. If it should not be indexed, remove the URL from the sitemap and add a <meta name="robots" content="noindex"> to the page itself — robots.txt alone is not enough to deindex.

    4. Example of a clean robots.txt

    User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /search?

Sitemap: https://yourdomain.com/sitemap.xml

    Common gotchas

  4. Wildcard patterns match more than you think. Disallow: /api blocks /api, /api-docs, /api/anything. Use Disallow: /api/ to scope it.
  5. Multiple sitemaps. Each Sitemap: directive must be on its own line; multiple is fine.
  6. Crawler-specific rules. A User-agent: Googlebot block overrides the User-agent: * block for Google only.

    7. What this doesn't do

    This is for the directive-conflict case. For pages that load but return weird content to Googlebot, see user-agent cloaking pitfalls.

    All playbooks