Skip to contentLockpick
How it worksPricingDocs

Read

  • BlogPlain answers to common security questions
  • GuidesStep by step fixes for common setups
  • Security checksEvery check we run, explained

Learn

  • Learn with PickTwo-minute lessons on how sites get attacked
  • Lockpick for ClaudeCheck sites from Claude and other assistants
  • GlossaryThe words we use, in plain words

Company

  • ChangelogWhat changed, newest first
  • StatusWhether scans are running right now
  • Security and trustHow we check and what we never do
Sign inCheck it
Checks

Does your sitemap list pages your robots.txt blocks?

By Mark KashefPublished October 8, 2026

This search visibility check flags pages your sitemap asks search engines to visit while your robots.txt tells them to stay away. Search engines report this as an error.

What does this check look at?

It looks at whether your sitemap lists pages that your robots.txt file tells search engines to skip. Google notes that robots.txt is not a mechanism for keeping a web page out of Google. Listed pages are never requested. A Pulse also reads your sitemap, the list of pages you give search engines.

Pulse and Deep scans both run it.

Curious what your site shows from outside? The quick check is free and needs no account.

Check it

What does a result mean?

What the report showsWhat it meansWhat to do
Info findingSome sitemap pages match a Disallow rule.Remove them from the sitemap, or lift the rule if they belong in search.

Listed pages are not requested, and it is about search visibility only. A clean result means nothing was found in what the scan read. It does not prove the problem is absent from parts the scan never sees.

How serious is a finding?

This is information only. It claims no weakness, so it carries no verdict, and you decide whether anything should change. Confidence starts high, because a response from your site shows the thing itself. The report lists what to fix by how directly someone could use it, not by the severity label. Sitemap cards are not security issues and never change your grade.

Fix it with your AI

Decide for each page whether it belongs in search, then make both lists agree.

Fix prompt for any coding agent
My sitemap lists pages that my robots.txt disallows: [paste the problem from the report]. For each one, ask me whether it should appear in search. If not, remove it from the sitemap generator. If it should, remove or narrow the Disallow rule. If a page must stay out of search entirely, use a noindex robots meta tag instead of robots.txt. Show me both files afterwards.

How to check

Say Atlas, our example app at atlas.example, blocks its pricing experiments folder in robots.txt but its sitemap still lists them. The check adds one card.

Run a Pulse scan from your Lockpick account to see it for your site. Pulse or Deep explains what each scan reads, and the full list of checks shows the rest.

Questions people ask

Does blocking a page in robots.txt keep it out of search?

Not reliably. A blocked page can still be listed if other sites link to it. A noindex tag is the way to keep it out.

Is this a security issue?

No. It is about search visibility and never changes your grade.

See what your site shows from outside

A quick check reads your homepage the way a visitor would and gives you a score. No account needed.

Check your site

Related pages

  • Does your robots.txt point search engines to your sitemap?
  • Does your sitemap point attackers to a private area?
  • Does your sitemap list addresses search engines will skip?

Sources

  1. How we rate findings Lockpick docs, read on October 7, 2026
  2. Read your report Lockpick docs, read on October 7, 2026
  3. Robots.txt introduction and guide Google Search Central, read on October 7, 2026

About the author

Mark Kashef, Founder of Lockpick

I'm Mark, and I build Lockpick. I write these pages, and each one is checked against its sources before it goes live. The Updated date changes only when the page itself does.

Lockpick
PricingMethodologyDocsBlogGuidesSecurity checksLearn with PickSecurity and trustLockpick for ClaudeChangelogStatusFacts for AI assistants

Lockpick is a security scanner for vibe-coded apps. It checks what your live site exposes and what a GitHub repo or AI skill would do before you install it, then gives you a fix prompt for Lovable, Bolt, Replit, Cursor or Claude Code.

Lockpick assesses public pages you are authorized to test, and public GitHub code it never runs.

© 2026 LockpickTermsPrivacyContactSign in