User-agent
Selects the crawler group that receives the rules.
Robots.txt Checker
Robots.txt Checker
Check crawl rules, blocked paths and sitemap declarations for any website.
Example: The checker requests /robots.txt at the site root. Results stay on this page.Googlebot is used by default to test whether this path is allowed.
Robots.txt check result
No check started
Enter a website and path, then click Run check.
ROBOTS.TXT SEO
What is a robots.txt check?
robots.txt is a plain-text file at a website's root that tells search crawlers which paths they may crawl and which they should avoid. It is a crawl hint, not an access-control mechanism, and does not stop deliberate requests.
This checker requests /robots.txt, parses User-agent, Allow, Disallow, Crawl-delay and Sitemap directives, then uses Googlebot to decide whether the path you entered is blocked.
Allow / Disallow
Controls which URL paths a compliant crawler may fetch.
Sitemap
Points crawlers to an XML sitemap for URL discovery.
Why should site owners check robots.txt?
A typo, an overly broad Disallow rule or a stale sitemap URL can keep search engines from discovering important pages. Recheck robots.txt after redesigns, migrations and CMS changes.
Avoid blocking key pages
Confirm that home, product and article paths are not accidentally covered by Disallow rules.
Find rule conflicts
Compare the longest matching Allow and Disallow patterns to explain an unexpected decision.
Verify sitemap discovery
Check that Sitemap declarations are complete absolute HTTP or HTTPS URLs.
How does the checker work?
The check reads public robots.txt data only. It never edits the file or writes a historical database record.
- 01
Normalize the website
Extract the host and protocol, then request /robots.txt at the root.
- 02
Fetch safely
Resolve the target IP, reject private networks, and limit redirects, time and response size.
- 03
Parse crawl rules
Group User-agent directives and read Allow, Disallow, Crawl-delay and Sitemap values.
- 04
Test the path
Apply longest-match rules for Googlebot and show diagnostics and recommendations.
How to read the result
File availability, rule parsing and path decisions describe different layers. A 404 means robots.txt was not found, not that the website is unreachable; a request error does not prove a rule is wrong.
Found
The file returned a successful HTTP response and was parsed.
Not found
The root file was not found; this does not mean the website is down.
Request error
A timeout or server error needs a retry before judging the rules.
Robots.txt checker FAQ
Where should robots.txt be located?
It must be at the root of the website host, such as https://example.com/robots.txt. A file under /blog/robots.txt does not control the whole host.
Will Disallow: / remove my site from search results?
It prevents compliant crawlers from fetching pages, but search engines may still learn URLs from external links. Indexing control also requires page-level signals such as noindex, which crawlers need to access.
What happens when Allow and Disallow both match?
Major crawlers generally use the longest matching path; when lengths tie, Allow wins. This checker applies that common rule for Googlebot.
Can robots.txt replace authentication or a firewall?
No. robots.txt is public crawl guidance and cannot protect sensitive data. Use authentication, authorization and server-side firewall controls for access restriction.
Does a sitemap have to be listed in robots.txt?
No. You can also submit it through Search Console and other webmaster platforms. A declaration helps automatic discovery, but it must be a complete absolute URL.
Related webmaster tools
Continue checking page responses, sitemaps and core website SEO signals.