Live deterministic tool
Robots.txt Checker
Fetch robots.txt from the submitted site and report crawl directives and sitemap references.
What this checks
robots.txt status
Disallow rules
Allow rules
Sitemap directives
FAQ
What URL should I enter?
Enter any URL on the same site. The tool checks robots.txt at the origin root.
Does this obey robots directives?
Yes. It only fetches robots.txt and does not crawl blocked paths.