Website Tools

Robots.txt Checker

Fetch a website's robots.txt and read the crawling rules it gives to search engines.

Not sure what to enter? Try an example: wikipedia.org

Free & instant No signup How to use FAQ

Enter a website to load its robots.txt file and see the allow and disallow rules it sets for search engine crawlers.

What is a robots.txt file?

robots.txt is a plain-text file at the root of a website (example.com/robots.txt) that tells search-engine crawlers which paths they may or may not crawl, and can point them to the XML sitemap. It is a request, not access control - private pages still need proper protection.

How to check a website's robots.txt

Enter the domain in the box above.
Click Check robots.txt.
Read the rules and make sure important sections are not blocked by a Disallow line.
Add a Sitemap: line pointing to your XML sitemap if it is missing.

Related checks

For the SEO view of robots.txt, use the SEO Robots.txt Checker.
To check the sitemap it references, use the Sitemap Checker.

One wrong "Disallow: /" left over from a staging site can remove a whole website from Google. JustChecker checks your robots.txt on a schedule and tells you if it suddenly starts blocking search engines.

Frequently Asked Questions

No. It only stops crawling; a blocked page can still be indexed if other sites link to it. To keep a page out of search results, use a noindex tag and let Google crawl it.

"Disallow: /" left over from a staging site, which blocks the whole website. Blocking CSS and JavaScript folders is another one - Google needs them to render your pages.

Yes. A "Sitemap: https://example.com/sitemap.xml" line helps every search engine find it, not only the ones where you submitted it manually.

Share this tool:

Need a website for your business? Only $599