Website Tools

Robots.txt Checker

Fetch a website's robots.txt and read the crawling rules it gives to search engines.

Not sure what to enter? Try an example: wikipedia.org

Free & instant No signup How to use FAQ

Found

wikipedia.org

File Exists

Yes

Accessible

Yes

Basic Validation

Looks Valid

Crawl Rules

4 Allow, 460 Disallow

Sitemap Reference

https://en.wikipedia.org/w/rest.php/site/v1/sitemap/0

# robots.txt for http://www.wikipedia.org/ and friends
#
# Please note: There are a lot of pages on this site, and there are
# some misbehaved spiders out there that go _way_ too fast. If you're
# irresponsible, your access to the site may be blocked.
#

# Observed spamming large amounts of https://en.wikipedia.org/?curid=NNNNNN
# and ignoring 429 ratelimit responses, claims to respect robots:
# http://mj12bot.com/
User-agent: MJ12bot
Disallow: /

# advertising-related bots:
User-agent: Mediapa...

Last checked: 0 seconds ago Live

Download JSON Download PDF

100/100

Result Summary

Passed

robots.txt found

Enter a website to load its robots.txt file and see the allow and disallow rules it sets for search engine crawlers.

Recent Searches

Robots.txt Checker - wikipedia.org 0 seconds ago

What is a robots.txt file?

robots.txt is a plain-text file at the root of a website (example.com/robots.txt) that tells search-engine crawlers which paths they may or may not crawl, and can point them to the XML sitemap. It is a request, not access control - private pages still need proper protection.

How to check a website's robots.txt

Enter the domain in the box above.
Click Check robots.txt.
Read the rules and make sure important sections are not blocked by a Disallow line.
Add a Sitemap: line pointing to your XML sitemap if it is missing.

Related checks

For the SEO view of robots.txt, use the SEO Robots.txt Checker.
To check the sitemap it references, use the Sitemap Checker.

One wrong "Disallow: /" left over from a staging site can remove a whole website from Google. JustChecker checks your robots.txt on a schedule and tells you if it suddenly starts blocking search engines.

Frequently Asked Questions

No. It only stops crawling; a blocked page can still be indexed if other sites link to it. To keep a page out of search results, use a noindex tag and let Google crawl it.

"Disallow: /" left over from a staging site, which blocks the whole website. Blocking CSS and JavaScript folders is another one - Google needs them to render your pages.

Yes. A "Sitemap: https://example.com/sitemap.xml" line helps every search engine find it, not only the ones where you submitted it manually.

Share this tool:

Need a website for your business? Only $599