Free robots.txt checker

Paste a site URL (or a full robots.txt URL). We fetch the file, show the rules, and let you test any path to see whether a crawler is allowed orblocked. No signup, no credit card.

How robots.txt rules work

robots.txt tells crawlers which paths are off-limits. The most specific matching rule wins, and Allow beats Disallow at the same specificity. This checker parses the rules the way major search engines do and lets you test any path against them.

Check a site before you crawl it

map returns every URL it discovers on a site — it does not filter byrobots.txt. Those rules state the site's preference for you, not what exists, so check the file here first and drop the disallowed paths from the URLsmap returns. Free tier: 1,500 credits, no card.