About the Robots.txt Tester
Paste a robots.txt or fetch it from any website, then test a list of URLs for a chosen crawler. Each URL shows whether it's allowed or blocked and which line decided it, using the same matching rules as Google.
Key features
- Fetch any site's live robots.txt or paste your own
- Test hundreds of URLs at once
- Choose Googlebot, Bingbot, AI crawlers or any bot
- Shows the exact line that allowed or blocked each URL
- Google's matching rules: longest match, wildcards and $
- Flags syntax problems and lists declared sitemaps
Check exactly what your robots.txt blocks
One misplaced slash in robots.txt can block important pages, or fail to block the ones you meant to. Paste your file (or fetch it from a live site), enter some URLs and pick a crawler to see allowed or blocked, with the rule responsible.
Matching rules (RFC 9309)
| Rule | Example | Matches |
|---|---|---|
| Prefix match | Disallow: /admin | /admin, /admin/, /administrator |
| Folder only | Disallow: /admin/ | /admin/ and below, not /administrator |
| Wildcard | Disallow: /*?sort= | Any URL with ?sort= |
| End anchor | Disallow: /*.pdf$ | URLs ending in .pdf, not .pdf?x=1 |
| Empty | Disallow: | Nothing (allows everything) |
Example
User-agent: *
Disallow: /admin/
Allow: /admin/help/
| URL | Result | Why |
|---|---|---|
| /admin/settings | Blocked | Disallow: /admin/ |
| /admin/help/faq | Allowed | Longer Allow: /admin/help/ wins |
| /about | Allowed | No rule matches |
Testing tips
- Test your most important pages first: home, category and product pages.
- Test with Googlebot and
*, as other search engines may fall back to*. - Re-test after every change. A new wildcard can block more than you expect.
- Use Google Search Console's robots.txt report to see what Google last fetched.
Related tools
Create a robots.txt with the Robots.txt Generator, or check a page's meta robots tag with the Meta Tag Checker.
How to use the Robots.txt Tester
- Enter the address of the page or website you want to check.
- Click the button. Our server fetches the public page (it can't reach private or local addresses).
- Review the results, starting with anything marked in red or amber.
- Fix the issues on your site, then run the check again to confirm.