robots.txt Tester
Test whether a URL is allowed or blocked by a robots.txt file.
Processed locally in your browser. Nothing you enter is uploaded.
- Applies the longest-match precedence rule Google uses, including wildcards and `$`.
- Test against a specific user-agent or the `*` default.
- Runs in your browser — paste an unpublished robots.txt to check it first.
Why this is private
- Processed locally in your browser. Nothing you enter is uploaded.
- Open your browser's DevTools → Network tab while using this tool — you won't see a request carrying your input, because it never leaves your device.
- Nothing you enter is uploaded to a server or shared with us — the processing happens entirely on your device.
Overview
Tests whether a specific URL path is allowed or blocked by a robots.txt file, applying the same longest-match-wins rule real crawlers use.
How to use it
- 1Paste your robots.txt content.
- 2Enter the URL path to test.
- 3Optionally set a specific user-agent (defaults to *).
Tips
- robots.txt only controls crawling, not indexing — a disallowed page can still appear in search results if linked elsewhere. Use a noindex meta tag to prevent indexing.
Frequently asked questions
What people use it for
Confirm an important page isn't blockedCheck that a key landing page or the sitemap URL is crawlable.
Verify a new disallow rule does what you meantTest both a path it should block and one it shouldn't.
Debug a "blocked by robots.txt" warningReproduce the crawler's decision for the exact URL Search Console flagged.
Check wildcard rulesSee how `Disallow: /*.pdf$` or `/*?` actually matches your URLs.
Related tools
All seo toolsrobots.txt GeneratorGenerate a valid robots.txt file for your site, starting from a WordPress, Shopify or other platform template.XML Sitemap GeneratorGenerate a sitemap.xml file from a list of your site's URLs, with lastmod, changefreq and priority.Meta Tags GeneratorGenerate title, description and Open Graph meta tags.Meta Tags AnalyzerSee the actual live meta tags on any public page — title, description, Open Graph, Twitter Card and more.
Ready to use robots.txt Tester?
robots.txt Tester