robots.txt Tester
Test whether a URL is allowed or blocked by a robots.txt file.
Processed locally in your browser. Nothing you enter is uploaded.
Overview
Tests whether a specific URL path is allowed or blocked by a robots.txt file, applying the same longest-match-wins rule real crawlers use.
How to use it
- 1Paste your robots.txt content.
- 2Enter the URL path to test.
- 3Optionally set a specific user-agent (defaults to *).
How it works
Parses User-agent groups and their Allow/Disallow rules, then finds the longest matching pattern for the given path — the same precedence rule Google's crawler documentation describes.
Tips
- robots.txt only controls crawling, not indexing — a disallowed page can still appear in search results if linked elsewhere. Use a noindex meta tag to prevent indexing.