robots.txt Generator
Generate a valid robots.txt file for your site.
Processed locally in your browser. Nothing you enter is uploaded.
Overview
Generates a robots.txt file that tells search engine crawlers which parts of your site to crawl or avoid, with support for multiple user-agent rules and a sitemap reference.
How to use it
- 1Add rules per user-agent (allow/disallow paths).
- 2Optionally add a sitemap URL.
- 3Copy or download the generated robots.txt.
Tips
- robots.txt controls crawling, not indexing — a disallowed page can still appear in search results if it's linked elsewhere. Use a noindex meta tag to prevent indexing.
Related tools
All seo toolsMeta Tags GeneratorGenerate title, description and Open Graph meta tags.SERP Snippet PreviewPreview how a page will look in Google search results.Open Graph PreviewPreview how a link will look when shared on social media.Schema Markup GeneratorGenerate JSON-LD structured data for Article, Product, FAQ and more.