robots.txt Generator
Generate a valid robots.txt file for your site, starting from a WordPress, Shopify or other platform template.
Processed locally in your browser. Nothing you enter is uploaded.
- Start from a template for common platforms (WordPress, Shopify) or a blanket block-all/allow-all baseline, instead of building every rule from scratch.
- Build per-user-agent allow/disallow rules without memorising the syntax.
- Adds a correctly formatted `Sitemap:` line.
- Runs in your browser — copy or download the file.
Why this is private
- Processed locally in your browser. Nothing you enter is uploaded.
- Open your browser's DevTools → Network tab while using this tool — you won't see a request carrying your input, because it never leaves your device.
- Nothing you enter is uploaded to a server or shared with us — the processing happens entirely on your device.
Overview
Generates a robots.txt file that tells search engine crawlers which parts of your site to crawl or avoid, with support for multiple user-agent rules and a sitemap reference.
How to use it
- 1Optionally click a template (WordPress, Shopify, block-all, allow-all) to start from a known-good baseline.
- 2Add or edit rules per user-agent (allow/disallow paths).
- 3Optionally add a sitemap URL.
- 4Copy or download the generated robots.txt.
Tips
- robots.txt controls crawling, not indexing — a disallowed page can still appear in search results if it's linked elsewhere. Use a noindex meta tag to prevent indexing.
Frequently asked questions
What people use it for
Set up robots.txt for a WordPress or Shopify siteStart from that platform's template — its typical admin/system paths are pre-filled — then adjust for your specific setup.
Keep crawlers out of admin or staging pathsDisallow `/admin/` or a preview directory from all bots, or use the "Block everything" template for a whole staging site.
Point crawlers at your sitemapAdd the `Sitemap:` reference so discovery doesn't rely on manual submission alone.
Set different rules for a specific botAllow Googlebot broadly while limiting a noisy scraper.
Create a first robots.txt for a new siteStart from a valid baseline instead of a blank file.
Related tools
All seo toolsrobots.txt TesterTest whether a URL is allowed or blocked by a robots.txt file.XML Sitemap GeneratorGenerate a sitemap.xml file from a list of your site's URLs, with lastmod, changefreq and priority.Meta Tags GeneratorGenerate title, description and Open Graph meta tags.Meta Tags AnalyzerSee the actual live meta tags on any public page — title, description, Open Graph, Twitter Card and more.
Ready to use robots.txt Generator?
robots.txt Generator