Robots.txt Generator
Build a robots.txt file with user-agent rules, allows, and sitemaps.
About this tool
Build a robots.txt file with user-agent rules, allows, and sitemaps. Scaffold placeholder content, cron schedules, test data, and project boilerplate without extra CLI installs.
Set the User-agent (use * for all bots, or a specific crawler name). List Disallow and Allow paths one per line. Example — Block admin, allow public, declare sitemap: try `User-agent: * Disallow: /admin/ Allow: /public/ Sitemap: https://example.com/sitemap.xml` and expect something like `User-agent: * Disallow: /admin/ Allow: /public/ Sitemap: https://example.com/sitemap.xml`.
Generates a single User-agent group — multi-bot files need manual combining. See also: Robots.txt Checker, and sitemap.xml Validator. Runs in your browser — nothing you paste gets uploaded.
How to use
- Set the User-agent (use * for all bots, or a specific crawler name).
- List Disallow and Allow paths one per line.
- Add absolute Sitemap URL(s) and an optional Crawl-delay.
- Generate, then validate with the Robots.txt Checker before deploy.
Example
Block admin, allow public, declare sitemap
Input
User-agent: * Disallow: /admin/ Allow: /public/ Sitemap: https://example.com/sitemap.xml
Output
User-agent: * Disallow: /admin/ Allow: /public/ Sitemap: https://example.com/sitemap.xml
Limitations
- Generates a single User-agent group — multi-bot files need manual combining.
- Does not fetch or verify that Sitemap URLs return 200.
- Crawl-delay is ignored by Googlebot; do not rely on it for Google crawl rate.
Related guides
1Related tools
2Frequently asked questions
What does the Robots.txt Generator do?
Build a robots.txt file with user-agent rules, allows, and sitemaps. Runs in your browser — no install, no account.
How do I use the Robots.txt Generator?
Set the User-agent (use * for all bots, or a specific crawler name). List Disallow and Allow paths one per line. Add absolute Sitemap URL(s) and an optional Crawl-delay. Generate, then validate with the Robots.txt Checker before deploy.
Is the Robots.txt Generator free?
Yes. No signup, no usage limits, no paywall.
Does the Robots.txt Generator upload my data?
No. Processing happens locally in JavaScript. Your input is not uploaded, stored, or logged.
What are the limitations of the Robots.txt Generator?
Generates a single User-agent group — multi-bot files need manual combining. Does not fetch or verify that Sitemap URLs return 200. Crawl-delay is ignored by Googlebot; do not rely on it for Google crawl rate.