Robots.txt Generator
Generate robots.txt with User-agent, Allow, Disallow, Crawl-delay and Sitemap rules, including common presets and automatic syntax and URL validation.
About This Tool
The robots.txt generator assembles crawl rules using standard syntax. Create multiple rule groups for different User-agents, each with its own Allow and Disallow paths and Crawl-delay, then append Sitemap URLs at the end. Three presets cover allow all, block all and block admin paths, so you can start from a working baseline.
Validation runs before generation: groups missing User-agent are reported and skipped, and Sitemap URLs must be absolute http or https links, preventing a single bad line from breaking the whole file.
Key Features
Multiple rule groups
Configure independent rules per search engine.
Common presets
Apply allow, block and admin-blocking configurations instantly.
Syntax validation
Checks for missing User-agent and invalid Sitemap URLs.
Deploy-ready output
Standard text you can copy into robots.txt at the site root.
When to Use It
New site launch
Generate baseline rules that allow crawling and declare the sitemap.
Block sensitive paths
Disallow admin, API and temporary directories to reduce exposure.
Per-engine control
Set crawl delays or blocks for specific search engines only.
How to Use It
- 1Pick a preset or add a rule group manually.
- 2Fill in User-agent plus Allow and Disallow paths, one per line.
- 3Enter absolute Sitemap URLs.
- 4Generate and copy the result to the site root.
Frequently Asked Questions
Does robots.txt prevent indexing?
robots.txt is only a crawl convention and does not guarantee removal from indexes. Use a noindex meta tag or header to block indexing completely.
What wins when Allow and Disallow conflict?
Allow wins for equally long prefixes, but engine implementations vary, so avoid writing conflicting rules.
