robots.txt Generator
Professional robots.txt Generator
Default Policy
Block Paths
Block Specific Bots
Sitemap URL
robots.txt
# Generated by OnDevTools.com User-agent: * Allow: / Disallow: /admin Disallow: /wp-admin Disallow: /administrator Disallow: /api/ Disallow: /cgi-bin/
robots.txt Generator
Professional robots.txt Generator
Features
- Configurable User-agent + Disallow + Allow rules
- Multiple User-agent blocks (separate rules for Googlebot, Bingbot, etc.)
- Crawl-delay directive (with per-bot support where you need it)
- Sitemap URL declarations (one or more)
- Common-case presets (allow all / disallow all / disallow private)
- Copy to clipboard or download as robots.txt
How to use
- Enter your input data in the provided field.
- Configure any options as needed.
- View the results instantly and copy or download the output.
Tips & Best Practices
- robots.txt is advisory, not security. Compliant crawlers respect it; malicious ones ignore it. Use auth for actual access control.
- Disallow blocks paths, not URL queries. Use noindex meta tags or HTTP headers for per-page exclusions.
- Sitemap directives can list multiple sitemaps; one per line, full URL.
- Crawl-delay is honoured by Bing and Yandex but ignored by Google (use Google Search Console to throttle Googlebot).
- Place the generated file at the site root: https://example.com/robots.txt
FAQ
Is this tool free to use?
Yes, completely free with no registration, no limits, and no hidden costs.
Is my data private?
All processing happens in your browser. No data is ever sent to any server or third party.
Does it work offline?
Once loaded, the tool works entirely offline. All processing is done client-side with JavaScript.
Which browsers are supported?
Works in Chrome, Firefox, Safari, Edge, and all modern browsers on desktop and mobile.
Can I use the results commercially?
Yes, all output is yours to use freely in any personal or commercial project without restrictions.