robots.txt Generator
Profesyonel robots.txt Generator
Default Policy
Block Paths
Block Specific Bots
Sitemap URL
robots.txt
# Generated by OnDevTools.com User-agent: * Allow: / Disallow: /admin Disallow: /wp-admin Disallow: /administrator Disallow: /api/ Disallow: /cgi-bin/
robots.txt Generator
Profesyonel robots.txt Generator
Özellikler
- Configurable User-agent + Disallow + Allow rules
- Multiple User-agent blocks (separate rules for Googlebot, Bingbot, etc.)
- Crawl-delay directive (with per-bot support where you need it)
- Sitemap URL declarations (one or more)
- Common-case presets (allow all / disallow all / disallow private)
- Copy to clipboard or download as robots.txt
Nasıl Kullanılır
- Girin sizin giriş veri içinde the provided field.
- Configure herhangi seçenekler as needed.
- Görüntüle the sonuçlar anında ve copy veya download the çıkış.
İpuçları ve En İyi Uygulamalar
- robots.txt is advisory, not security. Compliant crawlers respect it; malicious ones ignore it. Use auth for actual access control.
- Disallow blocks paths, not URL queries. Use noindex meta tags or HTTP headers for per-page exclusions.
- Sitemap directives can list multiple sitemaps; one per line, full URL.
- Crawl-delay is honoured by Bing and Yandex but ignored by Google (use Google Search Console to throttle Googlebot).
- Place the generated file at the site root: https://example.com/robots.txt
SSS
Is this tool free to use?
Yes, completely free with no registration, no limits, and no hidden costs.
Is my data private?
All processing happens in your browser. No data is ever sent to any server or third party.
Does it work offline?
Once loaded, the tool works entirely offline. All processing is done client-side with JavaScript.
Which browsers are supported?
Works in Chrome, Firefox, Safari, Edge, and all modern browsers on desktop and mobile.
Can I use the results commercially?
Yes, all output is yours to use freely in any personal or commercial project without restrictions.