Client-Side UtilityLocal memory execution • Zero server telemetry
Robots.txt Generator (RFC 9309 Protocol Standard)
Generate standard-compliant robots.txt files with crawl directives, AI scraper blocking, custom user agents, and sitemap references. 100% in-browser with zero tracking.
SMaintained by SEO & Webmaster Tools Desk
Published
Bot Crawler Directives
User-Agent Directives
1 RuleRule #1
Generated robots.txt
RFC 9309 CompliantUser-agent: *
Quick Directive Presets
SMaintained by SEO & Webmaster Tools Desk
Published
Calculation Methodology & Formula
Formats directives conforming to RFC 9309 (Robots Exclusion Protocol). Parsers read User-agent blocks matching specific crawlers or the wildcard * selector.
User-agent: * | Disallow: /admin/ | Allow: / | Sitemap: https://...Worked Example: Standard Production Website Configuration
Allow search crawlers across public routes while shielding administrative and private directories.
user Agent:*
allow Path:/
disallow Path:/admin/, /api/, /private/
sitemap:https://example.com/sitemap.xml
Expected OutputStandard robots.txt with disallows and verified sitemap line
Protects backend and staging routes from indexing while directing crawlers to the authoritative XML sitemap.
Technical Constraints & Limitations
- robots.txt acts as a voluntary directive for respectful search engine crawlers; malicious scrapers can ignore exclusion rules.
- Disallowing a page does not guarantee removal from search results if external websites link to that URL with descriptive anchor text.
Frequently Asked Questions
Where must the robots.txt file be placed?
The robots.txt file must reside in the exact root of your domain: https://example.com/robots.txt. Subdirectory locations (like /tools/robots.txt) are ignored by search bots.
Is robots.txt case-sensitive?
Directive names (User-agent, Disallow, Allow) are case-insensitive, but directory and file paths are strictly case-sensitive.
How do I block AI training crawlers specifically?
Select the "Block AI Scrapers" preset. It generates targeted Disallow directives for major training bots including GPTBot, CCBot, Google-Extended, and anthropic-ai.