SEOHub
robots.txt generator
Build a starter robots.txt with sitemap directive and disallow rules.
User-agent: * Allow: / Disallow: /admin Disallow: /staging Sitemap: https://example.com/sitemap.xml
Draft a robots.txt without blocking CSS or JS
Generate a conservative robots.txt starter, review Disallow rules carefully, and download the file. Never block required assets or entire sections you still want indexed.
Deploy to /robots.txt, verify with the inspector, and re-scan key templates. If AI crawlers matter for GEO, decide deliberately whether to allow GPTBot, ClaudeBot, and similar user-agents.
FAQ
Can this file hurt rankings?
Yes — a broad Disallow can hide important sections. Treat generated files as drafts and review before production.
Next: Inspect the live robots.txt · Run a free homepage scan