SEOHub

robots.txt & sitemap inspector

See whether crawlers can reach your site and how many URLs your sitemap lists.

Related tools

All free tools

robots.txt generator
Draft a starter robots.txt with sitemap rules.
Sitemap generator
Build a downloadable sitemap.xml from a URL list.
Security headers
Check response headers on the same host.
Redirect chain
Follow canonical and www/non-www hops.

Inspect robots.txt and sitemaps safely

Check whether robots.txt allows the paths you care about, whether a sitemap is advertised, and how many URLs are listed in the sample we fetch. Accidental Disallow rules and missing sitemaps are among the fastest ways to stall indexation.

If the file is missing or wrong, generate a starter robots.txt or sitemap with the free generators, deploy them, then re-inspect. A full-site crawl still helps catch noindex and canonical problems that robots.txt cannot show.

FAQ

Can a disallow block Google forever?

While Disallow is present, Googlebot should skip those paths. Remove or narrow the rule, then request indexing after the live file updates.

Next: Generate a robots.txt starter · Run a free homepage scan