Crawl budget is the attention search engines allocate to discovering and refreshing your URLs. Small sites rarely hit a hard ceiling; they still lose when bots waste fetches on junk while new or updated money pages wait.
The usual budget drains: soft 404s (200 responses with “not found” content), long redirect chains, infinite faceted URL spaces, session parameters, and thin duplicates that all stay crawlable. Each wasted request is a request that did not hit a page you care about.
Start with access control. Confirm robots.txt is not blocking important sections and that noindex is reserved for pages you truly want out of the index. Inspect rules with the robots.txt tool before you rewrite canonical strategy.
Collapse duplicates with canonicals and redirects. Prefer one HTTPS host, one slash policy, and 301s that finish in a single hop. Validate with the redirect checker — chains through legacy domains are a silent tax on every bot visit.
Kill or parameter-handle faceted crawl paths you do not want indexed. If every color × size × sort combination is linked in HTML, you have built an infinite calendar for the crawler. Internal-link hygiene matters as much as robots rules.
Fix broken links and error templates. 4xx/5xx targets and soft 404s show up in SEOHub’s crawl and checklist; the broken links tool helps when you want a focused pass on nav and footer templates.
Sitemaps should list canonical, indexable URLs only — not redirects, not noindexed thank-you pages, not staging leftovers. After regenerating, run a full-site scan to compare what your HTML links to versus what you intended to promote.
Watch crawl stats in Search Console alongside your audit. Spikes in “discovered, not indexed” or endless parameter URLs usually trace back to a template or filter UI change, not a mysterious penalty.
JavaScript-only navigation can hide important URLs from simple link graphs. Ensure critical paths exist as real <a href> links so both Googlebot and SEOHub’s crawler can discover them without executing every widget.
Re-scan after information-architecture changes. Crawl budget issues are systemic; one merged PR that adds an unbounded filter can reopen hundreds of URLs. Use SEOHub’s site-wide report as the before/after proof for engineering, not only for SEO stakeholders.
Run a free homepage scan · Technical SEO Checklist · Fix guides · Free tools