How Crawl budget works
Google describes crawl budget as the product of two things. The crawl capacity limit depends on how quickly and reliably the server responds: slow pages and server errors make Googlebot back off. Crawl demand depends on how popular and fresh URLs appear, so frequently updated, well-linked pages are revisited more often. Other search engines and AI crawlers have their own limits, but the logic is similar.
The biggest misconception is that every site needs to manage it. A site with a few thousand pages that are indexed soon after publishing does not have a crawl budget problem, and Google's guidance says keeping a sitemap current is usually enough. It becomes real for marketplaces, large catalogues, news publishers and sites that generate URLs from filters, where crawlers spend their visits on endless parameter combinations instead of new pages.
Another misconception is that noindex saves crawling. A noindexed page still has to be fetched for the tag to be read. Blocking low-value patterns in robots.txt does prevent crawling, but it also stops engines seeing any canonical or noindex on those pages, so the choice needs care.
How to improve it
- 01Open the Crawl Stats report in Google Search Console and look at response times, error spikes and how much crawling goes to parameters or redirects.
- 02Stop creating crawlable URLs for filter and sort combinations nobody searches for, or block those patterns in robots.txt.
- 03Remove redirect chains and broken internal links so crawlers reach the final URL in one step.
- 04Keep XML sitemaps limited to canonical, indexable URLs with accurate lastmod dates.
- 05Improve server response times and stability, especially during traffic peaks.
A fictional electronics retailer lets shoppers filter by brand, price, colour and rating, and every combination creates a new URL. Crawl Stats show Googlebot spending most of its visits there. The retailer keeps a few filtered pages with real demand, such as one brand within a category, as indexable landing pages and blocks the rest, so new product pages are found sooner.