Skip to content

Glossary · Technical

Definition

Crawl budget is the number of URLs a search engine is willing and able to crawl on a site within a given period. It combines crawl capacity, meaning how much the server can handle without slowing, and crawl demand, meaning how much the engine wants the content. It matters mainly for large or frequently changing sites.

  • Also: crawl rate
  • Also: crawl capacity
  • Also: crawl demand
  • Also: Googlebot crawl budget

How Crawl budget works

Google describes crawl budget as the product of two things. The crawl capacity limit depends on how quickly and reliably the server responds: slow pages and server errors make Googlebot back off. Crawl demand depends on how popular and fresh URLs appear, so frequently updated, well-linked pages are revisited more often. Other search engines and AI crawlers have their own limits, but the logic is similar.

The biggest misconception is that every site needs to manage it. A site with a few thousand pages that are indexed soon after publishing does not have a crawl budget problem, and Google's guidance says keeping a sitemap current is usually enough. It becomes real for marketplaces, large catalogues, news publishers and sites that generate URLs from filters, where crawlers spend their visits on endless parameter combinations instead of new pages.

Another misconception is that noindex saves crawling. A noindexed page still has to be fetched for the tag to be read. Blocking low-value patterns in robots.txt does prevent crawling, but it also stops engines seeing any canonical or noindex on those pages, so the choice needs care.

How to improve it

  1. 01Open the Crawl Stats report in Google Search Console and look at response times, error spikes and how much crawling goes to parameters or redirects.
  2. 02Stop creating crawlable URLs for filter and sort combinations nobody searches for, or block those patterns in robots.txt.
  3. 03Remove redirect chains and broken internal links so crawlers reach the final URL in one step.
  4. 04Keep XML sitemaps limited to canonical, indexable URLs with accurate lastmod dates.
  5. 05Improve server response times and stability, especially during traffic peaks.
An example

A fictional electronics retailer lets shoppers filter by brand, price, colour and rating, and every combination creates a new URL. Crawl Stats show Googlebot spending most of its visits there. The retailer keeps a few filtered pages with real demand, such as one brand within a category, as indexable landing pages and blocks the rest, so new product pages are found sooner.

Questions

Questions about Crawl budget

Still unsure? Ask the team or write to support@deylaxy.com.

Does my small website need to worry about crawl budget?

Usually not. If your site has a few thousand pages or fewer and new pages are indexed within days of publishing, crawl budget is unlikely to be what holds you back. Google's guidance says such sites rarely need to manage it beyond keeping a sitemap current. Slow indexing on a small site more often points to quality, duplication or weak internal linking than to crawl limits.

Do AI crawlers affect crawl budget?

AI crawlers do not use Google's crawl budget, but they do make their own requests and add load to your server. Many identify themselves by user agent, so you can allow or block them in robots.txt. Treat that as a visibility decision rather than a purely technical one: blocking the crawlers behind AI answer tools may keep your content out of some of those answers.

Related

Related terms

Get your free Visibility Intelligence Audit

A concise reading of verified signals across search engines and AI answers, enough to recognise the gaps, not a playbook for competitors.