Skip to content
GetProfit
Log in

← All terms

Crawl budget

Crawl budget is the set of a site's URLs that Google can and wants to crawl, limited by how much load the server can take and by Google's demand for those pages.

How it works

Two things set crawl budget. The crawl capacity limit depends on the server: fast, stable responses raise it; slowdowns and server errors lower it. Crawl demand depends on how many URLs Google knows about, how popular they are and how often they change. Duplicate and unneeded URLs waste the budget, and faceted navigation is a common source: each filter combination is a new URL.

All Google crawlers, Googlebot included, share the capacity limit. Google Shopping has its own demand for the products in a store’s Merchant Center feed. Google notes that high demand from one crawler can reduce capacity for others.

Crawl budget may not be your problem at all. Google’s guide is for sites with 1 million+ unique pages that change weekly, 10,000+ that change daily, or many URLs left as “Discovered – currently not indexed”. Google calls these rough estimates. Product pages not indexed explains which pages are worth the effort.

Where you see it

  • Search Console → Crawl Stats report: Googlebot’s crawl history and host availability.
  • Search Console → Page indexing report: the “Discovered – currently not indexed” status.

Example

Example store, not client data.

The tableware shop has 3,000 products in 40 categories. Each category can be filtered by 8 colours, 5 materials and 4 price bands, and every combination gets its own URL: 40 × 8 × 5 × 4 = 6,400 filter pages. Google can now find 9,440 URLs instead of 3,040, and two in three of them are filters (6,400 ÷ 9,440 = 68%).

Not to be confused with

  • Indexing — storing a page for search results. Not every crawled page is indexed.

Right and wrong readings

  • Wrong: “New products aren’t in Google a day later, so we’ve run out of crawl budget.” Right: Google says most sites shouldn’t expect same-day crawling: checking and indexing a page usually takes three days or more.
  • Wrong: “Put noindex on filter pages to save crawl budget.” Right: Google still requests a noindex page before dropping it. To stop crawling filter URLs not needed in search, Google suggests robots.txt.

Sources