Glossaire · SEO

Crawl budget

Crawl budget refers to the amount of resources Googlebot dedicates to exploring a given site over a defined period. It results from two factors: the crawl rate limit (the server's capacity to respond without slowing down) and crawl demand (Google's interest in your URLs, driven by their popularity and freshness). For most sites under a few thousand URLs, crawl budget is not a limiting factor. On large e-commerce sites, media platforms, or systems generating URLs on the fly, however, poor management wastes this budget on low-value pages (parameters, duplicates, error pages) and delays the exploration of strategic content. Optimizing crawl budget means concentrating the crawlers' attention on important URLs by eliminating waste and making priority content easier to discover.

Crawl budget is a core concept in technical SEO for large sites. It describes how Google arbitrates the time and resources it allocates to exploring your site, rather than a fixed limit communicated in advance.

How it works

Google determines crawl budget from two variables. The crawl rate limit depends on the server's technical health: if your pages respond quickly and without errors, Googlebot increases its pace; if the server slows down or returns 5xx errors, it reduces pressure. Crawl demand reflects Google's interest in your URLs: popular, frequently updated, and well-linked pages are visited more often. The combination of the two defines the volume of URLs explored per day.

A concrete example

An e-commerce site with 200,000 pages generates, through its navigation filters (color, size, price), hundreds of thousands of parameterized URLs. Without control, Googlebot exhausts its budget on these worthless variants and is slow to explore new product pages. By blocking these parameters, consolidating duplicates with canonical tags, and refining internal linking, you redirect crawling toward the pages that matter.

A retenir

Crawl budget cannot be purchased: it is earned through server performance and preserved by removing waste on worthless URLs.

Why it matters

A page that is never crawled will never be indexed or ranked. On a large site, a poorly managed crawl budget creates a delay between publishing content and Google taking it into account, which directly harms visibility. Controlling this budget accelerates the indexing of priority pages and improves the perceived freshness of the entire site.

FAQ

Questions fréquentes

No. For a site under a few thousand pages, Google generally explores everything without difficulty. Crawl budget optimization becomes critical beyond 10,000 URLs, or on sites generating many dynamic pages and parameters.

Review the Crawl Stats report in Search Console along with your server logs. If Googlebot spends most of its time on low-value pages (filters, duplicates, 404 errors), your budget is poorly distributed.

Pour aller plus loin