Crawl budget
Roughly how much attention Googlebot gives to crawling a site.
Crawl budget in plain English
Crawl budget is the amount of crawling a search engine is willing and able to do on your site in a given period. Google describes it as a combination of crawl capacity, meaning how much your server can handle, and crawl demand, meaning how much Google wants to crawl. Its crawl budget guide says it is mainly a concern for very large sites or sites with pages that change very often.
How to check crawl budget on your site
Look at Search Console's Crawl stats report, under Settings, to see how many requests Google makes and which kinds of URLs it fetches. If most requests go to parameter or filter URLs, or if new pages take a long time to be crawled, you may have a crawl efficiency problem. If not, you can probably move on.
A worked example
A holiday let company's booking calendar creates a new URL for every date for every property, producing hundreds of thousands of near-empty pages. Googlebot spends most of its visits there. Blocking calendar URLs from crawling and linking clearly to the property pages frees crawling up for what matters.
Where it usually goes wrong
- Blaming crawl budget for indexing problems on a small site, when quality or duplication is the real cause.
- Using noindex to save crawl budget. Google still has to crawl a page to see the tag.
- Letting filters and parameters create endless URL combinations.
- Ignoring slow server responses, which reduce how much Google will crawl.
Why a small site should care
For most small business sites with a few hundred pages, crawl budget isn't something to worry about. It becomes relevant when a site generates huge numbers of URLs, for example through filters, calendars or session parameters, so that crawlers spend time on junk instead of important pages.
Connected technical terms
- Crawl: When a search engine bot discovers URLs by following links and sitemaps.
- Googlebot: Google's main web crawler that fetches pages for indexing.
- Faceted navigation: Filter URLs that can create crawl and duplicate issues if uncontrolled.
- Log file analysis: Studying server logs to see how bots crawl your site.
Start with a free SEO audit
Your whole site, audited free. No card and no obligation.