Crawl budget myths for small sites

What crawl budget is

Crawl budget is the number of URLs Google can and wants to crawl on your site in a given time. Google's crawl budget guide describes it as a combination of:

• Crawl capacity limit: how much crawling your server can handle without slowing down.
• Crawl demand: how much Google wants to crawl, based on how popular and how often updated your pages are.

Who it's for

Google is clear that its crawl budget guide is aimed at:

• Large sites with around a million or more unique pages whose content changes moderately often, or
• Medium or larger sites with roughly 10,000 or more unique pages whose content changes very rapidly, such as daily, or
• Sites with a large share of URLs classified as "Discovered, currently not indexed" in Search Console.

A small business site with a few dozen or a few hundred pages isn't in that group. Google says if your site doesn't have a large number of pages that change rapidly, or if pages tend to be crawled the same day they're published, you don't need to read the guide.

Myth 1: "My pages aren't indexed because of crawl budget"

For small sites, almost never. When pages aren't indexed, the usual reasons are:

• A noindex tag, often left on from development or added by a plugin.
• Blocked by robots.txt.
• A canonical tag pointing somewhere else.
• Duplicate content, so Google picks another version.
• Low quality or thin content, so Google decides it's not worth indexing.
• No internal links pointing to the page.

Search Console's Page indexing report shows which reason applies. "Crawled, currently not indexed" usually means Google fetched the page and decided not to index it, which is a quality or duplication question, not a crawling one.

Myth 2: "I should block pages to save crawl budget"

Blocking unimportant pages with robots.txt doesn't help a small site rank better. It can even cause problems, because Google can't see a noindex tag on a page it's blocked from crawling. Only block what genuinely shouldn't be crawled, such as admin areas or endless internal search URLs.

Myth 3: "Submitting my sitemap daily will get me crawled more"

Sitemaps help Google discover URLs, but resubmitting them repeatedly doesn't increase crawling. Keep your sitemap accurate and let Google fetch it. If pages change, an accurate last-modified date can help.

Myth 4: "Nofollow on internal links saves crawl budget"

Using nofollow on your own internal links doesn't help and can make it harder for Google to understand your site. Leave internal links normal.

Myth 5: "Removing old pages always helps crawling"

Removing genuinely useless pages can make a site clearer, but for small sites the benefit is about quality and maintenance, not crawl capacity. See our content pruning guide.

When crawling can matter for smaller sites

There are a few situations where crawling is worth a look even on smaller sites:

• Faceted navigation on shops creating thousands of filter combinations.
• Calendar or booking systems generating endless date URLs.
• Session IDs or tracking parameters creating duplicate URLs.
• A slow or unreliable server that makes Google back off.
• Hacked content adding thousands of spam pages.

In these cases, the problem isn't really budget but wasted crawling on URLs that shouldn't exist. Fix the source.

What to focus on instead

If pages aren't being indexed or are slow to appear:

1. Check the Page indexing report for the specific reason. 2. Inspect the URL to see what Google found. 3. Improve internal linking so important pages are linked from prominent places. 4. Improve the content if it's thin, duplicated or unhelpful. 5. Make sure your server is fast and reliable. The Crawl stats report shows response times and host problems. 6. Keep your sitemap accurate, listing only canonical, indexable URLs.

A worked example

The owner of a holiday cottage letting site with around 200 pages read that crawl budget was why 40 of her property pages weren't indexed. She was about to block her blog in robots.txt to "save budget".

Search Console showed the 40 pages as "Duplicate, Google chose different canonical than user". Each property had two URLs: one from the search results and one from the property list, with slightly different parameters. Google had picked one version for each, sometimes the less tidy one.

She fixed the internal links to use one clean URL per property, set canonical tags correctly and updated the sitemap. Over the following weeks, the clean URLs were indexed. Blocking the blog would have done nothing except hide useful content.

Questions