Crawl

Crawling in plain English

Crawling is how search engines discover and revisit pages. Automated programs, such as Googlebot, follow links from page to page and read sitemaps to find URLs, then download the content so it can be processed. Google explains the process in How Search works. A page that's never crawled can't be indexed or ranked.

How to check crawling on your site

Use the URL Inspection tool in Search Console on an important page. It shows when Google last crawled it, whether crawling was allowed and how the page looked to Googlebot. If a new page hasn't been crawled after a couple of weeks, check that it's linked from other pages and listed in your sitemap.

A worked example

A shop adds a new range of products but only links to it from a pop-up banner. Weeks later the products still don't appear in search. Adding the range to the main menu and the sitemap gives crawlers a normal path in, and the pages are found soon after.

Where it usually goes wrong

• Assuming that publishing a page means Google has found it.
• Blocking CSS or JavaScript that Google needs to render pages.
• Leaving important pages with no internal links.
• Worrying about crawl budget on a small site, where it's rarely the issue.

You can see how Google crawls your site in Search Console's crawl stats report.

Why a small site should care

Most small sites get crawled without any trouble. Problems usually come from a few causes: pages that no other page links to, important sections blocked in robots.txt, or content that only appears after clicks or scripts. If Google can't reach a page, none of the work on it counts.

Connected technical terms

• Googlebot: Google's main web crawler that fetches pages for indexing.
• Index: The searchable database of pages a search engine may show in results.
• Crawl budget: Roughly how much attention Googlebot gives to crawling a site.
• robots.txt: A file that suggests which parts of a site crawlers may fetch.