Googlebot

Googlebot in plain English

Googlebot is the name of Google's main web crawler, the program that fetches pages for Google Search. It comes in smartphone and desktop versions, and with mobile-first indexing the smartphone crawler does most of the work. Google publishes how to verify that a visitor really is Googlebot, because fake bots often pretend to be it.

How to check Googlebot on your site

Search Console's Crawl stats report, under Settings, shows how often Googlebot visits, which responses it receives and whether your server struggled. A sudden jump in errors usually means something changed on your hosting or security setup. The URL Inspection tool's live test shows how Googlebot sees an individual page right now.

A worked example

A café owner installs a strict security plugin. Soon afterwards, Search Console reports a rise in server errors. The plugin was blocking Googlebot's requests. Whitelisting verified Google crawlers fixes it, and normal crawling resumes.

Where it usually goes wrong

• Blocking all bots at the firewall without allowing verified search engine crawlers.
• Trusting the user-agent string alone. Anyone can claim to be Googlebot.
• Serving Googlebot different content from visitors, which Google treats as cloaking under its spam policies.
• Assuming a page Googlebot can't render will still rank.

Why a small site should care

You rarely need to think about Googlebot directly. It matters when something blocks it: a firewall that treats it as an attacker, a robots.txt rule, a server that's slow or down, or a security plugin that serves it an error. Any of those can quietly stop your pages being updated in search.

Most website owners never need to think about it. It only becomes your concern when something in your hosting, security or code accidentally gets in its way.

Connected technical terms

• Crawl: When a search engine bot discovers URLs by following links and sitemaps.
• robots.txt: A file that suggests which parts of a site crawlers may fetch.
• Crawl budget: Roughly how much attention Googlebot gives to crawling a site.
• Log file analysis: Studying server logs to see how bots crawl your site.