Using crawl stats and server logs
Crawl stats and server logs show what search engine bots actually fetch on your site. Here's when they're worth checking and what to look for.
Why look at crawling at all
Most SEO tools show what could happen: what a crawler can find by following links. Crawl data shows what did happen: which pages Google's bots actually requested, how often and with what result.
For most small business sites, crawling is rarely a problem. But checking now and then can reveal issues nothing else shows: bots stuck in endless filter URLs, important pages rarely visited, or server errors happening at night.
Search Console's Crawl stats report
The easiest place to start is the Crawl stats report, found under Settings in Search Console. It shows:
- Total crawl requests over the last 90 days.
- Total download size.
- Average response time.
- Host status, flagging problems with robots.txt fetching, DNS or server connectivity.
- Breakdowns by response code, file type, purpose (discovery or refresh) and Googlebot type.
What to look for
- Host status warnings. If Google had trouble fetching robots.txt or connecting to your server, it may have slowed crawling.
- Rising server errors (5xx). Your server is struggling or misconfigured.
- Many 404s. Bots are following broken links, often from old URLs.
- Rising response times. Pages are getting slower to serve.
- Unexpected file types. Lots of requests for scripts or images you didn't expect may point to configuration issues.
Server logs
Every request to your website is usually recorded in server access logs. Each line shows the time, the requesting IP address, the URL requested, the response code and the user agent, which identifies whether it was a browser, Googlebot, Bingbot or something else.
Logs show everything, including crawling Search Console doesn't report in detail. They're particularly useful for:
- Larger sites with thousands of pages.
- Shops with filters creating many URL variations.
- Diagnosing why certain pages aren't being indexed.
- Checking whether bots respect robots.txt rules.
Getting access
On managed hosting, look for "raw access logs" in the control panel. Some hosts keep only a few days. Website builders such as Wix or Squarespace usually don't give log access.
Verify Googlebot
Plenty of bots pretend to be Googlebot. Google explains how to verify Googlebot using reverse DNS lookup or its published IP ranges. Filter out fakes before drawing conclusions.
What logs can reveal
Wasted crawling
If most Googlebot requests go to filter URLs, calendar pages, session IDs or internal search results, Google is spending effort on pages that don't matter. That's worth fixing on larger sites. Google's guide on crawl budget explains why this matters mainly for big or fast-changing sites.
Neglected pages
If important pages are crawled rarely, they may be poorly linked internally. Improve internal links from frequently crawled pages.
Errors bots see but people don't
Some errors only happen under bot load or at certain times. Logs show them.
Unwanted bots
Logs also show aggressive scrapers and bots hammering your server. You can block or rate-limit them, but be careful not to block legitimate search engine crawlers.
Do small businesses need log analysis?
Usually not regularly. A small brochure site with fifty pages rarely has crawl problems. The Crawl stats report is enough for an occasional check.
Log analysis becomes worthwhile when:
- You have thousands of URLs, such as a shop or listings site.
- Important pages aren't being indexed and you can't see why.
- You've had a migration and want to confirm bots are following redirects.
- Your server is struggling and you suspect bots.
A worked example
An online wine merchant with around 3,000 products noticed that new products took weeks to appear in Google. Search Console's Crawl stats showed plenty of crawling, so the merchant downloaded a month of server logs.
Filtering for verified Googlebot requests showed that a large share went to URLs combining several filters and sort orders, such as region, grape, price band and "sort by newest". Few requests reached new product pages, which were only linked from the bottom of a long category list.
The merchant stopped filter combinations being crawled where they had no search value, added a "new arrivals" section linked from the home page and kept only useful filter pages indexable. Over the following weeks, logs showed more Googlebot requests going to products, and new products were indexed sooner.
Tools
- Search Console Crawl stats, free and the best place to start.
- Spreadsheet software, for small log files.
- Dedicated log analysers, for larger sites. Several SEO crawler tools include log analysis.
- Your host's support team, who can often help with access and server errors.
Questions
Can I make Google crawl my site more often?
Not directly. Google crawls based on how useful and how frequently updated your pages are, and on server capacity. Fresh, useful content, good internal links and a fast server help. Sitemaps help Google find new pages.
Should I block AI crawlers?
That's a business decision. You can control some AI crawlers with robots.txt rules. Blocking Googlebot itself would remove you from Google Search, so be careful with the exact user agents you block.
Start with a free SEO audit
Your whole site, audited free. No card and no obligation.