Log file analysis
Studying server logs to see how bots crawl your site.
Log file analysis in plain English
Log file analysis means studying your web server's access logs to see exactly which URLs search engine crawlers request, how often and what responses they receive. Logs record every request, so they show real crawler behaviour rather than estimates. Analysts usually verify that requests really come from Googlebot before drawing conclusions.
How to check log file analysis on your site
Ask your host whether you can download access logs. Many shared hosting plans provide them. Start with Search Console's Crawl stats report, which summarises Google's requests, response codes and file types. Only move on to full log analysis if that report suggests a problem worth investigating.
A worked example
An online parts supplier with thousands of products notices new items take weeks to appear in Google. Its logs show Googlebot spending most of its time on old filter URLs and session parameters. Fixing those paths means new products get crawled much sooner.
Where it usually goes wrong
- Analysing logs without verifying which requests are really Googlebot.
- Spending time on logs for a 20-page site.
- Ignoring server errors that only crawlers seem to trigger.
- Forgetting that logs may contain personal data that needs handling carefully.
Why a small site should care
Most small business sites don't need log analysis. Search Console's Crawl stats report gives enough detail. Logs become useful on larger or more complex sites, such as ecommerce stores with filters, or when you suspect crawlers are wasting time on junk URLs or hitting errors that don't show elsewhere.
It's also a way to settle disagreements with data. If a developer insists Google never visits a section, or a plugin claims to block bots, the logs show what actually happened.
Connected technical terms
- Googlebot: Google's main web crawler that fetches pages for indexing.
- Crawl: When a search engine bot discovers URLs by following links and sitemaps.
- Crawl budget: Roughly how much attention Googlebot gives to crawling a site.
Start with a free SEO audit
Your whole site, audited free. No card and no obligation.