robots.txt

robots.txt in plain English

robots.txt is a plain text file at the root of your site, such as example.co.uk/robots.txt, that tells crawlers which parts of the site they may fetch. It controls crawling, not indexing. Google's robots.txt introduction explains that a blocked page can still be indexed if other pages link to it, just without its content.

How to check robots.txt on your site

Type your domain followed by /robots.txt into a browser to read your file. Look for any "Disallow: /" line under "User-agent: *", which blocks everything. Search Console's robots.txt report shows the version Google last fetched and any errors. If you're unsure what a rule does, ask whoever built the site before changing it.

A worked example

A new ecommerce site launches, but after a month nothing shows in Google. The robots.txt still blocks everything from the staging setup. Removing the rule and submitting the sitemap in Search Console starts the pages appearing within days.

Where it usually goes wrong

• Using robots.txt to keep a page out of Google. Use a noindex tag instead, and let the page be crawled so Google can see it.
• Blocking CSS and JavaScript files that Google needs to render pages.
• Treating it as security. The file is public and only polite bots obey it.
• Forgetting to list your sitemap location in it.

Why a small site should care

One wrong line can hide a whole website from search. A common example is a developer adding "Disallow: /" while building a site and forgetting to remove it at launch. On the other hand, a sensible robots.txt can keep crawlers out of areas such as internal search results or basket pages.

Connected technical terms

• Crawl: When a search engine bot discovers URLs by following links and sitemaps.
• Googlebot: Google's main web crawler that fetches pages for indexing.
• XML sitemap: A file listing important URLs to help search engines discover pages.
• Index: The searchable database of pages a search engine may show in results.