robots.txt reader

What robots.txt does

It tells crawlers which parts of your site they may visit. It doesn’t remove pages from Google. A blocked page can still appear in results if other sites link to it, just without a description.

The mistake to look for

A stray “Disallow: /” under “User-agent: *” blocks everything. It’s common after a site moves from a staging server. If your traffic fell off a cliff after a redesign, check this first.

Noindex and robots.txt don’t mix

If you want a page out of Google, use a noindex tag and let Google crawl the page so it can see the tag. Blocking it in robots.txt stops Google reading the noindex.

How to use it

1. Open yourdomain.co.uk/robots.txt in a browser.
2. Copy everything and paste it into the box.
3. Press the button to see the rules listed line by line.
4. Look for any Disallow line that covers pages you want in Google.

What it reads

It picks out every User-agent, Allow, Disallow and Sitemap line and lists them in order, ignoring comments and anything it doesn’t recognise. That makes long or messy files easier to scan.

It doesn’t test a specific address against the rules, and it doesn’t work out which group of rules a particular crawler will follow. For that, use the robots.txt report in Google Search Console, which shows the file Google actually fetched.

Robots.txt and AI crawlers

Some sites now add rules for AI crawlers, such as GPTBot or Google-Extended, as well as search engines. Each one has its own User-agent group. If you’ve added these, check the reader’s output to make sure the groups for search crawlers like Googlebot haven’t been blocked by accident. A rule meant for one crawler only applies to the User-agent lines directly above it.

Next step

If you’re not sure whether a rule is safe, leave the file alone and ask. Removing the wrong line is easier to fix than blocking your whole site.

On the £99 plan our daily checks watch your robots.txt and tell you if a change starts blocking pages that matter.

A worked example

A common file looks like this: User-agent: * then Disallow: /wp-admin/ then Allow: /wp-admin/admin-ajax.php then a Sitemap line. The reader lists those four rules so you can see that only the admin area is blocked, which is normal for a WordPress site.

The one to worry about is Disallow: / on its own under User-agent: *. That blocks every page for every crawler that follows the file. It often gets left behind when a site moves from a test address to the live one.

Related reading

Our technical SEO basics guide covers robots.txt, sitemaps and indexing together.

Other free tools