Robots txt mistakes that quietly remove a site from search

Aug 29
3 min read

Introduction

A site is rebuilt on a staging server with crawling blocked, which is correct. It goes live, the block is carried over with everything else, and for four months nothing new appears in search.

The site works perfectly for visitors. Nothing is broken, no error appears anywhere, and the only symptom is traffic that gradually stops growing. This is the characteristic shape of a robots file problem: rare, silent, and expensive by the time anybody investigates.

1. Robots txt mistakes are rare but severe

Understand the risk profile.

Most sites never have a problem here, and the ones that do can lose everything at once. That asymmetry is why the file deserves an occasional two-minute check rather than a standing project, and why the check should happen at the moments when it is most likely to have changed.

2. Read your own file before assuming it is fine

Start with the facts.

Add the standard robots path to your domain and read what is there. Many owners have never once seen theirs, and the file is usually short enough to understand in full even without any technical background at all.

3. Watch for the blanket disallow

The mistake that causes the disaster.

A single line disallowing everything for all crawlers blocks the entire site. It is standard on development environments and it is the thing to check first whenever indexing stops without another explanation.

4. Do not use it to hide pages from search results

The most common misunderstanding.

Blocking a page stops it being crawled, not necessarily being listed, and a blocked page can still appear as a bare address. To keep a page out of results, allow it to be crawled and use a no-index tag instead.

5. Do not block resources the page needs

A subtler failure.

Blocking script or stylesheet folders prevents a page being rendered as visitors see it, and it can be assessed as broken. Modern crawlers need those files, and blocking them serves no purpose.

6. Remember it is not a security measure

An important limit.

The file is a public request that well-behaved crawlers honour, and listing a private folder in it advertises the location to anybody who reads it. Anything genuinely sensitive needs authentication, not a line in a text file.

7. Check the pattern rules carefully

Small syntax, large consequences.

A trailing slash, a wildcard or a missing character changes what a rule matches, and blocking a directory blocks everything beneath it. Test the specific addresses you care about rather than reading the rule and assuming.

8. Keep the file simple

Complexity is where errors live.

Most small sites need a handful of lines: nothing blocked of consequence, and a reference to the sitemap. Long files accumulated over several years usually contain rules nobody can explain, a number of them obsolete and at least one of them blocking something that should not be blocked.

9. Check it after every deployment

The moment the damage happens.

Migrations, platform changes and staging deployments all overwrite this file. A check in the hour after going live is the single most valuable habit related to it, and it catches the disaster case before it costs anything.

Use the robots testing tool in your search console to confirm what is actually blocked. It reports what the live file does with a specific address, which is more reliable than reading the rules and reasoning about them.

Conclusion

Treat this as a rare problem with severe consequences, which justifies a short regular check rather than sustained attention.

Read your own file rather than assuming it is correct, look first for a blanket disallow left over from a build, use no-index tags rather than blocking when you want pages kept out of results, stop blocking scripts and stylesheets the page needs to render, remember the file offers no security at all, test specific addresses against your pattern rules, keep the file short, check it immediately after every deployment, and verify with the testing tool rather than by reading.