6 Robots.txt Mistakes That Can Quietly Deindex Your Site

Why one file causes this much damage

Robots.txt sits at your domain's root (yoursite.com/robots.txt) and it's the first thing well-behaved crawlers check before touching anything else on your site. It's plain text, it's usually only a few lines long, and it's often the single easiest file on a site to edit by accident — which is exactly why it causes so many real incidents. A staging-environment robots.txt (which typically blocks everything) accidentally deployed to production is one of the most common, and most damaging, mistakes in SEO.

How robots.txt actually works

The syntax is deceptively simple, which is part of the problem — it's easy to assume you understand it fully after reading two lines:

• User-agent: which crawler the following rules apply to (* means all crawlers).
• Disallow: a path crawlers shouldn't request. Disallow: / blocks the entire site — this is the single most common catastrophic typo.
• Allow: carves out an exception within a disallowed path.
• Sitemap: points crawlers to your sitemap file directly.

What trips people up: robots.txt controls crawling, not indexing. A page blocked in robots.txt can still show up in Google's index (usually with no description, just the URL) if other sites link to it — because Google never crawled the page to see a noindex tag, it just knows the URL exists. If you actually want a page out of Google's index, robots.txt is the wrong tool; you need a noindex meta tag, which requires the page to be crawlable in the first place.

The one-line incident, before and after

This is the exact pattern behind most "my traffic dropped to zero" panics — a staging config that was never swapped out:

User-agent: *
Disallow: /

Sitemap: https://example.com/sitemap.xml

Disallow: / blocks every path on the site for every crawler, full stop — the Sitemap: line underneath doesn't help, because Google won't even request pages to see it's there. The fix is almost always this simple:

User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /

Sitemap: https://example.com/sitemap.xml

Everything is allowed by default now, with two specific paths carved out — an admin area and a cart flow, neither of which need to be indexed. That's the whole difference between a site that's fully blocked and one that isn't: which side of Disallow: / the rest of the file sits on.

The most dangerous robots.txt mistakes

Directives worth knowing but rarely worth using

A few older directives still show up in robots.txt files and generators, worth knowing even though most sites shouldn't reach for them:

• Crawl-delay: asks a crawler to wait a number of seconds between requests. Google has ignored it for years — crawl rate for Google specifically is set in Search Console, not robots.txt — though some other crawlers (Bing, and various SEO/scraping bots) still respect it.
• Wildcards (* and $) — * matches any sequence of characters within a path, and $ anchors a rule to the end of a URL. Disallow: /*.pdf$ blocks every PDF on the site in one line, which is genuinely useful when a whole file type shouldn't be indexed.
• Multiple User-agent blocks — you can target specific crawlers differently (allow Googlebot everywhere, block an aggressive scraping bot entirely) by giving each its own block with its own rules underneath.

Robots.txt versus the meta robots tag versus the X-Robots-Tag header

These three sound similar and get confused constantly, but they operate at completely different levels. Robots.txt is a site-wide crawl instruction that applies before a crawler ever requests a page — it can only say "don't fetch this URL," nothing more granular. The meta robots tag lives inside a specific HTML page's <head> and can only take effect once that page has already been crawled, which is exactly why it's the right tool for noindex and robots.txt isn't. The X-Robots-Tag is the same directive set as the meta robots tag, but sent as an HTTP response header instead of embedded in HTML — the only option for non-HTML files like PDFs or images, which have no <head> to put a meta tag in. Knowing which layer a given directive operates at prevents exactly the mistake covered above: reaching for robots.txt to solve a problem that only the meta tag or header can actually fix.

Confirming the fix actually landed

Editing the file is the easy part — knowing Google has actually picked up the change is where people stop too early. After fixing a robots.txt issue, use Search Console's URL Inspection tool on one of the previously-blocked pages: it shows the live, current robots.txt ruling for that exact URL, not a cached version. If it still shows blocked, request indexing on that URL directly rather than waiting for the next scheduled crawl — Google generally re-checks robots.txt itself within a day, but individual pages can sit in the crawl queue longer than that if you don't nudge them.

How to check yours right now

CheckSEO's Robots.txt Checker fetches your live file, flags syntax errors and dangerous rules, and shows exactly which paths are blocked — before you find out the hard way via a drop in Search Console impressions.