Robots.txt: How It Works and How Not to Break Your SEO
robots.txt is a plain text file at the root of your domain that tells crawlers which paths they may or may not fetch. It's the first thing a crawler checks — and one wrong Disallow line can hide your entire site from search engines. Here's how to use it correctly.
What is robots.txt?
robots.txt is a plain text file that lives at the root of your domain (yoursite.com/robots.txt) and tells crawlers which paths they may or may not fetch. Every well-behaved crawler — Googlebot included — checks it before fetching anything else on your site.
Mistakes in it are dangerous: because it's the first gate, a single wrong Disallow rule can silently block search engines from your most important pages, or from the whole site.
Robots.txt blocks crawling — not indexing
This is the most misunderstood part of robots.txt. Blocking a URL in robots.txt stops crawling, not indexing: a blocked URL can still appear in search results if other sites link to it — just with no description, because Google never fetched the page.
To keep a page out of search results, use a noindex meta tag and let Google crawl it to see the tag. Never combine noindex with a robots.txt block, because Google can't read a tag on a page it isn't allowed to fetch.
What belongs in robots.txt
Good uses of robots.txt:
- Blocking genuinely useless URL spaces — faceted-filter combinations, internal search results, session-ID URLs — to protect crawl budget
- Pointing crawlers at your XML sitemap with a Sitemap: line
- Keeping crawlers out of admin areas and infrastructure paths
Common robots.txt mistakes
The errors we see most often in site audits:
- Disallow: / left over from staging, blocking the entire site
- Blocking CSS or JavaScript files that Google needs to render pages
- Using robots.txt to 'hide' pages that should use noindex instead
- Combining a robots.txt block with a noindex tag (the tag can never be seen)
- Forgetting the file is public — never list secret URLs in it
How to check your robots.txt
Fetch yoursite.com/robots.txt in a browser and read every Disallow line, asking: do I really want crawlers kept out of this path? Then verify important pages are crawlable with the URL Inspection tool in Google Search Console. A full site crawl also reveals exactly which pages crawlers can and cannot reach under your current rules.