Technical
robots.txt vs. noindex: which to use, and when.
These two get confused constantly, and using the wrong one causes real problems, including pages that stay in Google's index when you wanted them gone. They do different jobs: robots.txt controls crawling; noindex controls indexing. Here's the difference and when to use each.
In short
robots.txt controls crawling; noindex controls indexing. To keep bots out of an area, use robots.txt. To remove a page from search results, use noindex and let Google crawl it. Blocking it in robots.txt only hides the very instruction you need Google to read.
Crawling vs. indexing. The core distinction
Crawling is Google reading a page; indexing is Google storing it to potentially rank. They're separate steps, and the two tools act on different ones. robots.txt says 'don't crawl this'; a noindex tag says 'you can crawl it, but don't index it.' Conflating them is the source of most mistakes.
robots.txt: control crawling
robots.txt tells crawlers which URLs not to fetch. Use it to keep bots out of areas that waste crawl budget: admin paths, faceted-filter URLs, internal search results. What it does not reliably do is remove a page from the index: if a blocked page has links pointing to it, Google can still index the URL without its content, which is why blocked pages sometimes show as 'indexed, though blocked by robots.txt.'
noindex: control indexing
A noindex meta tag or header tells Google to keep a page out of the index even though it can crawl it. Use it for pages you want accessible but not ranking: thank-you pages, thin tag pages, duplicate-ish utility pages. The catch: Google has to crawl the page to see the noindex, so you must not also block it in robots.txt, or Google never reads the instruction.
The classic mistake
Blocking a page in robots.txt to get it out of the index. It backfires: Google can't crawl the page, so it never sees the noindex you may have added, and the URL can linger in the index anyway. To remove a page from the index, use noindex and let Google crawl it; once it's dropped, you can block it in robots.txt if you want to save crawl budget.
At a glance
robots.txt vs. noindex, when to use which
| Goal | Use robots.txt | Use noindex |
|---|---|---|
| Keep a page out of Google's index | No. A blocked page can still be indexed if it's linked | Yes. The reliable way to keep it out |
| Save crawl budget on large sections | Yes | No. The page must be crawlable to read the tag |
| Remove a page that's already indexed | No. It can stay indexed | Yes. Let Google crawl it and drop it |
| Block a resource (script, image) | Yes | Not applicable |
| Protect private or staging content | No. It isn't access control | No. Use real authentication |
Related questions
Quick answers.
How do I remove a page from Google?
Add a noindex tag and make sure the page is crawlable so Google can see it; it drops out on the next crawl. For urgent removal, use the Removals tool in Search Console to hide it temporarily while the noindex takes effect. Don't block it in robots.txt first, or Google won't see the noindex.
Why is a page I blocked in robots.txt still in Google?
Because robots.txt blocks crawling, not indexing. If other pages link to the blocked URL, Google can index the URL itself without reading its content. Remove the robots block, add noindex, let it be crawled once, and it will drop.
Does robots.txt help with duplicate content?
Not really, and it can hurt. Blocking duplicate URLs stops Google crawling them but doesn't consolidate their signals; canonical tags do that. For duplication, reach for canonicals or redirects, not robots.txt.
Keep reading
Related guides.
Primary sources
The sources behind this guide.
See also
Related across the site.
Start here
Bring us the search problem. You’ll get a straight answer on fit.
Tell us what you sell, the markets you serve and the visibility problem you need solved. Prefer to start smaller? Ask for a free, no-obligation SEO audit and we’ll show you what’s holding the site back. Either way you get a straight answer on fit: ongoing SEO, a free audit, website-foundation work, or no engagement at all.