Robots.txt vs Noindex: Which to Use & When

Robots.txt vs noindex trips up almost everyone because the two controls feel interchangeable, but they govern completely different stages of how Google handles a page. Robots.txt controls crawling (whether a bot is allowed to fetch the URL at all). Noindex controls indexing (whether the page is allowed to appear in search results). Mix them up and you get the worst outcome: a page you wanted hidden that stays in Google forever.

If you have ever blocked a URL in robots.txt and watched it still show up in search, that single distinction explains why. This guide breaks down crawl versus index in plain terms, shows the one combination that quietly backfires, and gives you a decision table so you always pick the right tool.

Robots.txt vs Noindex: The Core Difference#

The fastest way to understand robots.txt vs noindex is to separate two questions Google asks about every URL. First, "Am I allowed to read this page?" That is crawling. Second, "Am I allowed to list this page in search results?" That is indexing. Robots.txt answers the first question. Noindex answers the second.

A Disallow rule in robots.txt tells crawlers not to fetch a URL. A noindex directive (in a meta tag or HTTP header) tells crawlers not to display the page in search, even though they were allowed to read it. They are not two flavors of the same thing. They operate at different doors.

Key mental model: robots.txt is a sign on the front door saying "do not come in." Noindex is a note inside the house saying "do not put me in the directory." If the bot never comes inside, it never reads the note.

That sentence is the entire reason the two get combined incorrectly, and it sets up the most damaging mistake in this whole topic.

What robots.txt actually does#

Robots.txt is a plain text file at the root of your domain (example.com/robots.txt). It lists user-agents and the paths they may or may not crawl. Its job is crawl management, not index management.

• It saves crawl budget by keeping bots out of low-value areas (faceted filters, internal search, staging paths).
• It does NOT remove a URL from Google's index on its own.
• It is a directive most reputable bots respect, but it is publicly visible to anyone.

The critical nuance: a Disallow only stops the fetch. Google can still index a URL it was told not to crawl if it discovers that URL through links elsewhere. You have seen the result: a search snippet that reads "No information is available for this page" with just the URL and no description. Google knows the page exists, it just was not allowed to read what is on it.

What noindex actually does#

Noindex is a directive that lives on the page itself, either as a meta tag in the HTML head or as an HTTP response header. It tells search engines: you may read this, but do not include it in your index.

<meta name="robots" content="noindex">

Or, for files that are not HTML, as an HTTP header:

X-Robots-Tag: noindex

Because the directive lives in the response, Google has to crawl the page to see it. Once it does and processes the noindex, the URL drops out of search results, usually within a few crawls. This is the reliable way to keep a page out of Google. It is also exactly why the next section matters so much.

The Mistake That Keeps Pages Indexed Forever#

Here is the gotcha most articles bury at the bottom, and it is the single most damaging error in robots.txt vs noindex decisions.

If you add noindex to a page AND also Disallow that same URL in robots.txt, the noindex stops working. Google obeys the robots.txt block, never crawls the page, and therefore never sees the noindex tag. The page can stay indexed indefinitely, with no way for Google to learn you wanted it gone.

It is the worst of both worlds. You did the work to hide the page, you placed the right directive, and the robots.txt rule silently cancels it. Google's own documentation flags this directly: a noindex directive is only honored if the page is crawlable.

Warning: never block a URL in robots.txt if that URL also carries a noindex tag you are relying on. The block prevents Google from ever reading the noindex. Remove the disallow, let Google crawl the page, see the noindex, and drop it. Then, once it is deindexed, you can decide whether to block crawling.

The correct sequence to deindex a page that is currently in Google:

1. Make sure the page is crawlable (no Disallow covering it).
2. Add noindex (meta tag or X-Robots-Tag header).
3. Wait for Google to recrawl and process the directive. You can request indexing in Search Console to speed this up.
4. Confirm it has dropped from the index (search site:yourdomain.com/the-url).
5. Only after it is gone should you consider adding a robots.txt block, and usually you do not even need to.

This is the serp-gap most guides skip, and it is why a clean robots.txt file matters. When you build rules with a free robots.txt generator, you can sanity-check that you are not accidentally disallowing a URL you are trying to noindex.

Crawl vs Index: A Decision Table#

Once you internalize crawl versus index, picking the right control is mechanical. Use this table.

Your goal	Use robots.txt Disallow	Use noindex	Why
Keep a page out of search results	No (alone it fails)	Yes	Only noindex reliably deindexes
Stop bots wasting crawl budget on junk URLs	Yes	No	Crawl control, page need not be hidden
Hide an already-indexed page	No (it blocks the noindex)	Yes, keep it crawlable	Google must read the noindex to act
Block a non-HTML file (PDF, image) from search	No	Yes, via X-Robots-Tag header	Files cannot hold a meta tag
Keep a staging or private area unreadable	Yes (plus auth)	Optional	Robots.txt is not security; add a password
Prevent a page from passing or receiving crawl	Yes	No	That is crawling, not indexing

The short rule: if your goal contains the word "search results," reach for noindex. If your goal is about bot access or crawl budget, reach for robots.txt. They overlap far less often than people assume.

When robots.txt is the right call#

• Infinite or near-infinite URL spaces: faceted navigation, calendar pages, internal site search results that generate endless combinations.
• Resource-heavy paths you never want crawled, like large export endpoints.
• Sections that are genuinely worthless to index and where you do not care if a bare URL ever surfaces.

In these cases you want to save crawl budget, and you accept that a stray URL might appear without a description. That trade is fine for junk you would never want ranked anyway.

When noindex is the right call#

• Thin or duplicate pages you want fully removed from search (tag archives, internal search pages, printer-friendly versions).
• Pages users should reach via direct link or navigation but that should never compete in the SERPs (thank-you pages, certain account pages).
• Anything currently indexed that you need to pull out cleanly.

For thin or duplicate content, noindex paired with a clean information architecture usually beats a blunt robots.txt block. If duplication is your real problem, a canonical tag may serve you better than either, and a quick pass with a schema and SEO markup generator helps keep your signals consistent across the page.

X-Robots-Tag: Noindex for Non-HTML Files#

A meta robots tag only works inside HTML. So how do you keep a PDF, image, or other non-HTML file out of search? You cannot add a meta tag to a PDF, and disallowing it in robots.txt brings back the same trap (Google may index the URL anyway and will never see any directive).

The answer is the X-Robots-Tag HTTP header. It carries the same directives as the meta robots tag, but it travels in the server response, so it works for any file type.

An Apache example that noindexes all PDFs:

<FilesMatch "\.pdf$">
  Header set X-Robots-Tag "noindex"
</FilesMatch>

An Nginx equivalent:

location ~* \.pdf$ {
  add_header X-Robots-Tag "noindex";
}

Because the header applies before any HTML parsing, it is the cleanest way to deindex documents, downloads, and media. The same crawlability rule still applies: do not Disallow the file in robots.txt, or Google will never fetch it to read the header.

Tip: the X-Robots-Tag also accepts nofollow, noarchive, and unavailable_after directives, so you can, for example, set a PDF to drop out of the index after a specific date. That granularity is impossible with robots.txt.

How to Verify Which One Is Working#

Guessing is how pages stay indexed by accident. Verify instead.

• Search site:yourdomain.com/path in Google. If the page shows with no description, it is likely blocked in robots.txt but still indexed (the classic mistake).
• In Google Search Console, use the URL Inspection tool. It tells you whether a page is "Indexed," whether crawling is "Allowed," and whether indexing is "Allowed," separating the two states for you.
• Check the Coverage and Page indexing reports for statuses like "Indexed, though blocked by robots.txt" (a red flag) and "Excluded by noindex tag" (working as intended).
• Fetch the page or file and inspect the response. For HTML, look in the head for the meta robots tag. For files, look at the response headers for X-Robots-Tag.

If Search Console says "Indexed, though blocked by robots.txt," that is your cue: the robots.txt block is fighting your intent. Remove the disallow, let Google crawl, and let the noindex (or a removal request) do its job.

Robots.txt vs Noindex: The Bottom Line#

The robots.txt vs noindex question has one answer worth memorizing: robots.txt controls whether bots can crawl a page, and noindex controls whether a page can appear in search. They solve different problems, and the only time they truly conflict is the one combination you must avoid. Never disallow a URL in robots.txt while relying on a noindex tag on that same URL, because the block prevents Google from ever seeing the noindex, and the page stays indexed.

Pick by intent. Want a page gone from search? Noindex, and keep it crawlable. Want to save crawl budget on junk URLs? Disallow in robots.txt and accept that a bare link might surface. Need to hide a PDF or image? Use the X-Robots-Tag header. When you are ready to build or audit your rules, a free robots.txt generator helps you avoid the disallow-blocks-noindex trap, and if you also need to keep crawlers like GPTBot out, our guide on how to block AI crawlers in robots.txt covers the directives that work in 2026. For pages you want indexed and ranking, a quick check with a meta tag generator keeps your titles and descriptions clean.

Frequently Asked Questions#

What is the difference between robots.txt and noindex? Robots.txt controls crawling (whether a bot may fetch a URL), while noindex controls indexing (whether a page may appear in search results). Robots.txt does not reliably remove a page from Google on its own; noindex does. They operate at two different stages, so they are not interchangeable.

Why does my page still show in Google after I blocked it in robots.txt? Because robots.txt only stops crawling, not indexing. If Google discovers the URL through links, it can index it without ever reading the page, which is why you see a bare URL with "No information is available for this page." To remove it, allow crawling and add a noindex tag, or use Search Console's removal tool.

Can I use robots.txt and noindex together? Not on the same URL when you are relying on the noindex. If you disallow a URL in robots.txt, Google cannot crawl it, so it never sees the noindex directive, and the page can stay indexed indefinitely. Keep the page crawlable until it is deindexed, then decide whether a robots.txt block is even needed.

How do I noindex a PDF or image? You cannot put a meta tag in a non-HTML file, so use the X-Robots-Tag: noindex HTTP header in your server config (Apache, Nginx, or your CDN). It carries the same directive as the meta robots tag but works for any file type. Just do not block the file in robots.txt, or Google will never fetch it to read the header.

Which should I use to keep a page out of search results? Use noindex, and make sure the page stays crawlable so Google can read the directive. Robots.txt alone is the wrong tool for this because it can leave the URL indexed. Verify with Search Console's URL Inspection tool, which reports crawling and indexing status separately.

Does noindex save crawl budget like robots.txt does? No. A noindexed page still gets crawled (it has to be, so Google can read the noindex), so it consumes crawl budget. If your only goal is to stop bots from wasting time on junk URLs, robots.txt is the right tool. Use noindex when the goal is keeping the page out of search results.