Noindex in robots.txt: why it fails and what actually hides SaaS pages from AI answers

Noindex in robots.txt does not work. Google has never supported that syntax, and blocking a URL in robots.txt stops crawlers from seeing a real noindex tag on the page. To remove a page from Google, and from the AI engines reading Google's index, use a meta robots tag or an X-Robots-Tag header and let the page be crawled.

Does noindex work in robots.txt?

No. Google's own documentation says the noindex rule is not supported when you put it in robots.txt.

The failure is worse than a no-op. The usual pattern looks like this:

User-agent: *
Disallow: /pricing-internal/
Noindex: /pricing-internal/

The Noindex: line does nothing. The Disallow: line then creates a second problem: if robots.txt blocks a URL from Google's crawlers, they cannot see the noindex tag on that page even if you later add a correct one in the HTML. You have locked the door and left the instructions inside.

HubSpot's documentation makes the same point for its users: the noindex method should not be combined with the robots.txt method, because robots.txt prevents search engines from reading the noindex tag.

What is the difference between robots.txt and noindex?

They control two different stages, and conflating them is the root of nearly every robots.txt no index mistake I see on SaaS sites.

robots.txt controls crawling. It tells a crawler which URLs it may fetch. The robots meta tag influences indexing and other behavior, which is a decision made after the page has been fetched and read.

So a blocked page is a page nobody fetched. That does not mean it is out of the index. A page blocked by robots.txt may still be indexed if other sources link to it. Google can list a URL it has never read, with a title pulled from anchor text and no description. Founders see that bare listing in search results, assume the block failed, and add more Disallow lines. The block is working. It was the wrong tool.

The reverse is the combination that works: allow the crawl, serve the noindex. When Googlebot crawls a page and extracts a noindex tag or header, Google drops that page entirely from Search, regardless of whether other sites link to it.

How do you implement noindex correctly: meta tag vs HTTP header

There are two supported ways. Noindex can be implemented as a meta tag in the head section of a page, or as an HTTP response header.

The meta tag

In the <head>:

<meta name="robots" content="noindex">

That addresses every crawler that reads the directive. You can combine it with other rules, such as nofollow:

<meta name="robots" content="noindex, nofollow" />

If you only want Google out, and want other engines to keep the page, use the googlebot-specific tag:

<meta name="googlebot" content="noindex">

Make sure the tag is present in the HTML your server actually returns to the crawler, not only added after client-side rendering.

The HTTP header

An X-Robots-Tag HTTP header does the same job at the response level:

X-Robots-Tag: noindex

Google, Yahoo and Bing all support noindex in X-Robots-Tag, and it is the right choice for non-HTML content such as PDFs or images, where there is no <head> to put a tag in. For a SaaS site that means gated whitepapers, exported invoices, generated reports and anything served from a file endpoint.

Set it in nginx, in your CDN edge rules, or in the response from your app framework. One line in a middleware covers an entire route group, which is usually cleaner than templating a meta tag into twelve layouts.

Why is my page still showing up after adding noindex?

Work through these in order.

1. Is the URL blocked in robots.txt? If so, crawlers cannot see the noindex tag. Remove the Disallow line, keep the tag. This is the single most common cause.
2. Has Google recrawled the page? Google says a page still appearing after a noindex is added is probably because it has not been crawled since the rule went in. The noindex rule only takes effect after the robot revisits the page.
3. Is the tag actually in the response? Run curl -I for the header and curl -s | grep robots for the tag. Check what the server returns, not what your CMS preview shows.
4. Is something else linking to it? A page blocked by robots.txt can still be indexed from external links. Again, that is the block, not the tag.
5. Are you on a platform domain with its own rules? HubSpot, for example, always sets content on system domains containing hs-sites as no-index in a robots.txt file. Staging subdomains on other platforms often do something similar, which is fine until the production page inherits the config.

How long does it take Google to drop a noindexed page from search results?

There is no fixed number, and anyone quoting one is guessing. The timing is a function of recrawl frequency alone: the rule takes effect after the robot revisits the page, and until then nothing changes.

That means a page that gets crawled daily because it is linked from your main nav goes quickly. A deep help-center article nobody links to can sit for a long time. If a page is still in results, the most likely reason is simply that it has not been crawled since you made the change.

Two things speed it up. Keep the page crawlable (no Disallow), and request indexing for the URL in Search Console so the crawler is pointed at it rather than waiting its turn.

Does noindex stop AI answer engines like ChatGPT, Perplexity and Gemini from citing a page?

Partly, and unevenly.

The directive is widely respected. MDN describes noindex as used by all major crawlers. But Google also warns that some search engines might interpret the rule differently, so a page can still appear in another engine's results with noindex in place.

For AI answers the practical model is this. Engines that answer from live search results can only cite what their underlying index holds. Deindex the page and it stops being an available citation. Engines running their own crawler, or answering partly from model weights, may still surface the content or name your brand without linking the page.

How much citation surface each engine has varies a lot. In welaunch's test of 434 live AI answers to buyer questions across seven software categories (27 Sep 2026, US), Perplexity cited 10.1 sources per answer on average across 103 answers and gave no source at all in zero of them. Gemini averaged 9.2 sources across 110 answers, with no source in 11. ChatGPT averaged 5.8 across 110 answers, with no source in 7. Google AI Mode averaged 3.4 across 111 answers, with no source in 21.

So a deindexed page costs you the most on Perplexity, which cites heavily, and the least on Google AI Mode, which often answers with few sources or none.

How noindex methods and crawler behavior compare across engines

Engine rows: welaunch test of 434 live AI answers to buyer questions in seven software categories, 27 Sep 2026, US.

Can noindex keep content out of AI training data?

No, and it is worth being blunt about it. Noindex is an indexing instruction. It tells a crawler not to put the page in a search index. It says nothing about whether the content was already copied into a training corpus, and it does not retroactively remove anything from a model that has already been trained.

If training is your concern, that is a separate control surface: crawler-specific user-agent rules in robots.txt for the AI crawlers that publish one, plus whatever opt-out the vendor offers. Those block future fetches. A page that has been public for three years has already been read.

For genuinely private content, neither of these is the answer. Put it behind authentication. An unlinked, noindexed, Disallowed URL is still a public URL.

What should actually be in a robots.txt file for a SaaS site?

robots.txt is a crawl-budget and bandwidth tool. To reduce bandwidth it is more effective than indexing rules, because it blocks resources from being crawled at all.

Good uses on a SaaS site:

• Faceted search and filter URLs that generate endless combinations.
• Internal search result pages.
• Large asset directories that burn crawl budget without earning rankings.
• API endpoints and app routes behind login that still return a response.

The syntax for blocking a path is a Disallow line with the slug, in the form Disallow: /url-slug.

What should not be in it: anything you want removed from search results. That is the noindex tag's job, and the two should not be combined on the same URL.

Also worth auditing: platform defaults. HubSpot sets hs-sites system domain content to no-index in robots.txt automatically, and most hosting platforms ship a staging-domain rule of some kind. Check what your production robots.txt actually serves, not what your config file says it should.

If noindex removes a page, how do you still get a SaaS brand named in AI answers?

Fix the technical mistakes, then stop treating your own site as the whole game.

The engines mostly answer from third-party pages. In welaunch's 27 Sep 2026 test, 106 of 434 AI answers (24%) cited reddit.com. In the subset of questions about choosing an AI visibility tool, 22 of 80 answers cited reddit.com. And brand presence concentrated: Profound was named in 23 of 63 answers to AI-visibility buyer questions, with Otterly and Peec at 17 each.

None of those appearances depended on the vendor's own robots.txt. They came from being present on the comparison posts, forum threads and listings the engines were already pulling.

That is the work welaunch does. It asks your buyers' questions on ChatGPT, Perplexity, Gemini and Google AI several times a week, keeps every answer in full so you can read what each engine said, finds the pages those engines cite, and drafts the reply, listing or page that gets you onto one. Everything goes out from your own accounts: you review each draft and press submit. On Reddit and Quora it names the product with disclosure and does not stuff links.

How do AI answer engines differ in which sources they cite?

Enough that one strategy per engine is a mistake.

Reliance on Reddit is the clearest split. In welaunch's test (27 Sep 2026, US), Google AI Mode cited reddit.com in 40% of answers (44 of 111), Perplexity in 31% (32 of 103), Gemini in 20% (22 of 110), and ChatGPT in just 6% (7 of 110).

Pair that with citation volume and the picture sharpens. Perplexity cites a lot of sources and leans on Reddit moderately, so breadth of presence pays off there. Google AI Mode cites few sources but picks Reddit for a large share of them, so a single strong thread carries disproportionate weight. ChatGPT cites a middling number and rarely reaches for Reddit, which means your presence has to be somewhere else entirely.

Answers are also unstable run to run. In the same test, only 9 of 32 brand appearances (28%) in ChatGPT answers were named in all 4 runs of the same prompt, and 6 of 21 (29%) in Perplexity. Seeing your name once proves very little. That is the argument for checking repeatedly rather than screenshotting a good answer and calling it done.

FAQ

What is robots noindex?

It is a directive telling search engines not to include a page in their index. It is delivered either as a meta tag in the page head or as an X-Robots-Tag HTTP header, never as a line in robots.txt. When Googlebot crawls the page and reads it, Google drops the page entirely from Search even if other sites link to it.

Can robots.txt block indexing?

Not reliably. robots.txt controls crawling, not indexing, and a blocked page may still be indexed if other sites link to it. Worse, blocking the URL means crawlers cannot see a noindex tag you add later. Use noindex for index removal and leave the page crawlable.

What should be in a robots.txt file?

Crawl control: faceted URLs, internal search pages, heavy asset directories, app routes that waste crawl budget. Use the Disallow: /url-slug form. Blocking a path is also the more effective way to reduce bandwidth, since the resource is never fetched. Keep index-removal decisions out of it.

Does a canonical tag pointing at the homepage also deindex a page?

In effect, often yes. A canonical pointing elsewhere tells search engines the target URL is the one to index, so the page naming that canonical can drop out of results. Check the canonical in the rendered source of a few deep pages, not just the homepage.

Which AI systems respect noindex and which ignore it?

MDN describes noindex as used by all major crawlers, but Google notes that some search engines interpret the rule differently and a page may still appear elsewhere. Engines answering from live search results stop citing a page once it leaves the index they query. Engines using their own crawler or model weights may still reference the content.

Is the noindex meta tag the same as nofollow?

No. noindex keeps a page out of the index; nofollow concerns the links on it. They are separate rules that can be combined in one tag, as in <meta name="robots" content="noindex, nofollow" />. Using nofollow alone will not remove a page from search results.

Sources

• Block Search Indexing with noindex | Google Search Central | Documentation | Google for Developers
• Prevent content from appearing in search results
• <meta name="robots"> HTML attribute value - HTML | MDN
• welaunch test of live AI answers, 27 Sep 2026
• welaunch's test of 434 live AI answers to buyer questions in seven software categories, 27 Sep 2026, US

Related on welaunch

• Lovable SEO: why AI crawlers see a blank page on your site, and the fix
• Is SEO dead? What AI answers changed and what they did not
• SaaS marketing in 2026: the channels that still work for a small team