Oct 23, 2026 ·Alex Mercer

Most ecommerce and SaaS sites should keep every AI search crawler allowed, and block only the training crawlers if they object to model training. The mistake we see most often at OnyxRank is a blanket "block all AI bots" rule copied from a news publisher's playbook. It trains nobody and quietly removes the brand from ChatGPT search answers, Claude search results and Perplexity.

The reason is that AI companies now run separate crawlers for separate jobs. Blocking the wrong one costs you visibility. Blocking the right one costs you nothing you were using. This guide covers what each major crawler does according to the vendors' own documentation, a decision framework for store and software founders, a copy-ready robots.txt, and the checks that catch the common failures.

The core distinction: training, search, and user-triggered fetches

Every major AI vendor has split its crawling into at least two purposes, and some have three. If you treat them as one thing, you will make the wrong call.

1. **Training crawlers** collect content that may be used to build future models.

2. **Search crawlers** index pages so the product can cite and link to them in answers.

3. **User-triggered fetchers** visit a page because a person asked a question about it.

OpenAI states this split plainly in its [crawler documentation](https://platform.openai.com/docs/bots). GPTBot is for content that may be used to train generative AI foundation models. OAI-SearchBot surfaces sites in ChatGPT's search features. ChatGPT-User handles actions a user initiates. The same page says each setting is independent of the others, so disallowing GPTBot does not remove you from ChatGPT search.

That independence is the whole opportunity. You can opt out of training and stay visible in answers.

What each crawler does, per the vendors

OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User

Per [OpenAI's documentation](https://platform.openai.com/docs/bots), disallowing GPTBot signals that your content should not be used in training. Sites that opt out of OAI-SearchBot "will not be shown in ChatGPT search answers, though can still appear as navigational links." OpenAI notes it can take about 24 hours after a robots.txt update for its search systems to adjust.

ChatGPT-User is different. OpenAI says that because these actions are user-initiated, robots.txt rules may not apply. It is also not used to decide whether content appears in search. Do not expect a robots.txt line to fully control it.

Anthropic: ClaudeBot, Claude-User, Claude-SearchBot

[Anthropic's help center](https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) lists three bots. ClaudeBot collects web content that could contribute to training. Claude-User accesses sites when Claude users ask questions. Claude-SearchBot crawls to improve search result quality. Anthropic says that disabling Claude-SearchBot means your content is not indexed for search, which may reduce visibility and accuracy in results.

Two practical details from the same page. Anthropic supports the non-standard Crawl-delay directive, which helps if a bot is hammering a large catalog. And it advises against blocking by IP address, because that can stop the bot from reading your robots.txt at all, which makes the opt-out unreliable.

Google: Googlebot and Google-Extended

Google handles this differently. [Google's AI features documentation](https://developers.google.com/search/docs/appearance/ai-features) says a page must be indexed and eligible to appear in Search with a snippet to be a supporting link in AI Overviews or AI Mode, and that there are "no additional technical requirements." No special schema or AI text file is needed. Googlebot rules are the control for Search access.

Google-Extended is a separate product token. According to [Google's common crawlers page](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers), it lets publishers manage whether crawled content may be used to train Gemini models, and it "does not impact a site's inclusion in Google Search." That means a Google-Extended disallow is not an AI Overviews opt-out. If you want to limit what Search shows from a page, Google points to controls like nosnippet, data-nosnippet, max-snippet and noindex, all of which also reduce your organic visibility.

Perplexity: PerplexityBot and Perplexity-User

[Perplexity's bot documentation](https://docs.perplexity.ai/guides/bots) says PerplexityBot surfaces and links sites in Perplexity search results and is not used to crawl content for AI foundation models. To appear in results, sites should allow it in robots.txt and permit its published IP ranges. Perplexity-User handles user-initiated fetches and, because a person triggers it, generally ignores robots.txt. Changes can take up to 24 hours to apply.

Note that the IP point matters for stores behind Cloudflare, Akamai or a custom WAF. Allowing PerplexityBot in robots.txt while your firewall challenges its requests accomplishes nothing.

The decision framework for ecommerce and SaaS founders

Ask two separate questions. Never merge them.

**Question 1: Do I want to be cited and linked in AI answers?** For a store or software company trying to be found by buyers, almost always yes. Allow OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot. These are the crawlers that determine whether you can be recommended. Our post on [measuring AI crawler activity in your server logs](/blog/ai-crawler-log-analysis-geo-optimization-2026) shows how to confirm they are actually reaching your pages.

**Question 2: Am I comfortable with my content being used for model training?** This is a business and legal judgment, not an SEO one. Reasonable positions differ:

• A SaaS company whose docs and comparison pages exist to be read widely may be happy to be included in training, since being part of a model's background knowledge can help brand recall.
• An ecommerce brand with proprietary photography, original buying guides or paid research may prefer to opt out of GPTBot, ClaudeBot and Google-Extended.

Neither choice affects your ranking in Google Search, since Google states Google-Extended does not influence inclusion in Search. We cannot promise a training opt-out has zero effect on how models describe your brand over time. No vendor publishes data that settles it, so treat claims in either direction with caution.

What not to block

Do not block by wildcard. A rule such as `User-agent: *` with `Disallow: /` takes you out of everything. Do not block CSS, JavaScript or product feed paths that rendering depends on. And do not block your product and pricing pages while expecting AI tools to recommend you. The pages that win citations are the commercial ones: category pages, product pages, pricing, integrations and comparison content.

A copy-ready robots.txt

This version allows all search and retrieval crawlers and opts out of training crawlers. Remove the training block if you decide to allow training.

```

Search and answer crawlers: allowed

User-agent: OAI-SearchBot

Allow: /

User-agent: Claude-SearchBot

Allow: /

User-agent: PerplexityBot

Allow: /

User-agent: Googlebot

Allow: /

Training crawlers: opted out

User-agent: GPTBot

Disallow: /

User-agent: ClaudeBot

Disallow: /

User-agent: Google-Extended

Disallow: /

Everyone else: normal rules

User-agent: *

Disallow: /cart

Disallow: /checkout

Disallow: /account

Disallow: /*?sort=

Sitemap: https://www.example.com/sitemap.xml

```

Three notes on this file. First, a crawler follows the most specific user agent group that matches it, so named groups override the wildcard group; the rules for how matching works are in [Google's robots.txt guide](https://developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt) and the [RFC 9309 standard](https://www.rfc-editor.org/rfc/rfc9309). Second, repeat the file for each subdomain, since Anthropic notes opt-outs apply per host. Third, if you run a headless storefront or a separate docs subdomain, check that each has its own file.

For faceted filter URLs and crawl waste on large catalogs, keep your normal rules in the wildcard group and read our guide on [faceted navigation and crawl waste](/blog/faceted-navigation-seo-ecommerce-crawl-waste-fix-2026).

Five checks that catch most robots.txt failures

1. **Fetch the live file.** Open yourdomain.com/robots.txt in a browser. CMS plugins, CDNs and staging deploys regularly overwrite it. Confirm it is the file you think it is and that it returns a 200.

2. **Test with a real user agent.** Run `curl -A "OAI-SearchBot" -I https://yourdomain.com/` and the same for each search bot. A 403 or a challenge page means your firewall is blocking what robots.txt allows.

3. **Check your bot protection settings.** Many CDN and security tools now ship toggles that block "AI crawlers" in one click. Verify which bots that toggle covers before enabling it on a store that wants AI visibility.

4. **Review server logs after 24 to 48 hours.** Both OpenAI and Perplexity say changes can take about 24 hours. Look for the search bots requesting your key commercial URLs.

5. **Re-check after every platform migration or theme change.** A forgotten `Disallow: /` from staging is the most expensive one line in SEO.

What robots.txt will not fix

Allowing the right crawlers makes you eligible. It does not make you cited. Google is explicit that there are no extra technical requirements for AI Overviews beyond being indexed and snippet eligible, and that structured data is not a special lever there. Citation depends on content that answers the query better than alternatives, clear product and pricing information, and pages a crawler can actually read. If your key content only appears after heavy client side rendering, see our write-up on [the JavaScript rendering gap for AI crawlers](/blog/javascript-rendering-gap-ai-crawlers-geo-2026). And on the often overhyped side of AI-specific files, we covered [what the evidence says about llms.txt](/blog/llms-txt-geo-optimization-ai-overviews-2026).

FAQ

Should I block GPTBot to stay out of ChatGPT?

No. GPTBot relates to model training. ChatGPT search results depend on OAI-SearchBot, and OpenAI says the settings are independent. Block GPTBot only if you want to opt out of training.

Does blocking Google-Extended remove me from AI Overviews?

No. Google states Google-Extended does not affect inclusion in Google Search, and AI Overviews draw on pages that are indexed and snippet eligible. Controls such as nosnippet or noindex limit that, at the cost of organic visibility.

Will ChatGPT-User or Perplexity-User obey my robots.txt?

Not reliably. Both vendors say these fetchers act on a person's request, so robots.txt rules may not apply. If a page must not be fetched, use authentication rather than relying on robots.txt.

How long until a robots.txt change takes effect?

OpenAI and Perplexity both say about 24 hours for their search systems. Plan for a day before judging results.

Does robots.txt keep a page out of search results?

Not on its own. It controls crawling, not indexing, so a blocked URL can still appear if other pages link to it. Use noindex on a crawlable page to keep it out of results.

Takeaways

• Treat training, search and user-triggered crawlers as three separate decisions.
• Allow OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot if you want to be recommended by AI tools.
• Training opt-outs (GPTBot, ClaudeBot, Google-Extended) are a business choice and do not change your Google Search inclusion.
• Verify with curl and logs, because a firewall can override a permissive robots.txt.

If you want someone to check your robots.txt, CDN rules and rendered pages against these crawlers, run the [OnyxRank free SEO audit](/free-audit). If you are weighing outside help on AI search visibility, our [pricing plans](/pricing) show what an engagement includes.