To let AI answer engines read and cite your site, allow each vendor's search crawler and user fetcher in robots.txt, and decide separately about the training crawlers. The tokens that matter in 2026 are OAI-SearchBot, ChatGPT-User and GPTBot at OpenAI, Claude-SearchBot, Claude-User and ClaudeBot at Anthropic, PerplexityBot and Perplexity-User at Perplexity, and Googlebot plus the Google-Extended control at Google, with Applebot-Extended, Meta-ExternalAgent, Amazonbot and CCBot covering the rest. Every token below comes from the vendor's own documentation.
Which AI crawlers exist and what does each one do?
| Company | Token | Documented purpose | Honors robots.txt |
|---|---|---|---|
| OpenAI | OAI-SearchBot | "surface websites in search results in ChatGPT's search features" | Yes |
| OpenAI | GPTBot | "crawl content that may be used in training our generative AI foundation models" | Yes |
| OpenAI | ChatGPT-User | user actions in ChatGPT and Custom GPTs | "robots.txt rules may not apply" |
| Anthropic | Claude-SearchBot | "improve search result quality for users" | Yes |
| Anthropic | ClaudeBot | collecting content "that could potentially contribute to their training" | Yes |
| Anthropic | Claude-User | fetches pages when "individuals ask questions to Claude" | Yes |
| Perplexity | PerplexityBot | "surface and link websites in search results on Perplexity" | Yes |
| Perplexity | Perplexity-User | visits pages to answer a user question | "generally ignores robots.txt rules" |
| Googlebot | Search, including AI Overviews and AI Mode | Yes | |
| Google-Extended | training and grounding for Gemini; not a crawler | Yes | |
| Apple | Applebot-Extended | opt out of training Apple's foundation models; not a crawler | Yes |
| Meta | Meta-ExternalAgent | "training foundation AI models or improving products by indexing content" | Yes |
| Amazon | Amazonbot | improve products; "may be used to train Amazon AI models" | Yes |
| Common Crawl | CCBot | open crawl archive used by researchers and model builders | Yes |
Sources: OpenAI, Anthropic, Perplexity, Google, Apple, Meta, Amazon, Common Crawl. Perplexity states PerplexityBot "is not used to crawl content for AI foundation models" and documents no separate training crawler.
Why does the split between search, training and user fetchers matter?
Because the three tokens from one company do different things, and the docs say so. OpenAI: "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models", while OAI-SearchBot is managed on its own (OpenAI). Anthropic, which updated its documentation on February 20, 2026 to list all three bots, says disabling Claude-SearchBot "may reduce your site's visibility and accuracy in user search results" and disabling Claude-User "may reduce your site's visibility for user-directed web search" (Anthropic, Search Engine Roundtable).
Google-Extended is a control, not a crawler: it decides whether content Googlebot already fetched may train Gemini models or ground answers in Gemini Apps and Vertex AI, and it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal" (Google). Applebot-Extended works the same way for Apple: blocking it opts you out of training while pages stay discoverable in Spotlight, Siri and Safari (Apple).
What does a robots.txt that lets AI search in look like?
This file allows every search crawler and user fetcher, opts out of the training crawlers, and keeps normal search untouched. Adjust the training section to your own decision.
# Normal web search
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
# AI search crawlers: allow, so answers can cite you
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# User fetchers: allow (they may fetch anyway when a person asks)
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-User
Allow: /
User-agent: Perplexity-User
Allow: /
# Training crawlers and controls: your choice
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: CCBot
Disallow: /
# Everyone else
User-agent: *
Allow: /
Disallow: /admin/
Sitemap: https://www.example.com/sitemap.xml
Two rules of the format decide whether this works. Crawlers pick "the group with the most specific user agent that matches", and only one group applies to a given crawler, so a bot with its own group ignores the User-agent: * group entirely; if you disallow paths under *, repeat them in every named group that should honor them. Token names are case-insensitive, but path values are case-sensitive, and in a conflict Google "uses the least restrictive rule" (Google robots.txt specification).
What can robots.txt not do?
It cannot hide a page. Google's documentation says robots.txt "is not a mechanism for keeping a web page out of Google", and a disallowed page "can still be indexed if linked to from other sites" (Google). Use noindex or authentication for that.
It cannot stop every fetch. OpenAI says of ChatGPT-User that "Because these actions are initiated by a user, robots.txt rules may not apply" (OpenAI); Perplexity-User "generally ignores robots.txt rules" (Perplexity); and Meta-ExternalFetcher "may bypass robots.txt because it performs fetches that were requested by the user" (Meta). If you need a hard block, do it at the network layer using the published IP lists: OpenAI and Perplexity both publish JSON files of their crawler addresses (OpenAI, Perplexity).
It does not govern Google's AI features or Bing's. For AI Overviews and AI Mode, Google offers the Search generative AI control in Search Console, plus nosnippet, data-nosnippet and max-snippet (Google, Search Console Help). Microsoft uses meta tags: NOCACHE limits Copilot to your URL, title and snippet, and NOARCHIVE keeps content out of its answers and out of training, while both keep the page in ordinary Bing results (Microsoft Bing).
Should you block the training crawlers?
It depends on what your content earns you. Cloudflare's 2025 review found GPTBot, ClaudeBot and CCBot were the most frequently blocked crawlers with full-site disallow rules, while explicit allow directives for Google-Extended tripled over the year. The same review put crawl-to-refer ratios at up to 500,000:1 for Anthropic, up to 3,700:1 for OpenAI, generally below 400:1 for Perplexity, and up to 30:1 for Google (Cloudflare). Training crawls send almost nothing back; search crawls send citations.
Our default at Qubba Marketing is the file above: search and user fetchers open, training closed unless a partner has a reason to feed models, such as wanting its documentation to be part of what a model already knows. Either way, review the file every quarter, because the tokens change: Anthropic's earlier Claude-Web and Anthropic-AI agents are gone, and OAI-SearchBot did not exist when most AI blocks were written.
How do you check it is working?
Fetch your own /robots.txt and read it group by group as a crawler would: find the most specific matching token, apply only that group. Then look at server logs for the tokens above and confirm the search crawlers are getting 200 responses, not 403s from a firewall or bot manager that never read your robots.txt. Verify suspicious traffic against the vendors' IP lists rather than the user agent string, which anyone can forge. Finally, check the Search Console generative AI report and your own citation tests a few weeks later, because the point of the file is not a clean log, it is being cited.
Frequently asked questions
Does blocking GPTBot remove my site from ChatGPT search?
No.
OpenAI documents GPTBot as the training crawler and OAI-SearchBot as the crawler that surfaces sites in ChatGPT's search features, each with its own robots.txt token.
Disallowing GPTBot signals that your content should not train models.
To stay citable, allow OAI-SearchBot explicitly and make sure no wildcard rule or firewall blocks it.
Does Google-Extended block AI Overviews?
No.
Google-Extended controls whether content is used to train Gemini models and to ground answers in Gemini Apps and Vertex AI, and Google says it does not affect inclusion or ranking in Search.
AI Overviews and AI Mode are governed by the Search generative AI control in Search Console and by snippet directives such as nosnippet and max-snippet.
Why does a bot still fetch my page after I disallowed it?
Three common reasons.
The request came from a user fetcher such as ChatGPT-User, Perplexity-User or Meta-ExternalFetcher, which the vendors say may not honor robots.txt.
Or the bot matched a different group than you intended, since only the most specific group applies.
Or the user agent was forged, which you can check against the published IP lists.
Do I need an llms.txt file as well?
Not for search visibility.
Google states that it does not use such files and that they neither harm nor help visibility or rankings, and no other major engine documents reading them.
Robots.txt controls access; llms.txt is at most a courtesy summary for tools that choose to read it.
Spend the time on crawlability and clear answers instead.