12 AI Search Bots Every Brand Should Know in 2026
Kenson HuiThe exact robots.txt tokens that control AI search visibility • A reference for marketers and brand teams
TLDR: Your robots.txt file is not just a technical text file. It decides whether your brand shows up when someone asks ChatGPT, Perplexity, Claude, or Gemini a question about your category. Twelve AI crawler bots now control three distinct jobs: collecting training data, indexing content for search, and fetching pages on behalf of live users. Blocking the wrong one removes you from AI answers. Allowing the wrong one lets companies train models on your content for free. This guide names every bot that matters, explains what each one actually does, and gives you the exact robots.txt tokens to use.
Table of Contents
- The Three Jobs AI Bots Perform
- Why robots.txt Is a Brand Decision Now
- The 12 Bots You Need to Know
- The Quick-Reference Table
- Training Bots vs. Search Bots: The Distinction That Matters
- What to Allow, What to Block
- Copy-Paste robots.txt Rules for the Top 12
- Frequently Asked Questions
1. The Three Jobs AI Bots Perform
Most people hear "AI crawler" and picture one thing: a bot scraping data to train a model. That is one of 3 jobs, and it is not the one that decides whether you appear in AI answers.
Job 1: Training.
A crawler collects public web pages that may feed into future model training. Blocking this job stops your content from being used to improve a company’s models. It does not affect whether your site appears in that company’s AI search results today.
Job 2: Search indexing.
A crawler builds or updates a search index that the AI draws from when answering questions. Blocking this job removes your site from AI search results. Someone asks a question, the AI cannot find your page, and you are out of the answer.
Job 3: User fetch.
A bot retrieves a specific page in real time because a person asked an AI assistant a question and your page was a candidate answer. Blocking this job prevents the AI from reading your current page content, even if it appeared in the search index.
These jobs are independent. OpenAI, Anthropic, and Perplexity all run separate crawlers for each. That means you can allow search indexing while blocking training. You can allow training while blocking user fetches. The choice is granular if you use it correctly.
2. Why robots.txt Is a Brand Decision Now
Robots.txt used to be a server admin task. In 2026, it is a brand visibility decision that sits next to your content strategy.
More than half of consumers now use AI search. When someone asks ChatGPT "what is the best project management tool for a 50-person team," the answer either includes your brand or it does not. The brands that appear in those answers get consideration. The brands that do not exist in the retrieval pipeline get nothing.
Pierview’s tracking of 50,000+ real prompts across Perplexity, ChatGPT Search, and Gemini shows that AI engines cite only 2 to 7 domains per response. Getting cited is harder than ranking on page one of Google. And the traffic barely arrives anyway: about 70% of AI-referred traffic lands as Direct in GA4 with the referrer stripped.
The first gate is crawler access. If the search bot cannot fetch your page, no amount of content quality matters. The second gate is whether the content earns the citation. The first gate is the one you control with robots.txt.
3. The 12 Bots You Need to Know
These 12 bots come from five companies. Each one plays a distinct role in the pipeline between your content and an AI-generated answer. We picked these twelve based on documented impact on AI search visibility and brand control in 2026, not by counting every scraper or crawler on the web.
GPTBot
Company: OpenAI
Collects public web content that may train OpenAI’s generative AI foundation models. Disallowing GPTBot tells OpenAI your content should not be used for training.
OAI-SearchBot
Company: OpenAI
Indexes content for ChatGPT Search. If you want your pages to appear when someone uses ChatGPT’s search feature, this is the bot that needs access.
robots.txt token: OAI-SearchBot
ChatGPT-User
Company: OpenAI
Fetches a page when a ChatGPT user asks a question and the model decides to retrieve your content in real time. OpenAI says these actions are user-initiated, so robots.txt rules may not apply.
robots.txt token: ChatGPT-User
ClaudeBot
Company: Anthropic
Collects content that may train Claude’s models. Separate from Anthropic’s search and user-fetch crawlers.
robots.txt token: ClaudeBot
Claude-SearchBot
Company: Anthropic
Indexes content for Claude’s search features. When Claude searches the web to answer a question, this bot finds the pages it cites.
robots.txt token: Claude-SearchBot
Claude-User
Company: Anthropic
Retrieves a specific page after a Claude user asks a question. The request is triggered by a person, not by background crawling.
robots.txt token: Claude-User
PerplexityBot
Company: Perplexity
Crawls and indexes pages for Perplexity’s search results and answer generation. Perplexity states this bot is not used to train foundation models.
robots.txt token: PerplexityBot
Perplexity-User
Company: Perplexity
Fetches a page when a Perplexity user asks a question and the system retrieves your content in real time. Perplexity says this bot generally ignores robots.txt because each request begins with a user action.
robots.txt token: Perplexity-User
Googlebot
Company: Google
Indexes content for Google Search, including AI Overviews and AI-powered search features. This is the same Googlebot that has indexed the web for decades, now also feeding Google’s AI search surfaces.
robots.txt token: Googlebot
Google-Extended
Company: Google
Controls whether Google-crawled content may train future Gemini models and support certain grounding uses in Gemini products. It does not affect Google Search inclusion or rankings.
robots.txt token: Google-Extended
Applebot
Company: Apple
Indexes content for Siri, Spotlight, Safari Suggestions, and Apple Intelligence answers. With Apple’s AI features expanding through iOS, Applebot is the crawler feeding those surfaces.
robots.txt token: Applebot
Applebot-Extended
Company: Apple
Controls whether content collected by Applebot may be used to train Apple’s AI models. Blocking it prevents training use while leaving your pages eligible for Siri, Spotlight, and Safari search.
robots.txt token: Applebot-Extended
4. The Quick-Reference Table
| Company | Bot | Primary job | robots.txt token |
|---|---|---|---|
| OpenAI | GPTBot | Model training | GPTBot |
| OpenAI | OAI-SearchBot | ChatGPT Search indexing | OAI-SearchBot |
| OpenAI | ChatGPT-User | Live user page fetch | ChatGPT-User |
| Anthropic | ClaudeBot | Model training | ClaudeBot |
| Anthropic | Claude-SearchBot | Claude search indexing | Claude-SearchBot |
| Anthropic | Claude-User | Live user page fetch | Claude-User |
| Perplexity | PerplexityBot | Answer engine indexing | PerplexityBot |
| Perplexity | Perplexity-User | Live user page fetch | Perplexity-User |
| Googlebot | Search + AI Overviews | Googlebot | |
| Google-Extended | Gemini training/grounding | Google-Extended | |
| Apple | Applebot | Siri/Spotlight search | Applebot |
| Apple | Applebot-Extended | Apple model training | Applebot-Extended |
5. Training Bots vs. Search Bots
The most common mistake in practice is treating every AI bot as one thing. A copy-paste block list that blocks GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and Applebot-Extended together does not do what most site owners think it does.
When someone runs a blanket AI bot block, they typically block both training and search crawlers together. That removes their site from AI answers while achieving the narrower goal of stopping training. The fix is per-bot rules, not a single wildcard.
6. What to Allow, What to Block
Your decision depends on what you want:
If you want maximum AI visibility: Allow the search and user-fetch bots. Decide separately whether to allow the training crawlers.
If you want AI search visibility without training: Allow OAI-SearchBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, and Applebot. Block GPTBot, ClaudeBot, Google-Extended, and Applebot-Extended.
If you want to stop all AI use of your content: Block everything. Understand that this also removes you from AI search results, which means your competitors show up instead.
The pragmatic default for most brands: Allow the search bots. Block the training bots. This keeps you in AI answers while protecting your content from being used to train competing models. OpenAI supports per-bot granularity — OAI-SearchBot and GPTBot are governed independently, so you can allow one while blocking the other. Anthropic does the same with Claude-SearchBot vs. ClaudeBot.
7. Copy-Paste robots.txt Rules for the Top 12
# === OpenAI ===
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: GPTBot
Disallow: /
# === Anthropic ===
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: ClaudeBot
Disallow: /
# === Perplexity ===
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
# === Google ===
User-agent: Googlebot
Allow: /
User-agent: Google-Extended
Disallow: /
# === Apple ===
User-agent: Applebot
Allow: /
User-agent: Applebot-Extended
Disallow: /
This configuration keeps your site eligible for AI search across ChatGPT, Claude, Perplexity, Google Search, and Apple while blocking the training crawlers from OpenAI, Anthropic, and Apple. Google-Extended is the awkward exception: it combines Gemini training and grounding, so blocking it may also limit whether your content appears in Gemini Apps.
Apply these rules per host. If your content lives at docs.example.com, rules at example.com/robots.txt do not apply to the subdomain.
8. Frequently Asked Questions
Q: Should I block or allow AI crawlers?
A: Allow the search bots. Block the training bots. That is the default recommendation for brands that want to appear in AI answers without licensing their content for model training. The specific bots to allow are OAI-SearchBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, and Applebot. The ones to block are GPTBot, ClaudeBot, Google-Extended, and Applebot-Extended.
Q: Does blocking GPTBot stop ChatGPT from citing me?
A: No. GPTBot is the training crawler. OAI-SearchBot controls whether your pages appear in ChatGPT Search results. Blocking GPTBot while allowing OAI-SearchBot keeps you in ChatGPT answers while preventing training use.
Q: Does blocking Google-Extended hurt my Google rankings?
A: No. Google says Google-Extended does not affect inclusion or ranking in Google Search. It controls whether content supports Gemini training and certain grounding uses. Continue allowing Googlebot for Search and AI Overviews.
Q: How do I verify a crawler is legitimate?
A: Anyone can spoof a user-agent header, so a match in your logs does not guarantee the request came from the stated source. Verify by checking the source IP against the vendor’s published IP ranges. OpenAI publishes IPs at openai.com/gptbot.json and openai.com/searchbot.json. Perplexity publishes at perplexity.com/perplexitybot.json. Anthropic publishes IP ranges for all three bots. Google supports reverse-DNS verification.
Q: Can user-triggered bots ignore robots.txt?
A: Yes. OpenAI says robots.txt may not apply to ChatGPT-User. Perplexity says Perplexity-User generally ignores robots.txt because the request begins with a user action. Enforce blocking at the CDN, WAF, or application layer using the vendor’s verification data.
Q: Do I need separate rules for each bot?
A: Yes. OpenAI, Anthropic, and Perplexity each run multiple crawlers with independent roles. A single wildcard rule cannot express the difference between "allow search, block training." Explicit groups per user-agent make the policy readable and prevent one training decision from accidentally blocking a search agent.
Q: How long does it take for robots.txt changes to take effect?
A: OpenAI and Perplexity say changes may take about 24 hours to propagate. A successful update does not force an immediate recrawl or citation. The change takes effect on the next crawl cycle.
Q: What happens if I block everything?
A: You remove your site from all AI search results. When someone asks ChatGPT or Perplexity a question about your category, your brand will not appear in the answer. Your competitors who allow search bots will get the citation instead. Blocking everything is a valid licensing position but a poor visibility strategy.
