Treat AI crawlers as three groups: training crawlers, search crawlers, and fetchers that run when a user asks. If you want to appear in AI answers, don’t block the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot). Blocking training crawlers (GPTBot, ClaudeBot, Google-Extended) is a policy decision; OpenAI and Google both say their training controls are separate from search.
The three groups
| Group | Purpose | Main user agents |
|---|---|---|
| Training | Collect content that may be used to train models | GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google) |
| Search | Build the index behind AI search answers | OAI-SearchBot (OpenAI), Claude-SearchBot (Anthropic), PerplexityBot (Perplexity) |
| User-triggered | Fetch a page when a user asks the assistant to | ChatGPT-User (OpenAI), Claude-User (Anthropic), Perplexity-User (Perplexity) |
What each company says
OpenAI
- GPTBot crawls content that may be used to train generative AI foundation models. Disallowing it signals that your content should not be used for training.
- OAI-SearchBot surfaces sites in ChatGPT’s search features. Sites that opt out won’t be shown in ChatGPT search answers, though they can still appear as navigational links.
- ChatGPT-User acts on a user’s request and doesn’t crawl automatically. OpenAI says that, because these actions are user-initiated, robots.txt rules may not apply.
- Each setting is independent: you can allow OAI-SearchBot and disallow GPTBot.
Source: Overview of OpenAI Crawlers
Anthropic
- ClaudeBot collects content that could contribute to training. Blocking it excludes your future content from training datasets.
- Claude-SearchBot indexes content for search. Blocking it may reduce your visibility and accuracy in search results.
- Claude-User fetches pages when people ask Claude questions.
- Anthropic says all of its bots honor robots.txt — including Claude-User. That differs from OpenAI and Perplexity.
Source: Does Anthropic crawl data from the web? (April 7, 2026)
Perplexity
- PerplexityBot surfaces and links sites in Perplexity’s search results, and is not used to train foundation models.
- Perplexity-User supports user actions. Perplexity says that, since a user requested the fetch, it generally ignores robots.txt rules.
Source: Perplexity Crawlers
- Google-Extended controls whether content Google crawls may be used to train Gemini models and for grounding in Gemini Apps and Vertex AI. Google says it does not affect inclusion or ranking in Google Search.
- AI Overviews and AI Mode are part of Search. Google says robots.txt rules for Googlebot are how site owners manage access for Search, including its AI features.
Sources: Google’s common crawlers (updated 2026-07-14), AI features and your website
Example: allow search, block training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xml
Crawlers without a group of their own follow User-agent: * (RFC 9309). OpenAI and Perplexity say changes can take about 24 hours to take effect.
Check outside robots.txt too
A WAF or bot-protection service can block a crawler that robots.txt allows, and you won’t see it in the file. When TooruBot is blocked, Tooru reports the page as blocked rather than as a low score, so you can tell the difference.
How Tooru scores this
Allowing search and user-triggered crawlers earns points (OAI-SearchBot and PerplexityBot 2 each, Claude-SearchBot and Claude-User 1 each). Whether you block training crawlers is a policy choice and earns no points either way. Details are in the methodology.
Changelog
- 2026-10-09: Published