List of Top AI Search Crawlers + User Agents (October 2026)

This page lists the crawlers, fetchers and agents that AI companies use to request pages from websites, as of October 19, 2026. Each row gives the company, the user agent string the company publishes, the robots.txt token, what the bot does, what the company says about robots.txt, how to confirm a request is genuine, and a link to the company's own documentation. We checked each row against the operator's documentation on October 19, 2026. When an operator publishes nothing about a bot, the row says so.

Last checked: October 19, 2026. Previous update: November 20, 2025.

Terms used on this page

A user agent string is the text a client sends in the User-Agent header of each HTTP request to say what software it is. Your server logs record it.

A robots.txt token is the short name you write after User-agent: in your robots.txt file. For most bots the token is the bot's name, such as GPTBot. Under the robots.txt standard (RFC 9309), a token contains only letters, hyphens and underscores, so version numbers like /1.4 do not belong in robots.txt.

A training crawler collects pages that the company may use to train AI models.

A search crawler builds an index that an AI search or answer feature retrieves from.

A user-triggered fetcher requests a page because a person asked an AI assistant something that needs that page. Several operators say these fetchers may not follow robots.txt, because a person started the request.

An agent is software that browses and acts on websites for a person, such as filling a form or comparing products.

A token-only control is a robots.txt name that no crawler sends. Google-Extended and Applebot-Extended work this way. Google and Apple read your rules for these tokens and apply them to how they use content that Googlebot or Applebot already crawled.

AI bots grouped by purpose

Training crawlers include GPTBot (OpenAI), ClaudeBot (Anthropic), meta-externalagent (Meta), MistralAI-Training (Mistral), CCBot (Common Crawl), AI2Bot (Allen Institute for AI) and Amazonbot (Amazon). Google and Apple control training use through the Google-Extended and Applebot-Extended tokens instead of a separate crawler.

Search crawlers include OAI-SearchBot (OpenAI), Claude-SearchBot (Anthropic), PerplexityBot (Perplexity), MistralAI-Index (Mistral), meta-webindexer (Meta), Amzn-SearchBot (Amazon), DuckAssistBot (DuckDuckGo) and YouBot (You.com). Googlebot, bingbot and Applebot crawl for search engines whose results also supply AI features.

User-triggered fetchers include ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User, Amzn-User, meta-externalfetcher, Google-Agent and Google-GeminiNotebook.

Agents that sign their requests include OpenAI's Cloud browser and part of Google-Agent's traffic. Signing is explained under "How to confirm a bot is genuine."

Master table of AI crawlers, fetchers and agents

The "robots.txt, per the operator" column quotes or paraphrases what the operator's own page says. "No statement" means the operator's page does not say whether the bot follows robots.txt.

<div class="tbl-wrap"><table style="border-radius:12px;"><thead><tr><th>Company</th><th>Bot</th><th>User agent string (as published)</th><th>robots.txt token</th><th>Purpose</th><th>robots.txt, per the operator</th><th>How to verify</th><th>Operator docs</th></tr></thead><tbody><tr><td>OpenAI</td><td>GPTBot</td><td><code>Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot</code></td><td><code>GPTBot</code></td><td>Training</td><td>OpenAI says disallowing GPTBot indicates content should not be used to train its foundation models</td><td>IP list: openai.com/gptbot.json</td><td><a href="https://developers.openai.com/api/docs/bots">OpenAI crawlers</a></td></tr><tr><td>OpenAI</td><td>OAI-SearchBot</td><td><code>Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot</code></td><td><code>OAI-SearchBot</code></td><td>Search results in ChatGPT</td><td>Sites that opt out "will not be shown in ChatGPT search answers, though can still appear as navigational links"</td><td>IP list: openai.com/searchbot.json</td><td><a href="https://developers.openai.com/api/docs/bots">OpenAI crawlers</a></td></tr><tr><td>OpenAI</td><td>ChatGPT-User</td><td><code>Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot</code></td><td><code>ChatGPT-User</code></td><td>User-triggered fetch in ChatGPT, Custom GPTs and GPT Actions</td><td>"robots.txt rules may not apply"</td><td>IP list: openai.com/chatgpt-user.json</td><td><a href="https://developers.openai.com/api/docs/bots">OpenAI crawlers</a></td></tr><tr><td>OpenAI</td><td>OAI-AdsBot</td><td><code>Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot</code></td><td><code>OAI-AdsBot</code></td><td>Checks landing pages submitted as ChatGPT ads; not used for training</td><td>No statement</td><td>IP list: openai.com/adsbot.json</td><td><a href="https://developers.openai.com/api/docs/bots">OpenAI crawlers</a></td></tr><tr><td>OpenAI</td><td>Cloud browser (agent)</td><td>Not published</td><td>None</td><td>Agent that browses for a ChatGPT user</td><td>No statement</td><td>Web Bot Auth signature with <code>Signature-Agent: "https://chatgpt.com"</code></td><td><a href="https://help.openai.com/en/articles/11845367-chatgpt-works-cloud-browser-allowlisting">Cloud browser allowlisting</a></td></tr><tr><td>Anthropic</td><td>ClaudeBot</td><td>Not published (the user agent contains <code>ClaudeBot</code>)</td><td><code>ClaudeBot</code></td><td>Training</td><td>"Anthropic's Bots respect 'do not crawl' signals by honoring industry standard directives in robots.txt"</td><td>IP list: claude.com/crawling/bots.json</td><td><a href="https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler">Anthropic crawlers</a></td></tr><tr><td>Anthropic</td><td>Claude-SearchBot</td><td>Not published (the user agent contains <code>Claude-SearchBot</code>)</td><td><code>Claude-SearchBot</code></td><td>Search quality for Claude</td><td>Same statement as ClaudeBot</td><td>IP list: claude.com/crawling/bots.json</td><td><a href="https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler">Anthropic crawlers</a></td></tr><tr><td>Anthropic</td><td>Claude-User</td><td>Not published (the user agent contains <code>Claude-User</code>)</td><td><code>Claude-User</code></td><td>User-triggered fetch in Claude</td><td>Same statement as ClaudeBot</td><td>IP list: claude.com/crawling/bots.json</td><td><a href="https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler">Anthropic crawlers</a></td></tr><tr><td>Perplexity</td><td>PerplexityBot</td><td><code>Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)</code></td><td><code>PerplexityBot</code></td><td>Search results in Perplexity; "not used to crawl content for AI foundation models"</td><td>Perplexity documents robots.txt tags to manage it</td><td>IP list: perplexity.com/perplexitybot.json</td><td><a href="https://docs.perplexity.ai/guides/bots">Perplexity crawlers</a></td></tr><tr><td>Perplexity</td><td>Perplexity-User</td><td><code>Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)</code></td><td><code>Perplexity-User</code></td><td>User-triggered fetch</td><td>"this fetcher generally ignores robots.txt rules"</td><td>IP list: perplexity.com/perplexity-user.json</td><td><a href="https://docs.perplexity.ai/guides/bots">Perplexity crawlers</a></td></tr><tr><td>Google</td><td>Googlebot (smartphone)</td><td><code>Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)</code></td><td><code>Googlebot</code></td><td>Google Search, including AI Overviews and AI Mode</td><td>Common crawlers "always respect robots.txt rules for automatic crawls"</td><td>Reverse DNS to googlebot.com; IP list: common-crawlers.json</td><td><a href="https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers">Google common crawlers</a></td></tr><tr><td>Google</td><td>Googlebot (desktop)</td><td><code>Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36</code></td><td><code>Googlebot</code></td><td>Same as above</td><td>Same as above</td><td>Same as above</td><td><a href="https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers">Google common crawlers</a></td></tr><tr><td>Google</td><td>Google-Extended</td><td>None (token only)</td><td><code>Google-Extended</code></td><td>Controls use of crawled content for Gemini model training and grounding in Gemini Apps and Vertex AI</td><td>Rules for this token apply to content Googlebot crawls; it "does not impact a site's inclusion in Google Search"</td><td>Not applicable</td><td><a href="https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers">Google common crawlers</a></td></tr><tr><td>Google</td><td>GoogleOther</td><td><code>Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GoogleOther) Chrome/W.X.Y.Z Safari/537.36</code> (a mobile variant also exists)</td><td><code>GoogleOther</code></td><td>Generic crawls such as internal research and development; Google says it does not affect a specific product</td><td>Common crawler rule, as for Googlebot</td><td>Reverse DNS to googlebot.com</td><td><a href="https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers">Google common crawlers</a></td></tr><tr><td>Google</td><td>Google-CloudVertexBot</td><td>User agent contains <code>Google-CloudVertexBot</code></td><td><code>Google-CloudVertexBot</code></td><td>Crawls that site owners request for building Vertex AI Agents</td><td>Common crawler rule</td><td>Reverse DNS to googlebot.com</td><td><a href="https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers">Google common crawlers</a></td></tr><tr><td>Google</td><td>Google-Agent</td><td><code>Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Google-Agent; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-agent) Chrome/W.X.Y.Z Safari/537.36</code> (a mobile variant also exists)</td><td>None</td><td>Agents on Google infrastructure that browse and act on a user's request</td><td>User-triggered fetchers "generally ignore robots.txt rules"</td><td>IP list: user-triggered-agents.json; some requests signed with Web Bot Auth as <code>https://agent.bot.goog</code></td><td><a href="https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers">Google user-triggered fetchers</a></td></tr><tr><td>Google</td><td>Google-GeminiNotebook</td><td><code>Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36 (compatible; Google-GeminiNotebook; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-gemininotebook)</code> (a mobile variant also exists)</td><td>None</td><td>Fetches URLs that Gemini Notebook users add as sources (formerly Google-NotebookLM)</td><td>"generally ignore robots.txt rules"</td><td>IP lists for user-triggered fetchers</td><td><a href="https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers">Google user-triggered fetchers</a></td></tr><tr><td>Microsoft</td><td>bingbot</td><td><code>Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36</code> (mobile and short variants also exist)</td><td><code>bingbot</code></td><td>Bing search, which also supplies Copilot and Bing grounding results</td><td>Bing says robots.txt "can be configured to tell Bing crawlers how to interact with your website"</td><td>Verify Bingbot tool; IP list: bing.com/toolbox/bingbot.json</td><td><a href="https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0">Bing crawlers</a></td></tr><tr><td>Apple</td><td>Applebot</td><td><code>Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)</code> (an iPhone variant also exists)</td><td><code>Applebot</code></td><td>Search for Spotlight, Siri and Safari; Apple says the data may also help train its foundation models</td><td>"Applebot respects standard robots.txt directives"; it follows Googlebot rules when robots.txt does not name Applebot</td><td>Reverse DNS to applebot.apple.com; IP list: search.developer.apple.com/applebot.json</td><td><a href="https://support.apple.com/en-us/119829">About Applebot</a></td></tr><tr><td>Apple</td><td>Applebot-Extended</td><td>None (token only)</td><td><code>Applebot-Extended</code></td><td>Controls use of Applebot-crawled content to train Apple's foundation models</td><td>Disallowing it does not remove pages from Apple's search results</td><td>Not applicable</td><td><a href="https://support.apple.com/en-us/119829">About Applebot</a></td></tr><tr><td>Meta</td><td>meta-externalagent</td><td><code>meta-externalagent/1.1 (+/documentation/sharing/webmasters/web-crawlers)</code> or <code>meta-externalagent/1.1</code></td><td><code>meta-externalagent</code></td><td>Training foundation AI models or improving products by indexing content</td><td>Meta: "In order to block these crawlers, add a disallow for the relevant crawler to robots.txt"</td><td>Meta's page lists no IP ranges</td><td><a href="https://developers.facebook.com/docs/sharing/webmasters/web-crawlers">Meta web crawlers</a></td></tr><tr><td>Meta</td><td>meta-webindexer</td><td><code>meta-webindexer/1.1 (+/documentation/sharing/webmasters/web-crawlers)</code> or <code>meta-webindexer/1.1</code></td><td><code>meta-webindexer</code></td><td>Search quality for Meta AI; helps Meta AI cite and link to content</td><td>Same robots.txt statement as meta-externalagent</td><td>Meta's page lists no IP ranges</td><td><a href="https://developers.facebook.com/docs/sharing/webmasters/web-crawlers">Meta web crawlers</a></td></tr><tr><td>Meta</td><td>meta-externalfetcher</td><td><code>meta-externalfetcher/1.1 (+/documentation/sharing/webmasters/web-crawlers)</code> or <code>meta-externalfetcher/1.1</code></td><td><code>meta-externalfetcher</code></td><td>User-requested fetches, including agentic AI</td><td>"this crawler may bypass robots.txt rules"</td><td>Meta's page lists no IP ranges</td><td><a href="https://developers.facebook.com/docs/sharing/webmasters/web-crawlers">Meta web crawlers</a></td></tr><tr><td>Meta</td><td>facebookexternalhit</td><td><code>facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)</code></td><td><code>facebookexternalhit</code></td><td>Link previews for shared links</td><td>"might bypass robots.txt when performing security or integrity checks"</td><td>Meta's page lists no IP ranges</td><td><a href="https://developers.facebook.com/docs/sharing/webmasters/web-crawlers">Meta web crawlers</a></td></tr><tr><td>Amazon</td><td>Amazonbot</td><td><code>Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36</code></td><td><code>Amazonbot</code></td><td>Improves Amazon products and services; "may be used to train Amazon AI models"</td><td>Automated crawling "respects the Robots Exclusion Protocol"</td><td>IP page: developer.amazon.com/amazonbot/ip-addresses</td><td><a href="https://developer.amazon.com/amazonbot">Amazonbot</a></td></tr><tr><td>Amazon</td><td>Amzn-SearchBot</td><td><code>Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/0.1) Chrome/W.X.Y.Z Safari/537.36</code></td><td><code>Amzn-SearchBot</code></td><td>Search experiences such as Alexa; not used for model training</td><td>Respects the Robots Exclusion Protocol; follows other search bots' rules when not named</td><td>IP page: developer.amazon.com/amazonbot/searchbot-ip-addresses</td><td><a href="https://developer.amazon.com/amazonbot">Amazonbot</a></td></tr><tr><td>Amazon</td><td>Amzn-User</td><td><code>Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-User/0.1) Chrome/W.X.Y.Z Safari/537.36</code></td><td><code>Amzn-User</code></td><td>User-triggered fetch, such as an Alexa question</td><td>"may not follow all robots.txt directives"</td><td>IP page: developer.amazon.com/amazonbot/live-ip-addresses</td><td><a href="https://developer.amazon.com/amazonbot">Amazonbot</a></td></tr><tr><td>Mistral</td><td>MistralAI-User</td><td><code>Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-User/1.0; +https://docs.mistral.ai/robots)</code></td><td><code>MistralAI-User</code></td><td>User-triggered fetch in Mistral's Vibe assistant</td><td>Mistral documents robots.txt tags; no further statement</td><td>IP list: mistral.ai/mistralai-user-ips.json</td><td><a href="https://docs.mistral.ai/robots">Mistral crawlers</a></td></tr><tr><td>Mistral</td><td>MistralAI-Index</td><td><code>Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-Index/1.0; +https://docs.mistral.ai/robots)</code></td><td><code>MistralAI-Index</code></td><td>Index for Mistral search; not used for training</td><td>Mistral documents robots.txt tags; no further statement</td><td>IP list: mistral.ai/mistralai-index-ips.json</td><td><a href="https://docs.mistral.ai/robots">Mistral crawlers</a></td></tr><tr><td>Mistral</td><td>MistralAI-Training</td><td><code>Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-Training/1.0; +https://docs.mistral.ai/robots)</code></td><td><code>MistralAI-Training</code></td><td>Training datasets for Mistral models</td><td>"Webmasters can disallow this user agent in their robots.txt file"</td><td>No IP list linked</td><td><a href="https://docs.mistral.ai/robots">Mistral crawlers</a></td></tr><tr><td>DuckDuckGo</td><td>DuckAssistBot</td><td>Published as <code>DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html)</code></td><td><code>DuckAssistBot</code></td><td>Real-time crawl for DuckDuckGo's AI-assisted answers; "not used in any way to train AI models"</td><td>A disallow takes effect after 72 hours</td><td>IP list: duckduckgo.com/duckassistbot.json</td><td><a href="https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot">DuckAssistBot</a></td></tr><tr><td>You.com</td><td>YouBot</td><td><code>Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; YouBot/1.0; +https://docs.you.com/youbot; env:prod) Chrome/X.X.X.X Safari/537.36</code></td><td><code>YouBot</code></td><td>Crawler for the You.com search engine</td><td>"YouBot fully respects robots.txt directives"</td><td>Web Bot Auth; reverse DNS to search.you.com; IP range 68.67.112.0/24</td><td><a href="https://docs.you.com/youbot">YouBot</a></td></tr><tr><td>Common Crawl</td><td>CCBot</td><td><code>CCBot/2.0 (https://commoncrawl.org/faq/)</code></td><td><code>CCBot</code></td><td>Open web archive that anyone can download, including AI developers</td><td>Common Crawl gives robots.txt block instructions</td><td>Reverse DNS to crawl.commoncrawl.org; IP list: index.commoncrawl.org/ccbot.json</td><td><a href="https://commoncrawl.org/ccbot">CCBot</a></td></tr><tr><td>Allen Institute for AI</td><td>AI2Bot</td><td><code>Mozilla/5.0 (compatible) AI2Bot (+https://www.allenai.org/crawler)</code></td><td><code>AI2Bot</code></td><td>Content used to train open language models</td><td>No statement</td><td>No method published</td><td><a href="https://allenai.org/crawler">AI2 crawler</a></td></tr><tr><td>Diffbot</td><td>Diffbot</td><td>Not published</td><td><code>Diffbot</code></td><td>Crawling for a general search engine; Diffbot says it is not used for AI training</td><td>Crawls "adhere to a site's robots.txt instructions" by default</td><td>No method published</td><td><a href="https://docs.diffbot.com/docs/does-crawl-respect-robotstxt">Diffbot robots.txt</a></td></tr><tr><td>Diffbot</td><td>Diffbot-User</td><td>Not published</td><td><code>Diffbot-User</code></td><td>Requests on behalf of a person using Diffbot software</td><td>No separate statement</td><td>No method published</td><td><a href="https://docs.diffbot.com/docs/does-crawl-respect-robotstxt">Diffbot robots.txt</a></td></tr><tr><td>ProRata (Gist)</td><td>ProRataInc</td><td>User agent contains <code>ProRataInc</code></td><td>Not stated clearly</td><td>Crawler for the Gist answer engine</td><td>"respects robots.txt directives"</td><td>IP addresses and reverse DNS listed on the page</td><td><a href="https://platform.gist.ai/docs/prorata-ai-crawler-bot">ProRata crawler</a></td></tr></tbody></table></div>

W.X.Y.Z and X.X.X.X are placeholders the operators use for the current browser version. Match on the bot name, not the full string.

Bots without operator documentation we could find

The names below appear in server logs and third-party bot lists. On October 19, 2026, we could not find a page from the operator that documents them, so we do not list a user agent string or a robots.txt behavior for them.

<div class="tbl-wrap"><table style="border-radius:12px;"><thead><tr><th>Name seen</th><th>Company</th><th>What we found</th></tr></thead><tbody><tr><td>Bytespider</td><td>ByteDance</td><td>No ByteDance crawler page found</td></tr><tr><td>GrokBot, xAI-Grok, Grok-DeepSearch</td><td>xAI</td><td>xAI's developer docs document no crawler</td></tr><tr><td>cohere-ai, cohere-training-data-crawler</td><td>Cohere</td><td>Cohere's docs document no crawler</td></tr><tr><td>Timpibot</td><td>Timpi</td><td>The Timpi website did not respond</td></tr><tr><td>omgili, omgilibot, Webzio-Extended</td><td>Webz.io</td><td>No Webz.io crawler page found</td></tr><tr><td>PetalBot</td><td>Huawei</td><td>The crawler page did not load as text</td></tr><tr><td>LinkedInBot</td><td>LinkedIn</td><td>LinkedIn's own robots.txt has a LinkedInBot group; no crawler page found</td></tr><tr><td>anthropic-ai, Claude-Web</td><td>Anthropic</td><td>Anthropic's current crawler page lists ClaudeBot, Claude-SearchBot and Claude-User and does not mention these names</td></tr><tr><td>ChatGPT Atlas browser</td><td>OpenAI</td><td>OpenAI's crawler page does not document a user agent for Atlas</td></tr><tr><td>Brave Search crawler</td><td>Brave</td><td>Brave says it "does not advertise a differentiated user agent" and does not crawl pages that Googlebot cannot crawl</td></tr></tbody></table></div>

Keeping these names in your robots.txt does no harm. A bot that sends one of these names may not be operated by the company the name suggests, because any client can send any user agent string.

OpenAI

OpenAI documents four bots on its crawler page. GPTBot collects content that "may be used in training our generative AI foundation models." OAI-SearchBot is "used to surface websites in search results in ChatGPT's search features." ChatGPT-User fetches pages "for certain user actions in ChatGPT and Custom GPTs," and OpenAI says it "is not used for crawling the web in an automatic fashion." OAI-AdsBot visits only pages submitted as ads on ChatGPT.

OpenAI treats each robots.txt setting as independent. A website can allow OAI-SearchBot to appear in ChatGPT search and disallow GPTBot to opt out of training. If both are allowed, OpenAI says it "may use the results from just one crawl for both use cases." OpenAI also says a robots.txt change can take about 24 hours to affect search results.

For ChatGPT-User, OpenAI says: "Because these actions are initiated by a user, robots.txt rules may not apply." OpenAI points website owners to OAI-SearchBot for search opt-outs. When OpenAI fetches robots.txt itself, it may add a robots.txt marker inside the user agent string.

OpenAI publishes IP ranges for each bot at gptbot.json, searchbot.json, chatgpt-user.json and adsbot.json.

OpenAI's Cloud browser signs its requests with Web Bot Auth. Each request carries Signature, Signature-Input and a Signature-Agent header set to "https://chatgpt.com". OpenAI publishes the public keys at https://chatgpt.com/.well-known/http-message-signatures-directory. Security vendors list this traffic under names such as ChatGPT Agent, according to OpenAI's allowlisting article. OpenAI says that at launch the Cloud browser "cannot sign in to websites or complete payments."

Anthropic

Anthropic's crawler page, dated April 7, 2026, lists three bots. ClaudeBot collects web content that "could potentially contribute to their training." Claude-User accesses websites when people ask Claude questions. Claude-SearchBot "navigates the web to improve search result quality for users."

Anthropic says its bots "respect 'do not crawl' signals by honoring industry standard directives in robots.txt." The page also says Anthropic honors the non-standard Crawl-delay directive and does not try to bypass CAPTCHAs. Anthropic publishes IP addresses at claude.com/crawling/bots.json. It warns that blocking its IP addresses instead of using robots.txt "may not work correctly or persistently guarantee an opt-out," because a block also stops Anthropic from reading your robots.txt.

Anthropic publishes the bot names but not full user agent strings. The page no longer mentions anthropic-ai or Claude-Web, which earlier versions of this list included.

Perplexity

Perplexity's crawler page lists two bots. PerplexityBot is "designed to surface and link websites in search results on Perplexity" and is "not used to crawl content for AI foundation models." Perplexity-User fetches a page when a person asks Perplexity a question. Perplexity says, "Since a user requested the fetch, this fetcher generally ignores robots.txt rules." Perplexity publishes IP lists at perplexitybot.json and perplexity-user.json, and its page recommends pairing a user agent match with an IP check in firewall rules.

Google

Google moved its crawler documentation to developers.google.com/crawling on November 20, 2025. The older Search Central URLs still load. Google groups its bots into three sets.

Common crawlers, including Googlebot, GoogleOther and Google-CloudVertexBot, "always respect robots.txt rules for automatic crawls." Googlebot crawls for Google Search. Google's AI features page says a page appears in AI Overviews and AI Mode when it is indexed and eligible to show in Search with a snippet, with no additional technical requirements. So Googlebot is the crawler that matters for AI Overviews and AI Mode.

Google-Extended is a token, not a crawler. It "doesn't have a separate HTTP request user agent string." Your Google-Extended rules decide whether content Google already crawled may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. Google says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." To limit what AI Overviews and AI Mode show from a page, Google points to nosnippet, data-nosnippet, max-snippet and noindex.

Special-case crawlers, such as AdsBot-Google, "may ignore robots.txt rules." User-triggered fetchers, such as Google-Agent, Google-GeminiNotebook, Google-Read-Aloud and Google-Pinpoint, "generally ignore robots.txt rules" because a person requested the fetch.

Google-Agent is the entry for "agents hosted on Google infrastructure to navigate the web and perform actions upon user request." Google added it on March 20, 2026. Google also signs some Google-Agent requests with Web Bot Auth using the identity https://agent.bot.goog. Google calls this experimental and says it does not sign every request, so check IP addresses too. Google's crawler list does not include a separate Project Mariner entry.

Google publishes IP lists at https://developers.google.com/static/crawling/ipranges/: common-crawlers.json, special-crawlers.json, user-triggered-fetchers.json, user-triggered-fetchers-google.json and user-triggered-agents.json. Google's verification guide explains reverse DNS checks.

Microsoft (Bing and Copilot)

Microsoft documents bingbot, AdIdxBot, BingPreview, MicrosoftPreview and BingVideoPreview on its crawler page. Microsoft does not document a separate Copilot crawler. Bing's Webmaster Guidelines describe how Bing surfaces content "across Bing search experiences, Copilot, and grounding API results."

Bing controls Copilot use through meta tags, not a robots.txt token. The guidelines say: "NOARCHIVE prevents content from being used in Copilot responses and grounding results. NOCACHE limits Copilot to using only the URL, title, and snippet." Bing also advises against NOCACHE on content you want Copilot to cite. Bing publishes IP ranges at bingbot.json and offers a Verify Bingbot tool in Bing Webmaster Tools.

Apple

Apple's Applebot page describes Applebot as the crawler behind Spotlight, Siri and Safari search. Apple says the data Applebot crawls "may also be used to help train Apple foundation models." Applebot-Extended is a token that "does not crawl webpages." Disallowing Applebot-Extended tells Apple not to use your content to train its foundation models, and Apple says it does not remove your pages from search results.

Apple also uses web content for broad world knowledge answers in Siri and Search. A nosnippet meta tag opts specific content out of those answers.

Two Apple behaviors differ from most crawlers. When robots.txt does not mention Applebot but does mention Googlebot, Applebot follows the Googlebot rules. Applebot also does not follow Crawl-delay. You can verify Applebot through reverse DNS to applebot.apple.com and an IP list at applebot.json.

Amazon

Amazon's crawler page lists Amazonbot, Amzn-SearchBot and Amzn-User. Amazonbot data "may be used to train Amazon AI models." Amzn-SearchBot supports search experiences such as Alexa and does not crawl for model training. Amzn-User fetches pages for a person's request and "may not follow all robots.txt directives."

Amazon says automated crawling "respects the Robots Exclusion Protocol." Amazon also says its bots fetch robots.txt or use a cached copy from the last 30 days, do not support Crawl-delay and honor rel=nofollow. A noarchive meta tag tells Amazon not to use the page for training. Amazon publishes a separate IP page for each bot.

Mistral, DuckDuckGo and You.com

Mistral's crawler page lists MistralAI-User for user actions in Mistral's Vibe assistant, MistralAI-Index for Mistral search, and MistralAI-Training for training datasets. Mistral says content crawled by MistralAI-Index "is not used for generative AI training of any kind."

DuckDuckGo's DuckAssistBot page says the bot "crawls pages in real-time for our AI-assisted answers" and that the data "is not used in any way to train AI models." A robots.txt disallow takes effect after 72 hours.

You.com's YouBot page says, "YouBot fully respects robots.txt directives, including user-agent specific rules and crawl-delay settings." YouBot caches robots.txt for 30 minutes. You.com offers three ways to verify YouBot: Web Bot Auth signatures, reverse DNS to search.you.com, and the IP range 68.67.112.0/24.

Common Crawl and research crawlers

Common Crawl's CCBot builds a public web archive that anyone can download, including AI developers. You can verify CCBot through reverse DNS to crawl.commoncrawl.org and an IP list at ccbot.json. Common Crawl warns that other crawlers falsely identify as CCBot.

The Allen Institute for AI says AI2Bot collects content "used to train open language models." Diffbot documents two tokens, Diffbot and Diffbot-User, and says its crawls follow robots.txt by default.

How to allow or block each bot in robots.txt

A crawler chooses whether to follow robots.txt. RFC 9309 says: "These rules are not a form of access authorization." Here is how a crawler reads the file under RFC 9309:

The crawler looks for a group whose User-agent: line matches its token. The match is not case-sensitive.
If it finds a matching group, it follows that group and ignores the User-agent: * group.
If it finds no matching group, it follows the * group.
Within a group, the longest matching path rule wins. When an Allow rule and a Disallow rule match equally, Allow wins.

Two operator exceptions change step 3. Applebot follows Googlebot's group when robots.txt has no Applebot group. Amzn-SearchBot follows other search bots' rules when it is not named.

Write the token, not the full user agent string. User-agent: ChatGPT-User/2.0 is not a valid token under RFC 9309, because tokens contain only letters, hyphens and underscores. Use User-agent: ChatGPT-User.

Example 1 allows AI search and user-triggered fetches and opts out of model training:

# Opt out of AI model training User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: meta-externalagent User-agent: MistralAI-Training User-agent: CCBot User-agent: AI2Bot Disallow: / # Allow AI search and user-triggered fetches User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User User-agent: MistralAI-Index User-agent: MistralAI-User User-agent: meta-webindexer User-agent: Amzn-SearchBot User-agent: DuckAssistBot Allow: / User-agent: * Allow: /

Amazonbot and Applebot are missing from both groups in Example 1 on purpose. Amazon and Apple say data from those crawlers may train their models, and the same crawlers serve their search features. Applebot-Extended controls Apple's training use without blocking Applebot. Amazon offers a noarchive meta tag for the same purpose.

Example 2 blocks one AI search crawler from a folder and leaves the rest of the website open:

User-agent: PerplexityBot Disallow: /members/ Allow: /

Example 3 adds Content Signals, which state how you allow content to be used after it is crawled. Cloudflare published the Content Signals Policy in September 2025 under a CC0 license. It defines search, ai-input (using content in AI answers, such as retrieval and grounding) and ai-train. Momentic's own robots.txt uses this line:

User-Agent: * Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: /

Content Signals express a preference. As of October 19, 2026, none of the operator crawler pages linked on this page mentions Content Signals.

Robots.txt cannot reliably stop user-triggered fetchers. OpenAI, Perplexity, Google, Amazon and Meta each say their user-triggered fetchers may not follow robots.txt. To block those, use a firewall rule that matches the user agent and checks the operator's IP list. This Nginx rule refuses requests that name PerplexityBot:

if ($http_user_agent ~* "PerplexityBot") { return 403; }

A rule like this blocks any client that sends the name, including impostors. It does not block requests that arrive with a browser user agent.

Changes take time to reach each operator. RFC 9309 says crawlers should not use a cached robots.txt for more than 24 hours. OpenAI says search changes take about 24 hours. DuckDuckGo says 72 hours. Amazon says its bots may use a cached robots.txt from the last 30 days.

How to confirm a bot is genuine

Any client can send any user agent string. Operators offer three ways to confirm a request came from them.

Reverse DNS works for Google, Apple, Common Crawl and You.com. Look up the hostname for the request's IP address, check that it ends in the operator's domain, then look up that hostname and confirm it returns the same IP address. For Googlebot:

$ host 66.249.66.1 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com. $ host crawl-66-249-66-1.googlebot.com crawl-66-249-66-1.googlebot.com has address 66.249.66.1

IP lists work for OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Amazon, Mistral, DuckDuckGo and Common Crawl. Each list is a JSON file or web page linked in the master table. Check the request's IP address against the list for the bot it claims to be. The lists change, so fetch them on a schedule.

Web Bot Auth signatures work for OpenAI's Cloud browser, part of Google-Agent's traffic and YouBot. The bot signs each request with HTTP Message Signatures (RFC 9421) and names its key directory in a Signature-Agent header. Your server or CDN fetches the public key and checks the signature. The IETF's webbotauth working group is standardizing the method; its working draft is draft-ietf-webbotauth-httpsig-protocol.

Many CDNs and firewalls run these checks for you. Cloudflare's verified bots program and its AI Crawl Control feature identify AI crawlers by these methods and let you allow, block or charge each one. Charging uses Cloudflare's pay per crawl, which answers an unpaid crawler with HTTP 402 and a price.

To find AI bots in your own access logs, search for the tokens:

Then check the IP addresses on those lines with one of the three methods above.

AI bot requests to momenticmarketing.com

Momentic measures AI bot requests to momenticmarketing.com. The numbers below come from one website with a marketing audience, so other websites will see a different mix. User agents are self-reported unless we say a count is verified.

Between August 6 and October 19, 2026, our CDN counted 47,616 requests from AI bots. Training crawlers made 24,888 of them (52.3%). Bots fetching pages for a person's live question made 19,072 (40.1%). Search crawlers made 3,656 (7.7%). The largest single bots were meta-externalagent (14,653 requests), ChatGPT-User (13,325), ClaudeBot (4,074) and Amazonbot (3,674).

GPTBot made the fewest verified requests of OpenAI's three main bots. Between August 31 and October 18, 2026, our CDN verified 6,155 requests from ChatGPT-User, 2,401 from OAI-SearchBot and 176 from GPTBot. ChatGPT-User made 35 times as many verified requests as GPTBot. For Anthropic in the same window, ClaudeBot made 2,128 verified requests and Claude-User made 1,666. Unverified requests using these same names were about as numerous, and some of those come from impostors.

AI bots rarely requested our llms.txt file. Over 83 days (July 29 to October 19, 2026), requests identifying as OpenAI, Anthropic, Perplexity, Google, Microsoft or Meta fetched robots.txt 6,851 times and llms.txt 61 times. Claude's bots fetched robots.txt 2,495 times and llms.txt 6 times. OpenAI's bots fetched robots.txt 734 times and llms.txt 23 times.

Most AI bot requests asked for HTML. Of 46,541 responses our website served to AI bots between August 7 and October 19, 2026, 84.0% were HTML and 3.6% were markdown.

Few websites block AI bots in robots.txt, and firewalls refuse them more often. Atlas, Momentic's AI search measurement platform, checked 38 AI bots against 155 websites between September 25 and October 19, 2026. Robots.txt blocked at least one AI bot on 9 websites (5.8%). The server refused at least one bot that robots.txt allowed on 61 websites (39.4%). Our checks run from our own servers, not the operators' networks, so a refusal in our check is a lead to confirm in your logs.

Momentic's public MCP server at https://momenticmarketing.com/mcp includes a check_bots tool you can run on any URL. It requests the URL with 12 AI bot user agents and a regular browser user agent, then reports which bots received a different response. It does not test Google-Extended or Applebot-Extended, because no request carries those tokens. More answers from our data are on Momentic's AI search questions page.

llms.txt and the IETF AI Preferences drafts

llms.txt is a proposed markdown file at the root of a website that summarizes the website for language models (llmstxt.org). OpenAI, Anthropic, Perplexity and Google publish llms.txt files for their own developer docs. As of October 19, 2026, none of the crawler pages listed on this page says the crawler reads llms.txt. Google's AI optimization guide says Google Search does not use such files and lists llms.txt among "unnecessary AI text files." Our own counts above show AI company bots fetched robots.txt about 112 times as often as llms.txt.

The IETF's AI Preferences working group is drafting a standard vocabulary for AI use preferences. Its current drafts are draft-ietf-aipref-vocab, which defines train-ai, ai-use and search, and draft-ietf-aipref-attach, which adds a Content-Usage HTTP header and a Content-Usage robots.txt rule. Both are drafts, not standards, and their labels differ from Cloudflare's Content Signals (ai-train, ai-input).

What changed in this update

This update replaces the November 20, 2025 version. Each row was rechecked against the operator's documentation on October 19, 2026.

New bots and controls added:

OAI-AdsBot (OpenAI), which checks landing pages submitted as ChatGPT ads.
OpenAI's Cloud browser, which signs requests with Web Bot Auth.
Claude-SearchBot and Claude-User (Anthropic).
Google-Agent, which Google added on March 20, 2026, and Google-GeminiNotebook, which replaced Google-NotebookLM on July 16, 2026.
GoogleOther and Google-CloudVertexBot.
meta-webindexer, meta-externalfetcher and meta-externalads (Meta).
Amzn-SearchBot and Amzn-User (Amazon).
MistralAI-Index and MistralAI-Training (Mistral).
Diffbot-User and ProRataInc.

Corrections to earlier rows:

ClaudeBot is Anthropic's training crawler. The earlier list described it as a fetcher for citations in chat.
anthropic-ai and Claude-Web are no longer in Anthropic's documentation. They moved to the undocumented table.
meta-externalagent is Meta's crawler for AI training and product indexing. The earlier list described it as a backup fetcher.
FacebookBot is no longer on Meta's crawler page. facebookexternalhit is the link preview crawler.
Applebot-Extended is a token that opts content out of training. The earlier list described it as "opt-in only."
Apple now says Applebot data may also help train its foundation models.
GPTBot and OAI-SearchBot user agents now show version 1.4.
ChatGPT-User: OpenAI publishes the string with version 1.0. The earlier list mentioned versions 2.0 and 3.0 seen in logs, and its robots.txt examples used ChatGPT-User/2.0 as a token, which RFC 9309 does not allow.
GoogleAgent-Mariner is not in Google's crawler list. Google-Agent is Google's documented agent.
Grok, Cohere, Bytespider, Timpibot and omgili moved to the undocumented table, with no user agent strings, because we found no operator documentation.

New sections: purpose groups, verification by reverse DNS, IP lists and Web Bot Auth, Bing's NOARCHIVE and NOCACHE controls for Copilot, Content Signals, the IETF AI Preferences drafts, llms.txt, and Momentic's own request counts.

Other tools that test robots.txt

Knowatoa's AI Search Console tests a robots.txt file against a list of AI crawlers and shows which ones it blocks.

Knowatoa's AI search console dashboard — Knowatoa's AI Search Console dashboard

The robots.txt tester at technicalseo.com checks one user agent at a time against your file.

robots.txt Validator and Testing Tool from technicalseo.com

FAQ

What is the difference between a user agent and a robots.txt token?

A user agent is the full string a bot sends with each request, such as Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot. A robots.txt token is the short name you write in robots.txt, such as GPTBot. Two tokens, Google-Extended and Applebot-Extended, have no user agent at all.

Does blocking GPTBot remove my website from ChatGPT?

No, according to OpenAI. GPTBot controls training use. OAI-SearchBot controls whether pages appear in ChatGPT search answers. ChatGPT-User fetches pages when a person asks, and OpenAI says robots.txt rules may not apply to it.

Does blocking Google-Extended remove my website from AI Overviews?

No. Google says Google-Extended does not affect inclusion in Google Search. AI Overviews and AI Mode use pages that Googlebot crawls. To limit what those features show, Google points to nosnippet, data-nosnippet, max-snippet and noindex.

Is User-agent: * enough to cover AI bots?

It covers bots that have no group of their own in your robots.txt. Under RFC 9309, a bot with its own group follows that group and ignores *. Applebot follows Googlebot's group when no Applebot group exists. User-triggered fetchers may not read robots.txt at all.

Do AI bots have to follow robots.txt?

RFC 9309 says robots.txt rules "are not a form of access authorization." Whether a bot follows them depends on the operator. The master table quotes what each operator says. To enforce a block, use a firewall rule with IP or signature verification.

Which AI bots ignore robots.txt?

Operators say their user-triggered fetchers may not follow it: ChatGPT-User, Perplexity-User, Amzn-User, meta-externalfetcher and Google's user-triggered fetchers, including Google-Agent. Meta says facebookexternalhit might bypass robots.txt during security checks. Google says its special-case crawlers may ignore robots.txt.

Do AI crawlers read llms.txt?

None of the operators on this page documents that its crawler reads llms.txt. On momenticmarketing.com over 83 days, AI company bots fetched llms.txt 61 times and robots.txt 6,851 times.

How do I know a request claiming to be ClaudeBot is really from Anthropic?

Check the request's IP address against Anthropic's list at claude.com/crawling/bots.json. For bots from Google, Apple, Common Crawl and You.com, a reverse DNS check also works.

Does Copilot have its own crawler?

Microsoft does not document one. Bing's Webmaster Guidelines apply one set of crawling and indexing rules to Bing search, Copilot and grounding results. The guidelines say a noarchive meta tag keeps content out of Copilot responses and a nocache tag limits Copilot to the URL, title and snippet.

<div class="post-note">Spot a bot we missed, or a change in an operator's documentation? <a href="https://www.linkedin.com/in/tyler-einberger/" target="_blank">Message Tyler on LinkedIn</a> and we will check it for the next update.</div>

Originally published: February 7, 2025 | Updated: October 19, 2026

Est. Read Time:

18 minutes

You're reading:

AI user agents, bots and crawlers to watch (October 2026 update)

Tyler Einberger

You're reading:

AI user agents, bots and crawlers to watch (October 2026 update)

Like it? Share it!