For a brand that wants AI answers to mention and cite it, the best AI bot access policy allows three kinds of AI bot on public pages: search crawlers, assistant fetchers and training crawlers. A robots.txt file states the permission, and the firewall and CDN have to grant it, because a firewall can refuse a bot that robots.txt allows. Publishers and other businesses whose content is the product have reason to block training crawlers, and the opt-out robots.txt is under "When to choose a different policy."
Last checked: November 20, 2026. Momentic's own website uses this policy.
The starter policy
- Keep private paths closed to crawlers: admin, cart, checkout, account and internal API paths.
- Make the firewall and CDN match robots.txt. Verified bots receive the page with status 200 and do not receive a challenge page.
- Compare the status codes in your request logs with what robots.txt permits.
The robots.txt file
Copy this file and replace the paths and the sitemap address with your own.
The file has one group on purpose. Under the robots.txt standard (RFC 9309), a bot that finds a group with its own name follows that group and ignores the User-agent: * group. A file that names GPTBot in a separate group has to repeat the Disallow lines there, or GPTBot may crawl the paths you closed. Name a bot only when its rules differ from the rules for other bots.
The Content-Signal line states three permissions in a form software can read: appearing in search results, being used as a source in an AI answer, and being used for training. Cloudflare published the convention in September 2025. As of November 20, 2026, none of the AI companies' crawler pages says its bots read the line, so it documents your decision and does not enforce it.

Why the starter policy allows training crawlers
Most published robots.txt templates for AI bots block training crawlers and allow search crawlers. For a brand that sells a product or a service, we recommend allowing training.
What we measured. Atlas, Momentic's AI search measurement platform, asks the same category prompts to ChatGPT, Claude and Gemini with web search turned on and with it turned off. The window is September 1 to October 18, 2026, across 21 brands, with about 7,000 scored answers per lane. We also counted AI bot requests to momenticmarketing.com by kind from August 6 to October 19, 2026.
What we found. Without web search, ChatGPT mentioned the measured brand in 8.9% of answers, Claude in 6.8% and Gemini in 10.8%. With web search, the rates were 13.2%, 15.8% and 16.0%. On our website, training crawlers made 24,888 of 47,616 AI bot requests (52.3%).
About 7,000 scored answers per lane · 21 brands · September 1 to October 18, 2026
What it means for you. An answer without web search comes from what the model learned in training, and a training crawler is how a page gets into that training. A block today does not remove a brand from models that are already trained. An allow today can only affect models trained later.
What we don't know yet. We have not measured whether blocking training crawlers changes how AI answers describe a brand. That test takes months, because the effect depends on when each company trains its next model.
Why GPTBot is not the bot that decides ChatGPT citations
A policy written only for GPTBot leaves the two OpenAI bots that serve live answers unaddressed.
What we measured. OpenAI runs three main bots. GPTBot collects training data, OAI-SearchBot builds the index ChatGPT search uses, and ChatGPT-User fetches a page when ChatGPT needs it for a person's question. We counted the requests our CDN verified as coming from each bot on momenticmarketing.com from August 31 to October 18, 2026.
What we found. ChatGPT-User made 6,155 verified requests, OAI-SearchBot made 2,401 and GPTBot made 176. ChatGPT-User made 35 times as many verified requests as GPTBot.
Verified requests · Momentic's website · August 31 to October 18, 2026
What it means for you. OpenAI's crawler page says, "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers." Allow OAI-SearchBot and ChatGPT-User if you want ChatGPT to cite you, and test both at the firewall. For Google, AI Overviews and AI Mode use pages that Googlebot crawled, and Google says its Google-Extended training control "does not impact a site's inclusion in Google Search."
What we don't know yet. This is one website with a marketing audience. We have not measured how the mix of bots differs by industry or by how often a website changes.

Why robots.txt is half the policy
A firewall or bot protection product can refuse a bot that robots.txt allows, and robots.txt does not report the refusal.
What we measured. Atlas requested the homepage of 155 websites while presenting each of 38 AI bot identities. It recorded whether robots.txt allowed the bot and whether the server answered. The websites are brands Momentic measures and their competitors. The checks ran between September 25 and October 19, 2026.
What we found. Robots.txt blocked one or more bots on 9 websites (5.8%). The server refused one or more bots that robots.txt allowed on 61 websites (39.4%). Across the website and bot pairs, 525 of 5,890 (8.9%) were refused at the server and 131 (2.2%) were blocked by robots.txt.
155 websites · 38 AI bots · checks run September 25 to October 19, 2026
What it means for you. Give your firewall and CDN the same policy as your robots.txt.
- A verified bot that requests a public page receives the page with status 200.
- A verified bot does not receive a JavaScript challenge or a CAPTCHA, because a crawler cannot complete one.
- Rate limits do not apply to verified search and AI bots, or apply at a level those bots do not reach.
- A request that uses a bot's name from an IP address outside that company's published list is treated like other anonymous traffic.
- Bots receive the same URL and the same HTML as a browser.
A bot is verified when its IP address is on the list its company publishes, or when its hostname passes a reverse DNS check. OpenAI, Anthropic, Perplexity, Google and Microsoft publish IP lists. Verification matters because names are easy to copy. On our website between October 22 and November 19, 2026, 6,103 requests that used the Googlebot name were verified as Google and 11,324 were not. For ChatGPT-User the counts were 2,225 verified and 740 not.

What we don't know yet. Our checks run from our own servers and not from the AI companies' networks. Some firewalls refuse an unknown IP address that presents a bot's name and let the real bot through. Confirm any refusal from a check like ours in your logs.
When to choose a different policy
To opt out of training and stay in AI answers, add a group that names the training crawlers. Repeat your Disallow lines in the second group.
Robots.txt cannot reliably stop assistant fetchers. OpenAI says of ChatGPT-User that "robots.txt rules may not apply" because a person started the request, and Perplexity says Perplexity-User "generally ignores robots.txt rules." A firewall rule that checks the company's IP list is the control for those bots.
How to confirm the policy works
Request a page as a browser and as a bot, and compare the status code and the size of the response. This command sends a browser request, a GPTBot request and a ChatGPT-User request, using the user agent strings OpenAI publishes.
On momenticmarketing.com on November 20, 2026, the three requests each returned status 200 and 145,832 bytes. A 403, a 429 or a much smaller response for a bot means the firewall or CDN treats that bot differently from a browser. Momentic's public MCP server at https://momenticmarketing.com/mcp has a check_bots tool that runs the same comparison for 12 AI bots on any URL.
A test like this sends a bot's name from your own IP address, so it shows how the server treats the name and not how it treats the real bot. Request logs show how the server treated the real bot. For each verified bot, count requests by status code and read the result this way.
Controls for a page, a file or one HTML element
Robots.txt and the firewall decide whether a bot receives a URL. Narrower controls decide what a search engine or AI feature may show from a page it did receive. Use them when a URL should stay open to bots and part of its content should not be quoted.
Passages that suit data-nosnippet include legal disclaimers, a price that changes daily, a cookie notice and a phone number meant for existing customers. The attribute works only on span, div and section elements.
These controls are instructions to search engines. They do not remove text from the response. An assistant fetcher that requests the URL receives the full HTML, including text marked data-nosnippet. We have not found a statement from OpenAI, Anthropic or Perplexity that their bots follow data-nosnippet, nosnippet or max-snippet.
Text that must not reach a bot has to be left out of the HTML the server sends. For a gated report or a members' page, send a bot the same response a logged-out person receives: the summary and the form. A bot then reads and cites the summary, and the gated text stays on the server. A publisher that sends the full article to Googlebot under the paywall markup should verify Googlebot by IP address first, because any client can send the Googlebot name.
We recommend against serving AI bots a separate simplified version of a page. A second version has to be kept in sync with the first, and a person who follows a citation arrives on the simplified page.
Where llms.txt fits
An llms.txt file is optional in this policy. It is a short text summary of a website written for AI tools. On momenticmarketing.com, requests that identified as OpenAI, Anthropic, Perplexity, Google, Microsoft or Meta fetched robots.txt 6,851 times and llms.txt 61 times between July 29 and October 19, 2026. Google's AI optimization guide lists llms.txt among "unnecessary AI text files." Publish one if you want a short, accurate summary for the tools that do read it, and keep it current.
The starter policy as a checklist
- Publish the robots.txt above with your own private paths and sitemap address.
- Ask whoever runs your firewall or CDN which AI bots it blocks, challenges or rate-limits.
- Change the firewall and CDN settings until the five server requirements are met.
- Run the browser and bot comparison on your homepage and on one page of each main template.
- Check for
noindextags on pages you want cited, and mark passages you do not want quoted withdata-nosnippet. - Count requests by bot and status code in your logs one week later, using verified requests only.
- Review the bot list twice a year. AI companies add and rename bots.
Short answers about AI bot access
Method and limits
The bot request counts come from one website, momenticmarketing.com, as counted by its CDN. The 155-website check sent requests from Momentic's servers and not from the AI companies' networks. The mention rates come from Atlas prompt runs across 21 brands that Momentic measures. Operator statements were read on the operators' own crawler pages. The user agent strings in the test command are from OpenAI's crawler page as read on November 20, 2026.
Related Momentic research
- List of AI search crawlers and user agents: each bot's user agent string, robots.txt token, purpose and verification method.
- Momentic's perspective on hot topics in AI search: the measurements quoted on this page, with their limits.
- The AI Search Operating Manual: crawler access as one of four layers to check.
Sources
- OpenAI, Overview of OpenAI crawlers
- Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Perplexity, Perplexity crawlers
- Google, AI features and your website and Overview of Google crawlers
- Google, Robots meta tag, data-nosnippet and X-Robots-Tag specifications and Subscription and paywalled content
- Microsoft, Bing introduces support for the data-nosnippet HTML attribute
- IETF, RFC 9309: Robots Exclusion Protocol
- Cloudflare, Content Signals Policy
this text
