What AI bot access policy is best for AEO and GEO? Steal Momentic's starter policy

Posted in:
AI Updates
//
Friday, November 20, 2026
November 20, 2026

Momentic's starter AI bot access policy for brands that want AI answers to mention and cite them. It allows AI search, assistant and training bots, includes a robots.txt you can copy, lists the firewall checks, and gives the data behind each decision.

For a brand that wants AI answers to mention and cite it, the best AI bot access policy allows three kinds of AI bot on public pages: search crawlers, assistant fetchers and training crawlers. A robots.txt file states the permission, and the firewall and CDN have to grant it, because a firewall can refuse a bot that robots.txt allows. Publishers and other businesses whose content is the product have reason to block training crawlers, and the opt-out robots.txt is under "When to choose a different policy."

Last checked: November 20, 2026. Momentic's own website uses this policy.

The starter policy

Kind of botWhat it doesStarter decisionExamples
Search crawlerBuilds the index an AI answer cites fromAllowOAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, bingbot
Assistant fetcherRequests one page because a person asked an assistant a questionAllowChatGPT-User, Claude-User, Perplexity-User
Training crawlerCollects pages a company may use to train modelsAllow, unless your content is your productGPTBot, ClaudeBot, meta-externalagent, CCBot
  1. Keep private paths closed to crawlers: admin, cart, checkout, account and internal API paths.
  2. Make the firewall and CDN match robots.txt. Verified bots receive the page with status 200 and do not receive a challenge page.
  3. Compare the status codes in your request logs with what robots.txt permits.

The robots.txt file

Copy this file and replace the paths and the sitemap address with your own.

# Search engines, AI search, AI assistants and AI training may read public pages.
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /
Disallow: /admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/

Sitemap: https://www.example.com/sitemap.xml

The file has one group on purpose. Under the robots.txt standard (RFC 9309), a bot that finds a group with its own name follows that group and ignores the User-agent: * group. A file that names GPTBot in a separate group has to repeat the Disallow lines there, or GPTBot may crawl the paths you closed. Name a bot only when its rules differ from the rules for other bots.

The Content-Signal line states three permissions in a form software can read: appearing in search results, being used as a source in an AI answer, and being used for training. Cloudflare published the convention in September 2025. As of November 20, 2026, none of the AI companies' crawler pages says its bots read the line, so it documents your decision and does not enforce it.

Momentic's robots.txt file, with one User-agent group and a Content-Signal line that permits search, AI input and AI training
Momentic's robots.txt on November 20, 2026.

Why the starter policy allows training crawlers

Most published robots.txt templates for AI bots block training crawlers and allow search crawlers. For a brand that sells a product or a service, we recommend allowing training.

What we measured. Atlas, Momentic's AI search measurement platform, asks the same category prompts to ChatGPT, Claude and Gemini with web search turned on and with it turned off. The window is September 1 to October 18, 2026, across 21 brands, with about 7,000 scored answers per lane. We also counted AI bot requests to momenticmarketing.com by kind from August 6 to October 19, 2026.

What we found. Without web search, ChatGPT mentioned the measured brand in 8.9% of answers, Claude in 6.8% and Gemini in 10.8%. With web search, the rates were 13.2%, 15.8% and 16.0%. On our website, training crawlers made 24,888 of 47,616 AI bot requests (52.3%).

About 7,000 scored answers per lane · 21 brands · September 1 to October 18, 2026

What it means for you. An answer without web search comes from what the model learned in training, and a training crawler is how a page gets into that training. A block today does not remove a brand from models that are already trained. An allow today can only affect models trained later.

What we don't know yet. We have not measured whether blocking training crawlers changes how AI answers describe a brand. That test takes months, because the effect depends on when each company trains its next model.

Why GPTBot is not the bot that decides ChatGPT citations

A policy written only for GPTBot leaves the two OpenAI bots that serve live answers unaddressed.

What we measured. OpenAI runs three main bots. GPTBot collects training data, OAI-SearchBot builds the index ChatGPT search uses, and ChatGPT-User fetches a page when ChatGPT needs it for a person's question. We counted the requests our CDN verified as coming from each bot on momenticmarketing.com from August 31 to October 18, 2026.

What we found. ChatGPT-User made 6,155 verified requests, OAI-SearchBot made 2,401 and GPTBot made 176. ChatGPT-User made 35 times as many verified requests as GPTBot.

Verified requests · Momentic's website · August 31 to October 18, 2026

What it means for you. OpenAI's crawler page says, "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers." Allow OAI-SearchBot and ChatGPT-User if you want ChatGPT to cite you, and test both at the firewall. For Google, AI Overviews and AI Mode use pages that Googlebot crawled, and Google says its Google-Extended training control "does not impact a site's inclusion in Google Search."

What we don't know yet. This is one website with a marketing audience. We have not measured how the mix of bots differs by industry or by how often a website changes.

Atlas crawler traffic view for momenticmarketing.com showing requests by visitor type from October 22 to November 19, 2026
Crawler traffic to momenticmarketing.com in Atlas, October 22 to November 19, 2026. AI crawlers made 37,942 requests, 5.9% of the total.

Why robots.txt is half the policy

A firewall or bot protection product can refuse a bot that robots.txt allows, and robots.txt does not report the refusal.

What we measured. Atlas requested the homepage of 155 websites while presenting each of 38 AI bot identities. It recorded whether robots.txt allowed the bot and whether the server answered. The websites are brands Momentic measures and their competitors. The checks ran between September 25 and October 19, 2026.

What we found. Robots.txt blocked one or more bots on 9 websites (5.8%). The server refused one or more bots that robots.txt allowed on 61 websites (39.4%). Across the website and bot pairs, 525 of 5,890 (8.9%) were refused at the server and 131 (2.2%) were blocked by robots.txt.

155 websites · 38 AI bots · checks run September 25 to October 19, 2026

What it means for you. Give your firewall and CDN the same policy as your robots.txt.

  1. A verified bot that requests a public page receives the page with status 200.
  2. A verified bot does not receive a JavaScript challenge or a CAPTCHA, because a crawler cannot complete one.
  3. Rate limits do not apply to verified search and AI bots, or apply at a level those bots do not reach.
  4. A request that uses a bot's name from an IP address outside that company's published list is treated like other anonymous traffic.
  5. Bots receive the same URL and the same HTML as a browser.

A bot is verified when its IP address is on the list its company publishes, or when its hostname passes a reverse DNS check. OpenAI, Anthropic, Perplexity, Google and Microsoft publish IP lists. Verification matters because names are easy to copy. On our website between October 22 and November 19, 2026, 6,103 requests that used the Googlebot name were verified as Google and 11,324 were not. For ChatGPT-User the counts were 2,225 verified and 740 not.

Atlas top crawlers table listing verified and unverified request counts per crawler, including Googlebot and ChatGPT-User
Requests per crawler on momenticmarketing.com, October 22 to November 19, 2026. The Unverified column counts requests that used a crawler's name and failed verification.

What we don't know yet. Our checks run from our own servers and not from the AI companies' networks. Some firewalls refuse an unknown IP address that presents a bot's name and let the real bot through. Confirm any refusal from a check like ours in your logs.

When to choose a different policy

Your situationChange to the starter policy
Your content is the product: news, research, courses, paid dataDisallow training crawlers and keep search crawlers and assistant fetchers allowed
Pages show customer-written content such as reviews or forum postsDecide whether training crawlers may read those paths, and write the decision down. A model that learns customer opinions keeps them after you delete the page
Content is licensed from a third partyFollow the license. Disallow training on those paths if the license does not permit it
Pages are behind a login or a paywallNo change. Bots cannot read them, and the policy applies to public pages

To opt out of training and stay in AI answers, add a group that names the training crawlers. Repeat your Disallow lines in the second group.

# Opt out of AI model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
Disallow: /

# Other bots, including AI search crawlers and assistant fetchers
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /
Disallow: /admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/

Sitemap: https://www.example.com/sitemap.xml

Robots.txt cannot reliably stop assistant fetchers. OpenAI says of ChatGPT-User that "robots.txt rules may not apply" because a person started the request, and Perplexity says Perplexity-User "generally ignores robots.txt rules." A firewall rule that checks the company's IP list is the control for those bots.

How to confirm the policy works

Request a page as a browser and as a bot, and compare the status code and the size of the response. This command sends a browser request, a GPTBot request and a ChatGPT-User request, using the user agent strings OpenAI publishes.

for ua in \
  "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36" \
  "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" \
  "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot"
do
  curl -s -o /dev/null -w "%{http_code} %{size_download}\n" -A "$ua" https://www.example.com/
done

On momenticmarketing.com on November 20, 2026, the three requests each returned status 200 and 145,832 bytes. A 403, a 429 or a much smaller response for a bot means the firewall or CDN treats that bot differently from a browser. Momentic's public MCP server at https://momenticmarketing.com/mcp has a check_bots tool that runs the same comparison for 12 AI bots on any URL.

A test like this sends a bot's name from your own IP address, so it shows how the server treats the name and not how it treats the real bot. Request logs show how the server treated the real bot. For each verified bot, count requests by status code and read the result this way.

Status in the logMeaning
200 or 304The bot received the page
403A rule blocked the bot
429A rate limit applied to the bot
499 on nginx, 000 on Amazon CloudFrontThe bot closed the connection before the response finished, which means the page was too slow for that bot

Controls for a page, a file or one HTML element

Robots.txt and the firewall decide whether a bot receives a URL. Narrower controls decide what a search engine or AI feature may show from a page it did receive. Use them when a URL should stay open to bots and part of its content should not be quoted.

What you wantControlWho documents support
Keep a page out of search results and the AI answers built on themGoogle and Bing. The page has to stay crawlable, because Google says the rule is read "only if crawlers are allowed to access the pages"
Keep one passage out of snippets and AI summaries while the page stays indexeddata-nosnippet on a span, div or sectionGoogle, and Bing since October 2025
Keep the whole page out of snippets and AI summaries while it stays indexednosnippet in the robots meta tagGoogle says this "will also prevent the content from being used as a direct input for AI Overviews and AI Mode"
Limit how much text can be quotedmax-snippet:[number of characters]Google says this limits "how much of the content may be used as a direct input for AI Overviews and AI Mode"
Apply these rules to a PDF, image or video fileX-Robots-Tag response headerGoogle
Remove a page from results after a date, such as a promotion that endsunavailable_after:[date]Google
Opt a page out of Amazon's model trainingnoarchive in the robots meta tagAmazon
Let Google index a paywalled article in fullPaywall structured data: isAccessibleForFree set to false, with a cssSelector for the paywalled sectionGoogle, which says the markup helps it "differentiate paywalled content from the practice of cloaking"

Passages that suit data-nosnippet include legal disclaimers, a price that changes daily, a cookie notice and a phone number meant for existing customers. The attribute works only on span, div and section elements.

These controls are instructions to search engines. They do not remove text from the response. An assistant fetcher that requests the URL receives the full HTML, including text marked data-nosnippet. We have not found a statement from OpenAI, Anthropic or Perplexity that their bots follow data-nosnippet, nosnippet or max-snippet.

Text that must not reach a bot has to be left out of the HTML the server sends. For a gated report or a members' page, send a bot the same response a logged-out person receives: the summary and the form. A bot then reads and cites the summary, and the gated text stays on the server. A publisher that sends the full article to Googlebot under the paywall markup should verify Googlebot by IP address first, because any client can send the Googlebot name.

We recommend against serving AI bots a separate simplified version of a page. A second version has to be kept in sync with the first, and a person who follows a citation arrives on the simplified page.

Where llms.txt fits

An llms.txt file is optional in this policy. It is a short text summary of a website written for AI tools. On momenticmarketing.com, requests that identified as OpenAI, Anthropic, Perplexity, Google, Microsoft or Meta fetched robots.txt 6,851 times and llms.txt 61 times between July 29 and October 19, 2026. Google's AI optimization guide lists llms.txt among "unnecessary AI text files." Publish one if you want a short, accurate summary for the tools that do read it, and keep it current.

The starter policy as a checklist

  1. Publish the robots.txt above with your own private paths and sitemap address.
  2. Ask whoever runs your firewall or CDN which AI bots it blocks, challenges or rate-limits.
  3. Change the firewall and CDN settings until the five server requirements are met.
  4. Run the browser and bot comparison on your homepage and on one page of each main template.
  5. Check for noindex tags on pages you want cited, and mark passages you do not want quoted with data-nosnippet.
  6. Count requests by bot and status code in your logs one week later, using verified requests only.
  7. Review the bot list twice a year. AI companies add and rename bots.

Short answers about AI bot access

QuestionAnswer
Does blocking GPTBot remove a website from ChatGPT?No. GPTBot collects training data. OpenAI says OAI-SearchBot controls ChatGPT search visibility, and each robots.txt setting is independent.
Does blocking Google-Extended remove a website from AI Overviews?No. Google says Google-Extended does not affect inclusion in Google Search. AI Overviews and AI Mode use pages that Googlebot crawled and indexed.
Is User-agent: * enough for AI bots?Yes, when the same rules apply to each bot. A bot without its own group follows the * group. Apple and Amazon document one exception each: Applebot follows Googlebot's group when it has none of its own, and Amzn-SearchBot follows other search bots' rules.
How long does a robots.txt change take to apply?OpenAI says about 24 hours for search. RFC 9309 says a crawler should not use a cached robots.txt for more than 24 hours. Amazon says its bots may use a copy cached within the past 30 days.
Do AI bots have to follow robots.txt?No. RFC 9309 says robots.txt rules "are not a form of access authorization." The search and training crawlers named on this page say they follow it. Several assistant fetchers say they may not.

Method and limits

The bot request counts come from one website, momenticmarketing.com, as counted by its CDN. The 155-website check sent requests from Momentic's servers and not from the AI companies' networks. The mention rates come from Atlas prompt runs across 21 brands that Momentic measures. Operator statements were read on the operators' own crawler pages. The user agent strings in the test command are from OpenAI's crawler page as read on November 20, 2026.

Related Momentic research

Sources

post-help
post-action
post-note-cute
post-grade
post-alert
post-note
\