AI Crawler Policy Generator: who trains on your content, who quotes it
Every AI company sends more than one bot, and they do different jobs. Some put you in front of buyers inside ChatGPT, Claude, and Perplexity answers. Others copy your pages into training data. This tool writes a robots.txt that welcomes the first group and turns the second away, in one click.
Also known as
The guest list on your front door
robots.txt is the sign on the door
It is a small text file that tells visiting bots what they may do on your site. Well-behaved companies read it and obey. It sets policy; it is not a lock, and it works because the major AI labs have promised to respect it.
Search bots are the food critics
They read your pages so AI assistants can recommend you by name when someone asks a question. Turning them away makes you invisible in AI answers, which is where more buyers now start.
Training bots are the photocopiers
They collect your writing to teach future models. That builds no visibility for you; it just donates your work. Most sites want critics in and photocopiers out. That is the preset we recommend.
Why this page matters, in one sentence: the wrong robots.txt either hides you from the AI answers your buyers are reading, or hands your content to training runs you never agreed to, and most sites accidentally do one of the two.
Build your policy
Pick a preset, then flip any individual bot. The robots.txt below updates live. Copy it or download it and upload it to the root of your site so it answers at yourdomain.com/robots.txt.
AI search and assistant bots
These decide whether ChatGPT, Claude, Perplexity, Siri, and Copilot can read and cite you. Blocking them costs you visibility in AI answers.
OAI-SearchBot OpenAI
Indexes pages for ChatGPT search results and citations.
ChatGPT-User OpenAI
Fetches a page live when a ChatGPT user asks about it.
Claude-SearchBot Anthropic
Indexes pages for Claude's web search answers.
Claude-User Anthropic
Fetches a page live when a Claude user shares a link.
PerplexityBot Perplexity
Indexes pages for Perplexity answers.
Perplexity-User Perplexity
Fetches a page live for a Perplexity user.
Applebot Apple
Powers Siri and Spotlight suggestions.
Googlebot Google
Google Search, which also feeds AI Overviews.
Bingbot Microsoft
Bing search, which feeds Copilot answers.
AI training bots
These collect your pages as training data for future models. Blocking them donates nothing and costs you no search visibility.
GPTBot OpenAI
Crawls pages to train future GPT models.
ClaudeBot Anthropic
Bulk crawl used to gather training data.
Google-Extended Google
Opt-out token for Gemini model training.
Applebot-Extended Apple
Opt-out token for Apple model training.
CCBot Common Crawl
Builds the open dataset many labs train from.
Bytespider ByteDance
High-volume crawl feeding ByteDance models.
Meta-ExternalAgent Meta
Crawls pages for Meta AI model training.
Amazonbot Amazon
Crawl supporting Alexa and Amazon AI features.
cohere-ai Cohere
Crawl for Cohere's enterprise model training.
# AI crawler policy # Built with Tan Alto's AI Crawler Policy Generator # https://tanalto.com/tools/ai-crawler-policy-generator # --- AI search and assistant bots (visibility in AI answers) --- User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Applebot Allow: / User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / # --- AI training bots (model training data collection) --- User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: Amazonbot Disallow: / User-agent: cohere-ai Disallow: /
Upload this file to your site root so it answers at yourdomain.com/robots.txt. If you already have a robots.txt, merge these blocks into it instead of replacing the file, since your existing Sitemap and Googlebot rules still matter.
The 2026 bot roster, decoded
What each bot actually does, and what blocking it costs you. The single most common mistake we see is blocking a search bot while trying to block training, which removes the site from AI answers and does nothing for protection.
| Bot | Owner | Job | If you block it |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Indexes pages for ChatGPT search results and citations. | ChatGPT search stops showing and citing your pages. |
| ChatGPT-User | OpenAI | Fetches a page live when a ChatGPT user asks about it. | ChatGPT cannot open your pages during conversations. |
| Claude-SearchBot | Anthropic | Indexes pages for Claude's web search answers. | Claude search results drop your pages. |
| Claude-User | Anthropic | Fetches a page live when a Claude user shares a link. | Claude cannot read pages users point it to. |
| PerplexityBot | Perplexity | Indexes pages for Perplexity answers. | Perplexity stops citing you. |
| Perplexity-User | Perplexity | Fetches a page live for a Perplexity user. | Perplexity cannot open your pages on request. |
| Applebot | Apple | Powers Siri and Spotlight suggestions. | Siri and Spotlight lose your pages. |
| Googlebot | Google Search, which also feeds AI Overviews. | You leave Google entirely. Almost never block this one. | |
| Bingbot | Microsoft | Bing search, which feeds Copilot answers. | You leave Bing and Copilot answers. |
| GPTBot | OpenAI | Crawls pages to train future GPT models. | Your content stays out of future OpenAI training runs. |
| ClaudeBot | Anthropic | Bulk crawl used to gather training data. | Your content stays out of future Anthropic training runs. |
| Google-Extended | Opt-out token for Gemini model training. | Gemini training opts out. Search and AI Overviews are NOT affected. | |
| Applebot-Extended | Apple | Opt-out token for Apple model training. | Apple training opts out while Applebot keeps Siri working. |
| CCBot | Common Crawl | Builds the open dataset many labs train from. | Your pages stop flowing into downstream training sets. |
| Bytespider | ByteDance | High-volume crawl feeding ByteDance models. | ByteDance stops collecting your pages. |
| Meta-ExternalAgent | Meta | Crawls pages for Meta AI model training. | Meta stops collecting your pages for training. |
| Amazonbot | Amazon | Crawl supporting Alexa and Amazon AI features. | Amazon's AI features lose access to your pages. |
| cohere-ai | Cohere | Crawl for Cohere's enterprise model training. | Cohere stops collecting your pages. |
For the technical reader
robots.txt is a request, not enforcement
The major labs honor it and publish their user agents, but nothing stops a bad actor from ignoring the file. For content that genuinely cannot leak, pair the policy with CDN or WAF rules that return 403 to disallowed user agents, and monitor logs for spoofing.
Verify what the bot actually receives
CDNs, plugins, and frameworks sometimes serve a cached or rewritten robots.txt. Check the live file as the bot, then watch the logs:
# Fetch robots.txt the way the bot sees it
curl -A "GPTBot" https://yoursite.com/robots.txt
# Check whether a blocked bot still crawls (server logs)
grep "GPTBot" /var/log/nginx/access.log | tail -20AI Overviews are a different control
Google-Extended does not remove you from AI Overviews; they come from the Search index via Googlebot. The levers for that are nosnippet, max-snippet, and data-nosnippet, which trade snippet visibility in regular results too. Full walkthrough, including the llms.txt question, on our AI crawler policy page, and our GEO Readiness Scanner checks whether AI bots can read your pages at all.
Frequently asked questions
Does robots.txt actually block AI crawlers?+
It is a voluntary standard, not a lock. OpenAI, Anthropic, Google, Perplexity, and Meta publicly commit to honoring it, and for those bots it works. Bad actors ignore it, which is why sensitive content also needs server-side rules at the CDN or WAF layer. Think of robots.txt as the posted policy and the WAF as the security guard.
If I block GPTBot, will ChatGPT stop citing my site?+
No, and this is the split most sites get wrong. GPTBot collects training data only. ChatGPT search visibility runs through OAI-SearchBot (the index) and ChatGPT-User (live fetches). Block GPTBot and allow those two, and you opt out of training while staying fully citable. The recommended preset in this generator does exactly that.
Does blocking Google-Extended remove me from AI Overviews?+
No. AI Overviews are built from the regular Google Search index, which Googlebot crawls. Google-Extended only opts you out of Gemini model training; Google has stated it does not affect Search or AI Overviews. To control AI Overviews you use nosnippet or max-snippet directives, not robots.txt.
What is llms.txt, and is it the same thing?+
No. llms.txt is a proposed markdown file that points language models at your most useful pages, closer to a sitemap than a policy file. It grants and denies nothing. robots.txt is still the only widely honored access-control mechanism, which is what this generator builds.
Will blocking training bots hurt my Google rankings?+
There is no evidence of that. Ranking depends on Googlebot and the search index, and none of the training bots feed it. Google has explicitly said Google-Extended is independent of ranking, and blocking GPTBot or CCBot has no search-side effect.
How do I verify the policy is live and working?+
Fetch your robots.txt as the bot would: curl -A 'GPTBot' https://yoursite.com/robots.txt and confirm the rules you generated are what the server returns. Then watch server logs for a few weeks to see which bots respect the policy. Our GEO Readiness Scanner also flags crawl-access problems automatically.
Want the policy enforced, not just posted?
We implement the full stack: robots.txt, WAF rules, log monitoring, and the AI visibility side so the bots you want actually cite you.
