AI Crawler Policy Generator: who trains on your content, who quotes it

Every AI company sends more than one bot, and they do different jobs. Some put you in front of buyers inside ChatGPT, Claude, and Perplexity answers. Others copy your pages into training data. This tool writes a robots.txt that welcomes the first group and turns the second away, in one click.

Also known as

The guest list on your front door

robots.txt is the sign on the door

It is a small text file that tells visiting bots what they may do on your site. Well-behaved companies read it and obey. It sets policy; it is not a lock, and it works because the major AI labs have promised to respect it.

Search bots are the food critics

They read your pages so AI assistants can recommend you by name when someone asks a question. Turning them away makes you invisible in AI answers, which is where more buyers now start.

Training bots are the photocopiers

They collect your writing to teach future models. That builds no visibility for you; it just donates your work. Most sites want critics in and photocopiers out. That is the preset we recommend.

Why this page matters, in one sentence: the wrong robots.txt either hides you from the AI answers your buyers are reading, or hands your content to training runs you never agreed to, and most sites accidentally do one of the two.

Build your policy

Pick a preset, then flip any individual bot. The robots.txt below updates live. Copy it or download it and upload it to the root of your site so it answers at yourdomain.com/robots.txt.

AI search and assistant bots

These decide whether ChatGPT, Claude, Perplexity, Siri, and Copilot can read and cite you. Blocking them costs you visibility in AI answers.

  • OAI-SearchBot OpenAI

    Indexes pages for ChatGPT search results and citations.

  • ChatGPT-User OpenAI

    Fetches a page live when a ChatGPT user asks about it.

  • Claude-SearchBot Anthropic

    Indexes pages for Claude's web search answers.

  • Claude-User Anthropic

    Fetches a page live when a Claude user shares a link.

  • PerplexityBot Perplexity

    Indexes pages for Perplexity answers.

  • Perplexity-User Perplexity

    Fetches a page live for a Perplexity user.

  • Applebot Apple

    Powers Siri and Spotlight suggestions.

  • Googlebot Google

    Google Search, which also feeds AI Overviews.

  • Bingbot Microsoft

    Bing search, which feeds Copilot answers.

AI training bots

These collect your pages as training data for future models. Blocking them donates nothing and costs you no search visibility.

  • GPTBot OpenAI

    Crawls pages to train future GPT models.

  • ClaudeBot Anthropic

    Bulk crawl used to gather training data.

  • Google-Extended Google

    Opt-out token for Gemini model training.

  • Applebot-Extended Apple

    Opt-out token for Apple model training.

  • CCBot Common Crawl

    Builds the open dataset many labs train from.

  • Bytespider ByteDance

    High-volume crawl feeding ByteDance models.

  • Meta-ExternalAgent Meta

    Crawls pages for Meta AI model training.

  • Amazonbot Amazon

    Crawl supporting Alexa and Amazon AI features.

  • cohere-ai Cohere

    Crawl for Cohere's enterprise model training.

robots.txt
# AI crawler policy
# Built with Tan Alto's AI Crawler Policy Generator
# https://tanalto.com/tools/ai-crawler-policy-generator

# --- AI search and assistant bots (visibility in AI answers) ---

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Applebot
Allow: /

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# --- AI training bots (model training data collection) ---

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: cohere-ai
Disallow: /

Upload this file to your site root so it answers at yourdomain.com/robots.txt. If you already have a robots.txt, merge these blocks into it instead of replacing the file, since your existing Sitemap and Googlebot rules still matter.

The 2026 bot roster, decoded

What each bot actually does, and what blocking it costs you. The single most common mistake we see is blocking a search bot while trying to block training, which removes the site from AI answers and does nothing for protection.

BotOwnerJobIf you block it
OAI-SearchBotOpenAIIndexes pages for ChatGPT search results and citations.ChatGPT search stops showing and citing your pages.
ChatGPT-UserOpenAIFetches a page live when a ChatGPT user asks about it.ChatGPT cannot open your pages during conversations.
Claude-SearchBotAnthropicIndexes pages for Claude's web search answers.Claude search results drop your pages.
Claude-UserAnthropicFetches a page live when a Claude user shares a link.Claude cannot read pages users point it to.
PerplexityBotPerplexityIndexes pages for Perplexity answers.Perplexity stops citing you.
Perplexity-UserPerplexityFetches a page live for a Perplexity user.Perplexity cannot open your pages on request.
ApplebotApplePowers Siri and Spotlight suggestions.Siri and Spotlight lose your pages.
GooglebotGoogleGoogle Search, which also feeds AI Overviews.You leave Google entirely. Almost never block this one.
BingbotMicrosoftBing search, which feeds Copilot answers.You leave Bing and Copilot answers.
GPTBotOpenAICrawls pages to train future GPT models.Your content stays out of future OpenAI training runs.
ClaudeBotAnthropicBulk crawl used to gather training data.Your content stays out of future Anthropic training runs.
Google-ExtendedGoogleOpt-out token for Gemini model training.Gemini training opts out. Search and AI Overviews are NOT affected.
Applebot-ExtendedAppleOpt-out token for Apple model training.Apple training opts out while Applebot keeps Siri working.
CCBotCommon CrawlBuilds the open dataset many labs train from.Your pages stop flowing into downstream training sets.
BytespiderByteDanceHigh-volume crawl feeding ByteDance models.ByteDance stops collecting your pages.
Meta-ExternalAgentMetaCrawls pages for Meta AI model training.Meta stops collecting your pages for training.
AmazonbotAmazonCrawl supporting Alexa and Amazon AI features.Amazon's AI features lose access to your pages.
cohere-aiCohereCrawl for Cohere's enterprise model training.Cohere stops collecting your pages.

For the technical reader

robots.txt is a request, not enforcement

The major labs honor it and publish their user agents, but nothing stops a bad actor from ignoring the file. For content that genuinely cannot leak, pair the policy with CDN or WAF rules that return 403 to disallowed user agents, and monitor logs for spoofing.

Verify what the bot actually receives

CDNs, plugins, and frameworks sometimes serve a cached or rewritten robots.txt. Check the live file as the bot, then watch the logs:

bashVerify your policy like a crawler
# Fetch robots.txt the way the bot sees it
curl -A "GPTBot" https://yoursite.com/robots.txt

# Check whether a blocked bot still crawls (server logs)
grep "GPTBot" /var/log/nginx/access.log | tail -20

AI Overviews are a different control

Google-Extended does not remove you from AI Overviews; they come from the Search index via Googlebot. The levers for that are nosnippet, max-snippet, and data-nosnippet, which trade snippet visibility in regular results too. Full walkthrough, including the llms.txt question, on our AI crawler policy page, and our GEO Readiness Scanner checks whether AI bots can read your pages at all.

Frequently asked questions

Does robots.txt actually block AI crawlers?+

It is a voluntary standard, not a lock. OpenAI, Anthropic, Google, Perplexity, and Meta publicly commit to honoring it, and for those bots it works. Bad actors ignore it, which is why sensitive content also needs server-side rules at the CDN or WAF layer. Think of robots.txt as the posted policy and the WAF as the security guard.

If I block GPTBot, will ChatGPT stop citing my site?+

No, and this is the split most sites get wrong. GPTBot collects training data only. ChatGPT search visibility runs through OAI-SearchBot (the index) and ChatGPT-User (live fetches). Block GPTBot and allow those two, and you opt out of training while staying fully citable. The recommended preset in this generator does exactly that.

Does blocking Google-Extended remove me from AI Overviews?+

No. AI Overviews are built from the regular Google Search index, which Googlebot crawls. Google-Extended only opts you out of Gemini model training; Google has stated it does not affect Search or AI Overviews. To control AI Overviews you use nosnippet or max-snippet directives, not robots.txt.

What is llms.txt, and is it the same thing?+

No. llms.txt is a proposed markdown file that points language models at your most useful pages, closer to a sitemap than a policy file. It grants and denies nothing. robots.txt is still the only widely honored access-control mechanism, which is what this generator builds.

Will blocking training bots hurt my Google rankings?+

There is no evidence of that. Ranking depends on Googlebot and the search index, and none of the training bots feed it. Google has explicitly said Google-Extended is independent of ranking, and blocking GPTBot or CCBot has no search-side effect.

How do I verify the policy is live and working?+

Fetch your robots.txt as the bot would: curl -A 'GPTBot' https://yoursite.com/robots.txt and confirm the rules you generated are what the server returns. Then watch server logs for a few weeks to see which bots respect the policy. Our GEO Readiness Scanner also flags crawl-access problems automatically.

Want the policy enforced, not just posted?

We implement the full stack: robots.txt, WAF rules, log monitoring, and the AI visibility side so the bots you want actually cite you.