AI Crawler Access Policy
GPTBot and OAI-SearchBot are different crawlers. Half the sites blocking one are accidentally blocking both, and losing citations over it.
Training bots and search bots are different crawlers now. Block one group, welcome the other, and stop losing AI citations to a robots.txt file nobody has read since 2023.
Training crawlers and search crawlers are different bots
The major AI vendors split their crawling in two. Training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended) collect content to build models and send nothing back. Search and retrieval crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Perplexity-User) fetch pages to answer questions and cite you as the source. Blocking the first group is a business decision. Blocking the second is invisibility.
The mistake we find in audits is structural: one long Disallow list that mixes both groups, written in 2023 when they were the same bots. In one 2026 dataset, half of the top sites blocking GPTBot also blocked OAI-SearchBot, mostly by accident. If your goal was no training but do cite me, read your robots.txt today.
The 2026 default policy
Our default for clients: allow the search crawlers explicitly, disallow the training crawlers explicitly, and keep Googlebot and Bingbot out of the crossfire with a clean wildcard group. A crawler obeys only its own most specific group, so naming a bot means the wildcard rules stop applying to it entirely. That is the mechanic behind most accidental blocks.
One Google-specific trap: Google-Extended controls training, not AI Overviews. You cannot robots.txt your way out of AI Overviews; that uses Search Console's controls or snippet rules. Confusing the two is how sites disappear from the answers they wanted to keep.
# AI search crawlers: allowed, so assistants can cite you
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
# AI training crawlers: opted out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
Disallow: /
# Everyone else, including Googlebot
User-agent: *
Disallow: /admin/
Disallow: /cart/
Sitemap: https://yoursite.com/sitemap.xmlTest the way the bot reads
Almost no AI crawler executes JavaScript. They read the initial HTML response and move on, which makes the JavaScript render gap absolute for them: client-rendered content is not delayed, it is absent. This is why our own properties prerender every public route, and why the acceptance test for AI visibility is a curl, not a browser.
Run the test with the bot's user agent and grep for your money paragraph. If it is not in the raw response, no policy file will save you: the door is open but the room is empty.
# Fetch as OAI-SearchBot and check for your actual body copy
curl -s -A "OAI-SearchBot" https://yoursite.com/services | grep -c "your money phrase"
# Same page as Claude's search bot
curl -s -A "Claude-SearchBot" https://yoursite.com/services | grep -c "your money phrase"
# 0 means the content only exists after JavaScript runs: invisiblellms.txt, WAF enforcement, and monitoring
llms.txt is a navigation aid for inference-time retrieval, a curated map of your best content, not a training opt-out and not a replacement for robots.txt. Some crawlers ignore robots.txt entirely (Bytespider is the famous one); those get handled at the WAF, not in a text file they will never read.
Then close the loop: track which AI answers cite you for the prompts that matter, monthly. Policy without measurement is vibes. We run prompt-level mention tracking on our own brand and install the same loop on client accounts.
# llms.txt at your domain root
# Tan Alto
> Growth marketing agency: SEO, AI search visibility, paid media, web builds.
## Core services
- [SEO and AI Search](https://yoursite.com/services/seo-ai-search)
- [Pricing](https://yoursite.com/pricing)
## Best explainers
- [Technical SEO checklist](https://yoursite.com/services/seo-ai-search/technical-seo/technical-seo-checklist)
What you get
- AI bot inventory from your access logs
- Training vs search robots.txt policy
- WAF rules for non-compliant crawlers
- AI citation monitoring on the pages that matter
Also within Technical SEO
Website Migration SEO
Redesigns, replatforms, and domain moves without the ranking collapse: redirect mapping, parity checks, and launch-week monitoring run as a discipline, not a prayer.
ExploreJavaScript SEO
React, Vue, and SPA sites that Google and AI crawlers can actually read: prerendering, server injection, and render-budget fixes from a team that ships on this stack daily.
ExploreThe Technical SEO Checklist
The exact 40-point checklist we run on every engagement, published: crawl, render, index, schema, speed, and the header mistakes that silently kill analytics.
ExploreCanonicalization and Hreflang
Canonical tags, alternate language clusters, and the guard scripts that keep a consolidation layer from quietly telling Google your whole site lives on someone else's server.
ExploreCore Web Vitals (LCP, INP, CLS)
Field-data performance work against the 2026 thresholds: LCP under 2.5s, INP under 200ms, CLS under 0.1, verified in CrUX and Search Console, not just Lighthouse.
ExploreCrawl Budget and Indexation
Sitemaps that only list canonical URLs, redirect chains collapsed to one hop, orphan pages found and rehomed, faceted navigation stopped before it becomes a million junk URLs.
ExploreStructured Data and Schema
The schema that still moves the needle in 2026: Organization with sameAs chains, Service, Article, and FAQ. Entity clarity for Google and the answer engines, validated in the served HTML.
ExploreQuestions we hear about ai crawler access policy
Not necessarily, because ChatGPT search runs on OAI-SearchBot and Bing's index, which are separate from training. Block GPTBot for training and keep OAI-SearchBot open for citations. Just verify your robots.txt does not smear both into one list.
Before you scroll away
Booking takes sixty seconds. Here's what happens next.
01
Pick a time
Thirty minutes, video or phone, your agenda. The booking form takes about a minute.
02
Get the benchmark
We audit your position before the call: search rankings, AI answers, competitors. You keep the findings either way.
03
Decide with numbers
A scoped plan and one number to hold us to. No obligation, no retainer pitch on a first call, no drip campaign after.
Set my AI crawler policy
One conversation, one plan, one number to hold us to. Book your free consultation today.
Prefer to write first? Send us a message · Email [email protected] · Call (727) 280-6953
