Most companies treat every AI bot as one thing and make the wrong call because of it. A crawler that pulls your pages into a training set is not the same as a bot that fetches your page the instant a user asks ChatGPT a question. Lump the two together and you often block the exact traffic that would have made you visible.
This is the single most common technical finding we run into: a robots.txt that blanket-denies every AI user-agent, written on the assumption that this is the safe choice. It is not. This piece sorts out which bot does what, and shows how to draw the line on purpose.
Which AI crawlers visit my website?
Five operators account for most of them: OpenAI, Anthropic, Perplexity, Google, and ByteDance. Each runs several bots with different jobs, and that distinction is what decides your visibility.
One bot collects text to train future models. Another indexes pages for AI search. A third fetches a page live because a user just asked a question whose answer sits on the open web. These three jobs run under separate names, and in robots.txt you can address each name on its own.
Training or retrieval: the difference that decides visibility
Training means your content feeds the model and may later surface as a passing mention with no source. Retrieval means the model reads your page at the moment of the question and can cite it with a link.
Those two cases hang on different bots. Block training and you do not vanish from answers, as long as the retrieval bots stay allowed. Block everything and you forfeit the citation, the stage where a user sees your source with a link attached. Mention, citation, and recommendation are three separate things, and the retrieval bot is the precondition for the middle one.
What does each bot do?
Here is the mapping, grouped by operator. The names are the exact tokens you write after User-agent: in robots.txt.
- OpenAI:
GPTBotgathers training data.OAI-SearchBotindexes for AI search in ChatGPT.ChatGPT-Userfetches a page when a user lets ChatGPT browse. - Anthropic:
ClaudeBotis the training crawler.Claude-SearchBotindexes for search.Claude-Userretrieves a page on a direct user request. - Perplexity:
PerplexityBotindexes,Perplexity-Userfetches at the moment of the question. - Google:
Google-Extendedonly controls whether your content trains Gemini and the Vertex AI models. It has nothing to do with ordinary Google Search, which still runs onGooglebot. So a Disallow forGoogle-Extendeddoes not remove you from the AI overviews in Search, because those are served throughGooglebot. - ByteDance:
Bytespidercollects data for the company behind TikTok. This bot has a reputation for not always honoring robots.txt, so do not rely on the directive alone with it.
The point that matters: a Disallow for GPTBot alone does not block OAI-SearchBot or ChatGPT-User. Every name needs its own line.
robots.txt: block training, allow retrieval
If your goal is "not in the training set, but still findable," block the training bots and let the retrieval and search bots through. Here is a template:
# Do not allow model training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# Allow answer-time retrieval and AI search
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: Perplexity-User
Allow: /If instead you want maximum findability, drop the training blocks. If you want everything shut, set Disallow: / for each of the names above. The rule is to decide per job, not per vendor.
The most common mistake: blanket blocking
Many sites block every AI bot because a template online told them to. According to Cloudflare (August 2025), more than 2.5 million websites block AI training entirely, and a good share of them accidentally block the retrieval bots at the same time.
The result: the company no longer appears as a cited source in AI answers with web search, even though that is what it wanted. It confused training with retrieval. If you want training protection but also want to be named and linked when a customer asks, you have to split the two cases, or your own robots.txt works against your own marketing.
What robots.txt cannot do
robots.txt is a request, not a fence. Reputable operators like OpenAI and Anthropic follow it, but the file enforces nothing, and some crawlers ignore it.
And even a perfectly placed Allow guarantees no citation. It is the precondition, nothing more. Whether a model ends up naming, citing, or recommending you depends on many further factors: your page structure, third-party sources about you, the user's exact question. robots.txt clears the first obstacle. For the rest you need content a model is willing to cite in the first place.
From crawler setup to measured visibility
Getting robots.txt right is the cheapest lever there is, and still only the start. Whether the change works shows only once you measure how often the major AI systems name and cite you before and after.
How a clean measurement is built is described on our page about AI visibility. Before you touch robots.txt, it is worth running the self-test that tells you in a quarter of an hour whether a problem exists at all.
If you want to know which bots visit your site today and what your robots.txt actually permits right now, write to us via the contact page. We look at your file and your logs and tell you where you are standing in your own way.