robots.txt lets you tell AI crawlers what they may fetch, but the same file controls two very different things. One set of crawlers reads your pages to train future models. Another set fetches your pages live to write today's answer and cite you inside it. Block the first and you opt out of training; block the second and you quietly disappear from that engine's answers.
Three jobs hide in one file
AI crawlers are not interchangeable. The user-agents that visit your site fall into three groups by purpose. Training crawlers read pages to build the datasets future models learn from. Retrieval or search crawlers keep an index the engine consults when it answers. User-fetch agents open a page in real time because someone asked a question that needed it.
The distinction matters because blocking each one has a different consequence. Block a training crawler and your content is less likely to be absorbed into the next model version, which changes nothing about today's answers. Block a retrieval or user-fetch crawler and the engine can no longer open your page to quote it, so you drop out of the answers it writes right now.
This is the access-versus-visibility tradeoff. One Disallow line can mean do not train on me, and the next identical-looking line can mean do not cite me, depending only on which agent it targets. The file gives no warning about which you just chose, so reading the user-agent name is the only way to tell them apart.
A field guide to the AI user-agents
OpenAI runs three. GPTBot collects content for training. OAI-SearchBot surfaces and links pages inside ChatGPT search. ChatGPT-User fetches a specific page when a person, or a task, asks ChatGPT to read it. Blocking GPTBot is a training choice; blocking the other two removes you from ChatGPT's live answers and citations.
Anthropic runs ClaudeBot for general crawling and Claude-User for fetches made on a user's behalf. Perplexity runs PerplexityBot to build its index and Perplexity-User to open pages while answering a live question. In both cases the -User agent is the one tied most directly to being cited in a response, so it is the risky one to block.
Google is the exception worth memorizing. Googlebot crawls for Search, and Google's AI Overviews draw on that same Search index, so Googlebot is doing double duty. Google-Extended is not a crawler at all: it is a separate control token you can disallow to opt out of training Gemini and Vertex models, and disallowing it does not affect Search ranking or AI Overviews. Bingbot plays a similar double role, since the Bing index feeds Microsoft Copilot.
Several general crawlers round out the list. CCBot belongs to Common Crawl, whose public datasets have long been used to train many models. Amazonbot and Bytespider (ByteDance) crawl broadly and are commonly associated with training and product data collection. Names and behavior change over time, so treat any list as a snapshot and re-check each operator's published docs before you write rules.
Writing rules per user-agent
robots.txt groups rules under User-agent lines. A group starts with one or more User-agent entries and is followed by Disallow and Allow directives that apply only to those agents. A line of User-agent: * is the fallback for any crawler you did not name. Most crawlers obey the most specific group that matches their token, so a group written for GPTBot beats the wildcard group for GPTBot.
To block training on OpenAI while keeping the search and user agents, you name only the training bot. A group of User-agent: GPTBot with Disallow: / stops training crawling, while OAI-SearchBot and ChatGPT-User, left unlisted, stay allowed. The same pattern with User-agent: Google-Extended and Disallow: / opts you out of Gemini training while Googlebot, and therefore Search and AI Overviews, is untouched.
The common mistake is a blanket rule. A User-agent: * with Disallow: / (or a broad path block) catches every AI agent you did not explicitly allow, including the retrieval and user-fetch crawlers that put you in answers. If you want to be strict about training but visible in answers, list the training bots by name and leave the search and user agents out of your Disallow.
Control access, not the past
robots.txt is an instruction, not a wall. Well-behaved crawlers from the major AI companies honor it, but the file requests compliance rather than enforcing it, and it does nothing to content that was already collected. Anything crawled before you added a Disallow, or absorbed through a third-party dataset like Common Crawl, is out of the file's reach. This is why you cannot fully control training: you control future access, not the past.
Blocking training crawlers is a legitimate choice. Publishers, brands with licensing positions, and teams protecting original research all have reasons to opt out, and doing so costs no live visibility as long as you block only the training agents. The problem is not blocking; it is blocking the wrong agent by accident.
So decide on purpose. Separate the two questions before you write a single line: do I want my content used for training, and do I want to appear in AI answers? Map each operator's agents to those two answers, then write Disallow rules that match your intent. When in doubt, allow the search and user-fetch agents, because those are the ones that decide whether an engine can name you today.