How do I allow ChatGPT to crawl my site?

Make sure robots.txt does not disallow OAI-SearchBot (ChatGPT search) or GPTBot (training). A group with User-agent: OAI-SearchBot and Allow: / is explicit.

Does blocking Google-Extended remove me from AI Overviews?

No. Google-Extended only controls use of your content for Gemini model training. AI Overviews are built from the normal Googlebot index.

Can I appear in AI answers but opt out of training?

Yes. Allow the search and user-fetch agents (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot) and disallow the training agents (GPTBot, ClaudeBot, Google-Extended).

robots.txt for AI crawlers: who to allow, and how

If an AI assistant can't fetch your pages, it can't quote or recommend you.

The AI user agents that matter

Most AI companies now run separate agents for training and for live answers. That split lets you choose.

OpenAI: GPTBot (training), OAI-SearchBot (ChatGPT search results), ChatGPT-User (a user asked ChatGPT to open your page).
Anthropic: ClaudeBot (training), Claude-SearchBot (search), Claude-User (user-requested fetches).
Perplexity: PerplexityBot (indexing for answers), Perplexity-User (user-requested fetches).
Google: Google-Extended is not a crawler but a robots.txt token that controls Gemini training. AI Overviews come from the normal Googlebot index.
Apple: Applebot-Extended controls Apple Intelligence training, the same way.

Option A: allow everything (maximum AI visibility)

User-agent: *
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

If nothing in your file names an AI agent, this is already your situation. Explicit groups for the agents below make the intent obvious to anyone auditing the file.

Option B: appear in AI answers, opt out of training

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

User-agent: *
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

Common ways sites block AI by accident

A CDN or host setting, not your file

Cloudflare and some hosts offer one-click "block AI bots" or managed robots.txt options, and some turn them on by default for new domains. These can block AI agents at the edge even when your own robots.txt allows them. Check your CDN dashboard, then fetch yoursite.com/robots.txt to see what is actually served.

A blanket Disallow left over from staging

User-agent: * followed by Disallow: / blocks every crawler, search engines included. It is the single most common reason a live site is invisible.

Rules in the wrong group

A crawler obeys only the most specific group that names it. If you add a User-agent: GPTBot group, the rules under User-agent: * no longer apply to GPTBot, so repeat anything it still needs.

Frequently asked questions

Does blocking GPTBot remove me from ChatGPT answers?

Not by itself: ChatGPT search uses OAI-SearchBot. But a page OpenAI has never been allowed to see is a page it knows less about, so think before blocking.

Do AI crawlers respect robots.txt?

The major vendors above say they do. robots.txt is a request, not a lock: use authentication for anything genuinely private.

Check yours

Aeoray reads your live robots.txt and reports whether GPTBot, ClaudeBot, PerplexityBot and Google-Extended are allowed, together with six other AI-visibility signals.

Scan your site free

More guides: how to add llms.txt · JSON-LD for AI answers · the full AEO checklist.