Robots.txt for AI Crawlers: GPTBot, ClaudeBot, PerplexityBot, Google-Extended

A definitive 2026 reference for AI crawler user agents — what each one does, when to allow it, and when to block it. Includes a copy-pasteable robots.txt.

By ApexEcho AI · Published 2026-04-12 · 8 min read
AEOTechnical

Summary — In 2026 there are roughly a dozen AI crawler user agents that matter. The default for almost every brand should be: allow them all. Blocking removes you from training and live retrieval, which is the opposite of what AEO wants. There are narrow exceptions — paywalled content, regulated content, and proprietary datasets — but for the typical marketing site, allow is the right call.

For why crawler access matters, see What is AEO?.

The crawlers that matter

User agent Operator What it does Default recommendation
GPTBot OpenAI Crawls for ChatGPT training Allow
OAI-SearchBot OpenAI Live retrieval for ChatGPT search Allow
ChatGPT-User OpenAI On-demand fetches when a user clicks a citation Allow
PerplexityBot Perplexity Indexing + live retrieval for Perplexity Allow
Perplexity-User Perplexity On-demand user fetch Allow
ClaudeBot Anthropic Crawls for Claude training Allow
anthropic-ai Anthropic Live retrieval for Claude Allow
Google-Extended Google Opt-out for Gemini / AI Overviews training Allow (don't block)
Googlebot Google Classic search crawl + AI Overview retrieval Allow (always)
Bingbot Microsoft Bing + Copilot retrieval Allow
Applebot-Extended Apple Opt-out for Apple Intelligence training Allow
Bytespider ByteDance Crawls for Doubao and related models Allow if relevant market
Meta-ExternalAgent Meta Crawls for Meta AI Allow if relevant market

This list will continue to grow. Re-audit your robots.txt at least quarterly.

A copy-pasteable robots.txt for the typical brand

User-agent: *
Allow: /

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: Meta-ExternalAgent
Allow: /

Sitemap: https://www.example.com/sitemap.xml

When to block

There are three legitimate reasons to block one or more AI crawlers:

  1. Paid / paywalled content. If your archive is a revenue product, blocking training crawlers protects it. (Live-retrieval crawlers are a separate question — many publishers block training but allow retrieval.)
  2. Regulated or proprietary content. Medical records, legal documents, or proprietary datasets that should never end up in a training corpus.
  3. Internal-only sections. Authenticated areas, customer dashboards, admin pages. These should already be behind auth, but explicit disallow is belt-and-suspenders.

For category 1, you can split the strategy:

# Block training but allow live retrieval
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

This pattern says: "Don't ingest us into training, but if a user clicks a citation, fetch us."

When NOT to block

Most brands that block AI crawlers do so because of a 2023 panic narrative ("AI is stealing our content!"). For a marketing site, this is almost always self-defeating:

  • You can't be cited if you're not crawled.
  • You can't be in training data if you're not crawled.
  • Competitors who do allow crawling soak up your category's share of voice.

If your goal is AEO visibility, blocking AI crawlers is functionally the same as blocking Googlebot was in 2010.

The most common robots.txt mistakes

  1. Inheriting a 2023 default. Many CMSs and security plugins added blanket AI bot blocks 2–3 years ago. Audit and remove.
  2. Blocking with User-agent: * and forgetting the exceptions. A single broad disallow plus per-bot allows is fragile; explicit per-bot blocks are clearer.
  3. Confusing crawl budget with access. AI crawlers don't have the same crawl-budget pressure as Googlebot; you don't need to disallow your archive to "save budget."
  4. Forgetting the sitemap reference. Always include Sitemap: at the end. AI crawlers use it.
  5. Blocking via firewall rather than robots.txt. Some CDN rulesets block by user agent. Audit there too — robots.txt is only one layer.

How to verify your robots.txt is doing what you think

  1. Visit https://your-domain.com/robots.txt and read it line by line.
  2. For each AI crawler you care about, run curl -A "GPTBot" https://your-domain.com/some-page and confirm it returns the page.
  3. Check your CDN or WAF dashboard for blocked-by-user-agent rules.
  4. Re-audit after any platform migration or theme update — robots.txt regressions are silent and common.

meta robots and X-Robots-Tag

For per-page control beyond robots.txt, you can use:

  • <meta name="robots" content="noindex"> to hide a page from indexing
  • <meta name="GPTBot" content="noindex"> for GPTBot-specific control
  • X-Robots-Tag HTTP header for non-HTML files (PDFs, etc.)

In practice, page-level controls are over-engineered for most teams. Get robots.txt right first.

The bottom line

For 95% of brands, the right move is to allow every major AI crawler and stop worrying about it. The competitive cost of blocking is far higher than the theoretical risk of being trained on.

For the next layer of work — schema, content, and measurement — see the 25-step AEO checklist and the best AEO tools.

Run a free AEO scan and confirm your robots.txt isn't quietly costing you visibility.