Should You Block GPTBot and Other AI Crawlers in robots.txt?
For most businesses, allow AI search and user-triggered bots (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot) so you can be cited, and decide separately whether to allow training bots (GPTBot, ClaudeBot, Google-Extended, CCBot). Blocking everything protects content but removes you from AI answers.
Why does this decision matter now?
A growing share of research now starts in ChatGPT, Gemini, Perplexity and Copilot. These assistants can only cite pages their crawlers are allowed to fetch. At the same time, publishers have good reasons to limit unpaid use of their work for model training. The good news is that the major AI companies run separate bots for different jobs, so you do not have to choose all or nothing.
Which AI crawlers exist and what do they do?
| User-agent | Company | Purpose |
|---|---|---|
| GPTBot | OpenAI | Training data |
| OAI-SearchBot | OpenAI | ChatGPT search results |
| ChatGPT-User | OpenAI | Live fetches when a user asks |
| ClaudeBot | Anthropic | Training data |
| Claude-SearchBot / Claude-User | Anthropic | Search and user-triggered fetches |
| Google-Extended | Gemini training and grounding control | |
| PerplexityBot / Perplexity-User | Perplexity | Search index and live fetches |
| Applebot-Extended | Apple | Apple Intelligence training control |
| CCBot | Common Crawl | Open dataset used by many models |
| Bytespider, Meta-ExternalAgent, Amazonbot | ByteDance, Meta, Amazon | Training and assistants |
See exactly which of these your current robots.txt allows with the free AI Crawler Checker.
When should you allow AI crawlers?
- You sell services, products or software and want to be recommended.
- Your content is marketing content whose job is to be seen.
- You compete in markets where buyers ask AI for shortlists.
When does blocking make sense?
- Your content is the product: paid research, courses, premium journalism.
- You have licensing negotiations with AI companies.
- Private or user-generated areas that should never be collected.
What is a sensible middle-ground robots.txt?
This setup blocks training bots and keeps AI search visibility:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
User-agent: Applebot-Extended
Disallow: /
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Claude-SearchBot
Allow: /
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap_index.xml
Generate your own version with presets in our AI Robots.txt Generator. For most of our clients in the USA, UK, Canada, Australia and India, we recommend allowing everything, because visibility is worth more than the theoretical cost of training use.
What common mistakes block AI bots by accident?
- Security plugins and CDNs with “block AI bots” toggles switched on by default.
- A blanket Disallow left over from a staging site.
- Grouping errors, such as a blank line splitting user-agent lines from their rules.
- Forgetting that a named group overrides
*: if you add a GPTBot group, it ignores your general rules.
How do you verify your setup?
Check robots.txt with a parser that follows the official precedence rules (longest match wins, Allow wins ties), then review server logs for each bot’s user-agent over a few weeks. If you use Cloudflare or another CDN, also review its bot settings, because they can block requests before robots.txt is read.
Need a second opinion? Our AI SEO team audits crawl access, robots rules and AI visibility for a fixed fee. Contact us.
Frequently asked questions
What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects public content that may be used to train OpenAI models. OAI-SearchBot fetches pages to show and link them in ChatGPT search results. Blocking GPTBot does not block OAI-SearchBot.
Does blocking Google-Extended affect Google rankings?
No. Google-Extended is a control token for Gemini training and grounding. Blocking it does not change how Googlebot crawls or ranks your pages in Search.
Do all AI crawlers respect robots.txt?
The major providers state that their crawlers follow robots.txt. Some less reputable scrapers do not, which is why firewall or CDN rules are used for enforcement.
Should news publishers block AI crawlers?
Many publishers block training bots while negotiating licences, but keep search and user-triggered agents allowed so their articles can still be cited with links.
Keep reading
Related insights
AI SEOllms.txt Explained: What It Is and Does Your Website Need One?
llms.txt is a small Markdown file that tells AI models which pages matter. Here is the format, a working example…
Read article
AI SEOHow to Get Your Business Recommended by ChatGPT (2026 Guide)
ChatGPT recommends brands it can find, understand and verify. This 9-step playbook shows how to become one of them, with…
Read article
AI SEOAEO vs GEO vs SEO: What Is the Difference in 2026?
SEO, AEO and GEO are not rivals. They are three layers of the same search strategy. Here is what each…
Read article