In short: For ChatGPT, Claude and Perplexity to cite your site, their search AI crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) must be able to reach your pages. Check three layers: robots.txt, Cloudflare or another CDN, and your server firewall and WordPress security plugin. Training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended) are a separate decision – blocking them does not remove you from AI search. Verify the result in your server logs and with curl -A.
Many businesses blocked AI crawlers at some point “just in case” – or a hosting, CDN or security plugin default did it for them. The result: ChatGPT and Perplexity cannot read the site and recommend a competitor instead. This guide covers which AI crawlers exist, what each one does and how to open your site to them properly. For a quick list of checks, see our AI visibility checklist.
Which AI crawlers visit your site?
Every major AI company runs several crawlers. The key is to separate three roles: search (your page appears and is cited in AI answers), training (your content may be used to train a model) and user fetch (the AI opens your page because a user asked something right now).
| Company | Crawler | Role | Follows robots.txt? |
|---|---|---|---|
| OpenAI | OAI-SearchBot | ChatGPT search and citations | Yes |
| OpenAI | GPTBot | Model training | Yes |
| OpenAI | ChatGPT-User | User-initiated fetch | Not necessarily |
| OpenAI | OAI-AdsBot | Checks landing pages of ChatGPT ads | Visits only submitted ad pages |
| Anthropic | Claude-SearchBot | Claude search index | Yes |
| Anthropic | ClaudeBot | Model training | Yes |
| Anthropic | Claude-User | User-initiated fetch | Yes, according to Anthropic |
| Perplexity | PerplexityBot | Perplexity search (not training) | Yes |
| Perplexity | Perplexity-User | User-initiated fetch | Generally no |
| Googlebot | Search, AI Overviews, AI Mode | Yes | |
| Google-Extended | Gemini training and grounding | Yes (a token only, no separate crawler) | |
| Microsoft | Bingbot | Bing and Copilot | Yes |
| Apple | Applebot | Siri and Spotlight search | Yes |
| Apple | Applebot-Extended | Apple model training | Yes (a token only) |
Which AI crawlers should you allow and which can you block?
If you want to appear in AI answers, the search crawlers must be allowed. OpenAI says plainly that sites opted out of OAI-SearchBot are not shown in ChatGPT search answers. Google AI Overviews and AI Mode are handled by the regular Googlebot – blocking Google-Extended does not affect them.
Training crawlers are your call. If your content is your product (paid guides, research), you can block GPTBot, ClaudeBot, Google-Extended and Applebot-Extended. For a typical company website whose job is to win customers, leaving everything open makes sense.
How do you set up robots.txt for AI crawlers?
robots.txt always lives in the site root, for example yoursite.com/robots.txt. Every subdomain needs its own file. Here are two ready-made variants.
Variant A: everything open
# Variant A: everything open
User-agent: *
Allow: /
Sitemap: https://example.ee/sitemap_index.xmlVariant B: AI search yes, AI training no
# Variant B: AI search yes, AI training no
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: *
Allow: /
Sitemap: https://example.ee/sitemap_index.xmlWatch out: if robots.txt contains User-agent: * with Disallow: /, every crawler is blocked, including those not named separately. This often happens with settings copied from a staging site. In WordPress, also check Settings → Reading → Discourage search engines from indexing this site.
Does Cloudflare block AI crawlers?
It can. Cloudflare has dedicated settings for blocking AI crawlers and for bot management, and they work independently of robots.txt. If they are on, a crawler gets a 403 even when robots.txt allows it. Check:
- in the Cloudflare dashboard under Security → Bots, whether AI crawler blocking is switched on;
- whether Bot Fight Mode or Super Bot Fight Mode is also blocking verified bots;
- your WAF rules – is there a custom rule blocking user agents that contain “bot”;
- the security log (Security → Events): search for OAI-SearchBot, Claude-SearchBot and PerplexityBot and see what happened to them.
If you want to let AI search crawlers in but keep training crawlers out, do it in robots.txt or with a precise WAF rule, not with a blanket “block all AI bots” switch.
Can a firewall or WordPress security plugin stop AI crawlers?
Yes. Hosting firewalls, ModSecurity rules and WordPress security plugins often limit bots that send many requests in a short time. What to look at:
- rate limiting rules and automatic IP bans;
- country blocking – AI crawlers often come from US data centres;
- “bad bot” lists that sometimes include AI crawlers too.
OpenAI and Perplexity publish the IP ranges of their crawlers. If you need to allow bots by IP, use those official lists rather than random forum posts.
How do you check whether AI crawlers can reach your site?
The simplest test is a command-line request with the crawler’s user agent:
curl -I -A "OAI-SearchBot" https://example.ee/
curl -I -A "Claude-SearchBot" https://example.ee/
curl -I -A "PerplexityBot" https://example.ee/A 200 response is good. A 403 or 503 means something is blocking. This test shows firewall and CDN rules, but not everything: Cloudflare can tell a real crawler from a spoofed user agent by IP. So always check your server logs too: search for the same names and their response codes over the last 30 days.
Do user-initiated fetchers follow robots.txt?
Not always. ChatGPT-User and Perplexity-User open a page because a user asked something right now. These companies treat it as a human action rather than automated crawling, so robots.txt may not stop them. Anthropic says Claude-User does follow robots.txt. If you must stop one of them for sure, you need a firewall rule – but for a company website that is rarely wise: those requests mean someone asked about you.
How fast do changes take effect?
According to OpenAI, a robots.txt change for OAI-SearchBot takes effect in about 24 hours. Showing up in AI answers takes longer: the page has to be indexed, answer the question and come from a trusted source. The next step is Schema markup, which tells AI directly who you are, and llms.txt as an optional extra.
AI crawler setup in short
| Layer | What to check | Who |
|---|---|---|
| robots.txt | OAI-SearchBot, Claude-SearchBot, PerplexityBot allowed | You / developer |
| WordPress | “Discourage search engines” is off | You |
| Cloudflare / CDN | AI crawler blocking, Bot Fight Mode, WAF | Developer |
| Server | Firewall, rate limiting, country blocks | Hosting / developer |
| Verification | curl -A and server logs: response 200 | Developer |
Frequently asked questions
Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot collects training data. Appearing in ChatGPT search is controlled by OAI-SearchBot. If that one is allowed, you can block GPTBot.
How do I know if Cloudflare is blocking AI crawlers?
Look under Security → Bots in the Cloudflare dashboard, and check the Security → Events log for OAI-SearchBot or PerplexityBot getting a 403.
Do I need to allow Google-Extended for Google AI Overviews?
No. According to Google, Google-Extended does not affect Google Search or AI Overviews. Those are handled by the regular Googlebot.
Do AI crawlers overload the server?
Usually not. If the load is noticeable, limit the request rate rather than blocking search crawlers entirely. Anthropic, for example, supports the Crawl-delay rule in robots.txt.
Not sure whether your site is open to AI crawlers? Run the free website audit, see our AI search visibility service or contact us – we will check your robots.txt, Cloudflare and server settings.
Sources
- OpenAI – Overview of OpenAI crawlers (OAI-SearchBot, GPTBot, ChatGPT-User, OAI-AdsBot)
- Anthropic – Does Anthropic crawl data from the web (ClaudeBot, Claude-SearchBot, Claude-User)
- Perplexity – Perplexity crawlers
- Google Search Central – Overview of Google crawlers and fetchers (Google-Extended)
- Google Search Central – AI features and your website
- Apple – About Applebot
- Cloudflare Docs – Bots
