{"id":6954,"date":"2026-11-06T19:48:21","date_gmt":"2026-11-06T19:48:21","guid":{"rendered":"https:\/\/itsolutions.ee\/uncategorized\/open-your-site-to-ai-crawlers\/"},"modified":"2026-11-06T19:48:21","modified_gmt":"2026-11-06T19:48:21","slug":"open-your-site-to-ai-crawlers","status":"publish","type":"post","link":"https:\/\/itsolutions.ee\/en\/blog\/open-your-site-to-ai-crawlers\/","title":{"rendered":"How to Open Your Site to ChatGPT, Claude and Perplexity: robots.txt, Cloudflare and AI Crawlers"},"content":{"rendered":"
\n

In short:<\/strong> For ChatGPT, Claude and Perplexity to cite your site, their search AI crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) must be able to reach your pages. Check three layers: robots.txt, Cloudflare or another CDN, and your server firewall and WordPress security plugin. Training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended) are a separate decision \u2013 blocking them does not remove you from AI search. Verify the result in your server logs and with curl -A.<\/p>\n<\/div>\n

Many businesses blocked AI crawlers at some point \u201cjust in case\u201d \u2013 or a hosting, CDN or security plugin default did it for them. The result: ChatGPT and Perplexity cannot read the site and recommend a competitor instead. This guide covers which AI crawlers<\/strong> exist, what each one does and how to open your site to them properly. For a quick list of checks, see our AI visibility checklist<\/a>.<\/p>\n

Which AI crawlers visit your site?<\/h2>\n

Every major AI company runs several crawlers. The key is to separate three roles: search<\/strong> (your page appears and is cited in AI answers), training<\/strong> (your content may be used to train a model) and user fetch<\/strong> (the AI opens your page because a user asked something right now).<\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Company<\/th>\nCrawler<\/th>\nRole<\/th>\nFollows robots.txt?<\/th>\n<\/tr>\n<\/thead>\n
OpenAI<\/td>\nOAI-SearchBot<\/td>\nChatGPT search and citations<\/td>\nYes<\/td>\n<\/tr>\n
OpenAI<\/td>\nGPTBot<\/td>\nModel training<\/td>\nYes<\/td>\n<\/tr>\n
OpenAI<\/td>\nChatGPT-User<\/td>\nUser-initiated fetch<\/td>\nNot necessarily<\/td>\n<\/tr>\n
OpenAI<\/td>\nOAI-AdsBot<\/td>\nChecks landing pages of ChatGPT ads<\/td>\nVisits only submitted ad pages<\/td>\n<\/tr>\n
Anthropic<\/td>\nClaude-SearchBot<\/td>\nClaude search index<\/td>\nYes<\/td>\n<\/tr>\n
Anthropic<\/td>\nClaudeBot<\/td>\nModel training<\/td>\nYes<\/td>\n<\/tr>\n
Anthropic<\/td>\nClaude-User<\/td>\nUser-initiated fetch<\/td>\nYes, according to Anthropic<\/td>\n<\/tr>\n
Perplexity<\/td>\nPerplexityBot<\/td>\nPerplexity search (not training)<\/td>\nYes<\/td>\n<\/tr>\n
Perplexity<\/td>\nPerplexity-User<\/td>\nUser-initiated fetch<\/td>\nGenerally no<\/td>\n<\/tr>\n
Google<\/td>\nGooglebot<\/td>\nSearch, AI Overviews, AI Mode<\/td>\nYes<\/td>\n<\/tr>\n
Google<\/td>\nGoogle-Extended<\/td>\nGemini training and grounding<\/td>\nYes (a token only, no separate crawler)<\/td>\n<\/tr>\n
Microsoft<\/td>\nBingbot<\/td>\nBing and Copilot<\/td>\nYes<\/td>\n<\/tr>\n
Apple<\/td>\nApplebot<\/td>\nSiri and Spotlight search<\/td>\nYes<\/td>\n<\/tr>\n
Apple<\/td>\nApplebot-Extended<\/td>\nApple model training<\/td>\nYes (a token only)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Which AI crawlers should you allow and which can you block?<\/h2>\n

If you want to appear in AI answers, the search crawlers must be allowed<\/strong>. OpenAI says plainly that sites opted out of OAI-SearchBot are not shown in ChatGPT search answers. Google AI Overviews and AI Mode are handled by the regular Googlebot \u2013 blocking Google-Extended does not affect them.<\/p>\n

Training crawlers are your call. If your content is your product (paid guides, research), you can block GPTBot, ClaudeBot, Google-Extended and Applebot-Extended. For a typical company website whose job is to win customers, leaving everything open makes sense.<\/p>\n

How do you set up robots.txt for AI crawlers?<\/h2>\n

robots.txt always lives in the site root, for example yoursite.com\/robots.txt<\/code>. Every subdomain needs its own file. Here are two ready-made variants.<\/p>\n

Variant A: everything open<\/h3>\n
# Variant A: everything open\nUser-agent: *\nAllow: \/\n\nSitemap: https:\/\/example.ee\/sitemap_index.xml<\/code><\/pre>\n
Variant B: AI search yes, AI training no<\/h3>\n
# Variant B: AI search yes, AI training no\nUser-agent: OAI-SearchBot\nAllow: \/\n\nUser-agent: Claude-SearchBot\nAllow: \/\n\nUser-agent: PerplexityBot\nAllow: \/\n\nUser-agent: GPTBot\nDisallow: \/\n\nUser-agent: ClaudeBot\nDisallow: \/\n\nUser-agent: Google-Extended\nDisallow: \/\n\nUser-agent: Applebot-Extended\nDisallow: \/\n\nUser-agent: *\nAllow: \/\n\nSitemap: https:\/\/example.ee\/sitemap_index.xml<\/code><\/pre>\n
Watch out: if robots.txt contains User-agent: *<\/code> with Disallow: \/<\/code>, every crawler is blocked, including those not named separately. This often happens with settings copied from a staging site. In WordPress, also check Settings \u2192 Reading \u2192 Discourage search engines from indexing this site<\/em>.<\/p>\n
Does Cloudflare block AI crawlers?<\/h2>\n
It can. Cloudflare has dedicated settings for blocking AI crawlers and for bot management, and they work independently of robots.txt. If they are on, a crawler gets a 403 even when robots.txt allows it. Check:<\/p>\n
    \n
  1. in the Cloudflare dashboard under Security \u2192 Bots<\/em>, whether AI crawler blocking is switched on;<\/li>\n
  2. whether Bot Fight Mode<\/em> or Super Bot Fight Mode<\/em> is also blocking verified bots;<\/li>\n
  3. your WAF rules \u2013 is there a custom rule blocking user agents that contain \u201cbot\u201d;<\/li>\n
  4. the security log (Security \u2192 Events<\/em>): search for OAI-SearchBot, Claude-SearchBot and PerplexityBot and see what happened to them.<\/li>\n<\/ol>\n
    If you want to let AI search crawlers in but keep training crawlers out, do it in robots.txt or with a precise WAF rule, not with a blanket \u201cblock all AI bots\u201d switch.<\/p>\n
    Can a firewall or WordPress security plugin stop AI crawlers?<\/h2>\n
    Yes. Hosting firewalls, ModSecurity rules and WordPress security plugins often limit bots that send many requests in a short time. What to look at:<\/p>\n