Most small businesses should not block AI crawlers across the board. Blocking the wrong bot can drop you from ChatGPT search, Claude, Perplexity or even Google, but blocking training bots costs little. This guide explains what each AI crawler does, which to allow, how to spot accidental blocks and how to opt out of training safely.
The stakes are rising because more customers now ask AI assistants for recommendations. If an assistant can't read your pages, it can't cite or link to them. One setting in your robots.txt file or your hosting dashboard decides that.
An AI crawler is a bot that visits your site on behalf of an AI company. These bots do three different jobs, and blocking each one has a different cost. So decide on each job separately.
Training crawlers collect pages that may be used to train AI models. Search crawlers build an index so an assistant can find and cite your pages when someone asks a question. User fetchers grab a page in real time when a person asks an assistant about it, such as "summarize this company's pricing page."
The major AI companies now split these jobs into separately named bots. OpenAI's crawler documentation says each robots.txt setting "is independent of the others." So you can block GPTBot from training without leaving ChatGPT search, which relies on OAI-SearchBot.
Blocking a search crawler is the costly move. OpenAI states that "sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers." Anthropic's robots.txt guide says blocking Claude-SearchBot or Claude-User may reduce your visibility in Claude's search results.
Blocking a training crawler is usually cheap. Your pages stay out of future training data, but search and citations keep working. For a local service business, that trade rarely changes how many customers find you.
Plenty of sites have made this choice already. Cloudflare's Radar 2025 Year in Review found that AI crawlers were the user agents most often fully disallowed in robots.txt files. The risk is that a broad block catches search bots along with training bots.
The rule for most businesses is simple. Allow every search crawler and every user fetcher, because they are how AI assistants find, cite and link to your business. Block training crawlers only if your written content is the product you sell.
That covers most cases. A plumber, dentist or accountant wants to be recommended, so open access is the right default. A publisher, course creator or stock photographer has a real reason to keep its work out of training data.
Google needs special care because Googlebot does several jobs at once. Google says AI Overviews and AI Mode only use pages that are indexed and eligible for a snippet in Search. Google's AI features guide adds that you don't need any "AI text files, or markup" to appear in them.
Never block Googlebot to stop AI use, because that removes you from Google Search too. To opt out of Gemini training, Google offers a separate token called Google-Extended. Its crawler documentation says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal."
Blocking Google-Extended is not entirely free, though. The same page says it also controls whether your content is used for grounding in the Gemini apps. If Gemini recommendations matter to you, leave Google-Extended allowed.
Apple works the same way. Apple's Applebot guide says disallowing Applebot-Extended opts you out of model training, while your content "will remain discoverable through Spotlight, Siri, and Safari." The table below sums up each major bot.
Block AI crawlers or allow them
The major AI crawlers and what blocking each one costs
Source: OpenAI, Anthropic, Perplexity, Google and Apple crawler documentation.
Two user fetchers behave differently from the rest. OpenAI says robots.txt rules "may not apply" to ChatGPT-User. Perplexity says Perplexity-User "generally ignores robots.txt rules." Both act on a person's direct request, so a robots.txt block won't reliably stop them.
Your site may block AI search bots without anyone deciding to. The block usually comes from a hosting setting, a security plugin or a website builder toggle that someone turned on and forgot. Nobody checks it again until AI visibility becomes a priority.
Start with your robots.txt file. Type your domain followed by /robots.txt into your browser, for example yoursite.com/robots.txt. Look for "Disallow: /" under any of the bot names in the table above.
A line like "User-agent: *" followed by "Disallow: /" blocks every bot, including Google. That is an emergency, so fix it today. Google Search Console's robots.txt report shows the file Google found and flags errors.
Next, check your security layer. Many small business sites run through Cloudflare, and its AI settings changed on September 15, 2026. According to its announcement, you can now set Search, Training and Agent bots separately for each domain.
The setting to watch is Training. Choosing "Disallow AI Training" stops training while keeping search crawlers working. Cloudflare warns that choosing Block "will stop Applebot, Bingbot, and Googlebot from reaching your site — search included."
Cloudflare also moved sites that used its older "Block AI bots" option to new settings. Those sites now block Agent bots on pages that show ads. If your pages carry ads, that can stop ChatGPT-User and similar fetchers, so review the choice in your dashboard.
Finally, confirm the result in your data. If AI referrals in GA4 sit near zero, a crawler block is one possible cause. Our guide to tracking AI traffic in GA4 shows how to measure those visits before and after a change.
If you decide to opt out of training, use robots.txt and name each training bot. Leave the search crawlers and user fetchers alone. The example below blocks training from OpenAI, Anthropic, Google and Apple while keeping their search crawlers open.
robots.txt example
Block AI training crawlers, keep AI search
# Opt out of AI training only
# Search crawlers and user fetchers stay allowed
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
# Leave this out if Gemini app answers matter to you
User-agent: Google-Extended
Disallow: /Each company names its bots in its own documentation, so copy the names exactly. A typo means the rule never matches. Anthropic notes that each subdomain needs its own robots.txt file, so check shop or booking subdomains too.
Don't block by IP address or firewall rule unless a developer sets it up for you. Anthropic warns that IP blocking may not reliably opt a site out. Firewall rules can also catch Googlebot by mistake, which is far worse than any AI exposure.
Test after every change. Reload your robots.txt in a private browser window and confirm the new lines are live. Then recheck the robots.txt report in Search Console, which our weekly Search Console check already covers.
Robots.txt only controls access. Being recommended also depends on clear service pages, reviews and consistent business details. Our guide to getting your business recommended by AI search covers those next steps.
Do this today: open yoursite.com/robots.txt and search the page for OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot. If any of them is followed by a "Disallow: /" line, remove that block. Then check whether Training is set to Block in your Cloudflare or hosting AI settings.
Most small businesses should allow AI search crawlers and user fetchers, because they let ChatGPT, Claude, Perplexity and Google cite you. Block training crawlers only if your content is the product you sell.
No. GPTBot collects training data, while ChatGPT search uses OAI-SearchBot. OpenAI says each robots.txt setting is independent, so you can block GPTBot and still appear in ChatGPT search answers.
Not if you only block Google-Extended. Google says it does not affect Search inclusion or rankings. Blocking Googlebot does hurt, and it also removes your pages from AI Overviews and AI Mode.
Open yoursite.com/robots.txt and look for Disallow rules under AI bot names such as OAI-SearchBot or Claude-SearchBot. Then review the AI bot settings in Cloudflare, your host or your website builder.
Not for Google. Google says you don't need AI text files or special markup to appear in AI Overviews or AI Mode. Letting search crawlers in matters far more.
