Can I block only the training crawlers while staying visible in AI search?

Yes. Disallow GPTBot while leaving OAI-SearchBot untouched, or Disallow ClaudeBot while leaving Claude-SearchBot untouched — each company publishes separate User-agent tokens for training versus search, so you can block one without the other.

What happens if I block every AI crawler?

Your content stops showing up in AI search answers from ChatGPT, Claude, and similar tools. Regular Google and Bing search indexing is unaffected as long as you target AI-specific tokens like Google-Extended — note that Bing has no separate AI-only bot, so Bingbot itself can't be blocked without also losing regular Bing search.

How long does it take for a robots.txt change to take effect?

OpenAI's official documentation notes it can take roughly 24 hours for a robots.txt change to affect search results (OAI-SearchBot). Other crawlers don't publish an exact figure, so it's safest to assume similar delays rather than expect instant results.

How do I set up robots.txt for AI crawlers?

OpenAI's GPTBot, Anthropic's ClaudeBot, and PerplexityBot each identify themselves with a distinct User-agent, so robots.txt lets you allow or block them one by one. Keeping training crawlers out while leaving search-facing bots open is a common setup for staying visible in AI search without feeding your content into model training.

Updated 2026-07-07

Frequently asked questions

How do I set up robots.txt for AI crawlers?: OpenAI's GPTBot, Anthropic's ClaudeBot, and PerplexityBot each identify themselves with a distinct User-agent, so robots.txt lets you allow or block them one by one. Keeping training crawlers out while leaving search-facing bots open is a common setup for staying visible in AI search without feeding your content into model training.
Can I block only the training crawlers while staying visible in AI search?: Yes. Disallow GPTBot while leaving OAI-SearchBot untouched, or Disallow ClaudeBot while leaving Claude-SearchBot untouched — each company publishes separate User-agent tokens for training versus search, so you can block one without the other.
What happens if I block every AI crawler?: Your content stops showing up in AI search answers from ChatGPT, Claude, and similar tools. Regular Google and Bing search indexing is unaffected as long as you target AI-specific tokens like Google-Extended — note that Bing has no separate AI-only bot, so Bingbot itself can't be blocked without also losing regular Bing search.
How long does it take for a robots.txt change to take effect?: OpenAI's official documentation notes it can take roughly 24 hours for a robots.txt change to affect search results (OAI-SearchBot). Other crawlers don't publish an exact figure, so it's safest to assume similar delays rather than expect instant results.