Does blocking GPTBot alone block every OpenAI-related crawler?

No. OpenAI runs GPTBot for training-data collection, OAI-SearchBot for ChatGPT search exposure, and OAI-AdsBot for ad landing-page verification, each under its own user-agent, and robots.txt lets you set these tokens independently. You can disallow GPTBot while allowing OAI-SearchBot, keeping your content out of training data while staying eligible for citation in ChatGPT search results.

If I block everything in robots.txt, is my site completely invisible to ChatGPT?

Not necessarily. OpenAI's documentation notes that ChatGPT-User — the agent that fetches a page in real time when a user asks about a specific URL — may not honor robots.txt rules. robots.txt stops bulk crawling and indexing, but it doesn't guarantee it blocks every real-time fetch a user request triggers.

Where is robots.txt located, and can anyone view it?

It sits at the root of every website and is public — anyone can open it by adding /robots.txt to the domain, like example.com/robots.txt. No login or authentication is required.

How can I tell if my site is blocking AI crawlers?

Check your site's root robots.txt file to see whether crawlers like GPTBot and ClaudeBot are blocked with a Disallow rule. If they're blocked, that AI can't read your pages and drops you from its citation candidates entirely. A diagnostic tool can help you check this automatically.

Updated 2026-07-11

Frequently asked questions

How can I tell if my site is blocking AI crawlers?: Open the robots.txt file at your site's root (e.g., example.com/robots.txt) and check whether GPTBot, ClaudeBot, PerplexityBot, and Google-Extended are listed under Disallow. If they are, those AI systems can't crawl your content and you're excluded from citation candidates. A diagnostic tool can check this for you automatically.
Does blocking GPTBot alone block every OpenAI-related crawler?: No. OpenAI runs GPTBot for training-data collection, OAI-SearchBot for ChatGPT search exposure, and OAI-AdsBot for ad landing-page verification, each under its own user-agent, and robots.txt lets you set these tokens independently. You can disallow GPTBot while allowing OAI-SearchBot, keeping your content out of training data while staying eligible for citation in ChatGPT search results.
If I block everything in robots.txt, is my site completely invisible to ChatGPT?: Not necessarily. OpenAI's documentation notes that ChatGPT-User — the agent that fetches a page in real time when a user asks about a specific URL — may not honor robots.txt rules. robots.txt stops bulk crawling and indexing, but it doesn't guarantee it blocks every real-time fetch a user request triggers.
Where is robots.txt located, and can anyone view it?: It sits at the root of every website and is public — anyone can open it by adding /robots.txt to the domain, like example.com/robots.txt. No login or authentication is required.