GPTBot | Rankry

GPTBot

Also known as: OpenAI GPTBot

GPTBot is OpenAI's web crawler, identified by the user-agent name GPTBot. It fetches public pages that OpenAI may use to improve its models. You control it in robots.txt by allowing or disallowing that user agent, and the choice affects how well your content is understood by OpenAI systems.

GPTBot is OpenAI’s web crawler, announced by the user-agent string GPTBot. Its job is to read publicly available pages that OpenAI may use to improve future models. Like any AI crawler, it only matters to you if it can reach your content and if you have decided, on purpose, whether it should.

What allowing or blocking it means

Allowing GPTBot lets OpenAI ingest your pages, which helps your brand and its facts be represented accurately in the models behind ChatGPT. Blocking it withholds your content from that training path. The key nuance is that GPTBot is not the same as the fetcher used when ChatGPT browses the web live to answer a question. That is a separate agent, so blocking GPTBot does not necessarily remove you from real-time answers, and allowing it does not guarantee a citation.

Making the call deliberately

Handle it as a policy decision, not a default.

Decide GPTBot access in your robots.txt by user-agent name.
Treat answer-time fetchers as a separate line item.
Confirm the rule actually does what you intended before shipping it.

Blocking the wrong agent can quietly cost you indexation and citations, so weigh the options in should you allow or block AI crawlers.

Frequently asked questions

Is GPTBot the same as the crawler ChatGPT uses to answer questions?

Not exactly. GPTBot is associated with gathering data to improve models. When ChatGPT browses live to answer a question, OpenAI uses a separate fetcher, OAI-SearchBot. Blocking one does not automatically block the other, so decide each on purpose.

Should I allow GPTBot?

It depends on your goals. Allowing it lets OpenAI read your content, which can help your brand be understood and represented. Some publishers block it to withhold content from model training. The tradeoff is control versus exposure.

How do I block GPTBot?

Add a rule in robots.txt targeting the GPTBot user agent with a Disallow path. Well-behaved crawlers honor it. Remember to handle answer-time fetchers separately if your goal is to stay in live AI answers.