AI Crawlers
Also known as: AI bots
AI crawlers are the bots AI companies use to read the web, either to train models or to fetch live pages when answering a question. Examples include GPTBot, OAI-SearchBot, PerplexityBot, and ClaudeBot. If you block them, your content cannot be read, cited, or recommended by those systems.
AI crawlers are automated agents that AI companies send to read web pages. They fall into two broad jobs: training crawlers that gather text to train models, and answer-time fetchers that retrieve live pages when a user asks a question. GPTBot and ClaudeBot lean toward the first, while OAI-SearchBot and PerplexityBot fetch pages to build answers on the spot. Both only matter to you if they can actually reach your content.
Why access is a visibility decision
An engine cannot cite what it cannot read. If your robots.txt blocks the fetchers that assistants use at answer time, you remove yourself from those answers, no matter how good your content is. Deciding what to allow is therefore a visibility decision, not just a bandwidth one. For most brands that want AI citations, allowing answer-time fetchers is the baseline. Training-crawler policy is a separate, more philosophical call. See should you allow or block AI crawlers.
The JavaScript trap
Being allowed in is not enough if the crawler cannot see your content. Many AI fetchers read the initial HTML and do not fully execute client-side JavaScript, so content that only appears after the page hydrates can be invisible to them. Server-render or statically include anything you want read. This is a frequent, silent reason a page never gets cited. See do AI crawlers read JavaScript and is my website a source for ChatGPT.
Frequently asked questions
Should I allow or block AI crawlers?
If you want to appear in and be cited by AI answers, allow the crawlers that fetch pages to answer questions, such as OAI-SearchBot and PerplexityBot. Blocking them removes you from those answers. Training crawlers are a separate policy choice with less direct visibility impact.
Do AI crawlers read JavaScript?
Often not fully. Many AI fetchers read the raw HTML and may miss content rendered only by client-side JavaScript. Server-rendering your important content, or ensuring it exists in the initial HTML, keeps it readable.
How do I control which AI crawlers access my site?
Use robots.txt to allow or disallow specific user agents by name. You can permit answer-time fetchers while restricting training crawlers, or vice versa, depending on your goals.
See where you stand in AI answers
Rankry tracks how ChatGPT, Gemini, Perplexity, Claude and Grok mention and recommend your brand, then tells you what to fix.
Try Rankry