robots.txt

Also known as: robots exclusion protocol, robots file

robots.txt is a plain text file at the root of a domain that tells crawlers, including AI crawlers, which paths they may or may not fetch. It works by user-agent name and path rules. A single wrong Disallow line can silently keep your pages out of the index an answer engine draws from.

robots.txt is a plain text file at the root of your domain that tells automated crawlers what they are allowed to fetch. It is the first thing most bots request, and it works by matching a user-agent name to a set of Allow and Disallow path rules. Because well-behaved crawlers honor it, robots.txt is a quiet but powerful gate on whether your content ever reaches an answer engine.

Why one line can make you invisible

The failure mode is subtle. A Disallow: / left over from a staging site, or a rule that blocks a whole section, will keep compliant AI crawlers out with no error message and no warning in your analytics. The engine simply never sees the page, so it cannot be part of the index it retrieves from, and it cannot be cited. This is one of the most common silent causes of missing AI visibility.

How to use it deliberately

Treat access as a strategy, not an afterthought.

Name the crawlers you care about, such as GPTBot and answer-time fetchers, and decide each explicitly.
Keep the file minimal and test it before shipping.
Pair it with an llms.txt file to guide models toward your best content.

Before you tighten any rule, read should you allow or block AI crawlers so a cleanup does not quietly cost you answers.

Frequently asked questions

Does robots.txt affect AI visibility?

Yes. If your robots.txt blocks the user agents that AI assistants use to fetch pages, those systems cannot read, index, or cite your content. The rules are honored by well-behaved bots, so a mistaken block removes you from AI answers even if your pages are excellent.

Where does robots.txt live?

It must sit at the root of the domain, for example example.com/robots.txt. A file placed in a subfolder is ignored. Each subdomain needs its own file.

Can robots.txt block AI crawlers but allow Google?

Yes. You target specific user agents by name, so you can allow Googlebot while disallowing GPTBot, or the reverse. Order and specificity of the rules matter, so test changes carefully.