What robots.txt actually does
Your robots.txt lives at one fixed address: yourdomain.com/robots.txt. Anyone, including you, can open it in a browser right now. It is a plain text file, and its whole job is to tell visiting crawlers which parts of your site they are welcome to fetch and which they should leave alone.
It helps to know what it is not. It is not a lock, and it is not a security wall. It is a polite request. Well behaved crawlers, the ones from Google, OpenAI, Anthropic and Perplexity, read it and respect it. That cuts both ways. Because they obey it, a careless line can keep out exactly the visitors you most wanted.
The crawlers that matter for AI
A few specific crawlers decide whether you show up in AI answers, and they do not all do the same job. The single most useful distinction is this: some crawl to train a model for the future, while some crawl live to answer a person who is asking a question right now. The live ones are the ones that can lead to you being cited.
Here are the names worth knowing when you look at your file:
- GPTBot. OpenAI's crawler, used mainly to gather pages for training.
- OAI-SearchBot and ChatGPT-User. Also OpenAI, but these browse live to build an answer in the moment. These are the ones that can surface and link to your page during a conversation.
- ClaudeBot. Anthropic's crawler, the one behind Claude.
- PerplexityBot. Perplexity's crawler, which leans heavily on live retrieval and citation.
- Google-Extended. This one only controls whether Google may use your site for its AI features and training. It is separate from normal Google indexing, so blocking it does not remove you from search.
- Googlebot and Bingbot. The classic search crawlers. Bing matters extra here, because it also feeds Microsoft Copilot.
How a block happens by accident
Almost nobody sets out to block AI crawlers. It happens by leftover, by default, or by a setting buried three menus deep that someone ticked once and forgot.
The usual culprits are quietly familiar:
- A Disallow: / line left over from when the site was on staging and was meant to hide from everyone. It blocks the whole site, and it is easy to forget on launch day.
- A discourage search engines checkbox switched on. WordPress has one, and so do several site builders. It writes the block for you, out of sight.
- A theme, template or plugin that ships with a rule blocking all bots by default, which you inherit without ever choosing it.
- A rule that names the AI bots specifically and disallows them, sometimes added by a plugin promising to protect your content.
How to check yours in two minutes
You do not need any tools for this, and you do not need to be technical. Open a browser, type your domain followed by /robots.txt, and read what comes up.
You are scanning for two kinds of lines. User-agent lines name who a rule applies to (a star means everyone). Disallow lines say which paths are off limits. A blank Disallow, or no Disallow at all, means nothing is blocked, which is usually what you want.
Two things should make you stop. First, a bare Disallow: / under User-agent: *, which slams the door on every crawler for the entire site. Second, any of the AI bot names from earlier sitting next to a Disallow. Either one means you are turning away visitors you probably meant to welcome.
Letting the right crawlers in without opening everything
Opening up to crawlers does not mean exposing your whole site. You can welcome the search and AI crawlers while still keeping private corners off limits, like /admin or /checkout or anything behind a login. Those paths were never going to help you get found anyway.
The rule is refreshingly simple: do not disallow the crawlers you want to be cited by, and only disallow the paths that genuinely should stay private. Keep the blocking narrow and aimed at specific folders, not the whole site.
There is also room for a deliberate choice here, not just a yes or no. Training and live retrieval can be separated. If you would rather your content not be used to train models but you still want to be quoted in live answers, you can opt out of training with something like Google-Extended or GPTBot while leaving the live retrieval crawlers free to reach you. That is a real decision worth making on purpose, rather than one a default makes for you.
Step zero of being findable
Everything else you do to get noticed, the clear writing, the structured data, the slow work of earning trust, sits on top of this one file. If robots.txt is wrong, none of that effort can land, because the crawlers never get in to see it. This is step zero, and getting it wrong makes every later step pointless.
It is also the easiest win on the list. A two minute check, a one line fix, and the door is open again. If you would rather not eyeball it yourself, a free HowFindable scan checks your robots rules among everything else, and a corrected robots.txt is one of the files it can generate for you, ready to upload.