What is robots.txt?
robots.txt is a plain-text file at the root of your domain (https://yoursite.com/robots.txt). It tells crawlers which parts of your site they may request. Every major search engine checks it before crawling, and so do most well-behaved AI and SEO bots.
robots.txt controls crawling, not indexing. A blocked page can still appear in Google (without a description) if other sites link to it. To keep a page out of search results, let Google crawl it and add <meta name="robots" content="noindex">. Never use robots.txt to hide private data. The file is public, and anyone can read it.
How the rules are matched
Our tester follows the official standard (RFC 9309) and Google’s documented behaviour exactly:
- One group per crawler: a crawler uses the group with the most specific matching
User-agentand ignores all others. If Googlebot has its own group, the rules underUser-agent: *don’t apply to it. - Longest rule wins:
Allow: /admin/public/beatsDisallow: /admin/because it is more specific. On an exact tie, Allow wins. - Wildcards:
*matches any characters.$anchors the end of the URL, soDisallow: /*.pdf$blocks PDFs. - Case-sensitive paths:
/Adminand/adminare different. - An empty
Disallow:means “nothing is blocked”.
Should you block AI crawlers?
AI companies crawl the web to train large language models and to power AI answers. Blocking them is a business decision:
| Crawler | Company | Purpose |
|---|---|---|
| GPTBot | OpenAI | Model training |
| OAI-SearchBot / ChatGPT-User | OpenAI | ChatGPT search results and user-requested browsing |
| ClaudeBot | Anthropic | Model training |
| Google-Extended | Gemini training (doesn’t affect Google Search) | |
| CCBot | Common Crawl | Open dataset used by many AI models |
| PerplexityBot | Perplexity | AI search answers |
| Applebot-Extended | Apple | Apple Intelligence training |
| Bytespider | ByteDance | Model training |
| Meta-ExternalAgent | Meta | Model training |
Block training bots if you don’t want your content used to train models. Keep search/answer bots (like OAI-SearchBot or PerplexityBot) if you want to be cited, and get traffic, from AI search tools. Note that robots.txt is voluntary: reputable companies follow it, but it isn’t a technical barrier.
Common robots.txt mistakes
- Leaving
Disallow: /from a staging site on the live site, which removes you from Google over the following weeks. - Blocking CSS and JavaScript. Google needs them to render and understand your pages.
- Putting the file in a subfolder. It must be at the root:
/robots.txt. - Using robots.txt to “noindex”. Google stopped supporting
Noindex:in robots.txt in 2019. - Forgetting the sitemap line, which is an easy way to help every search engine find your pages.