Robots.txt has quietly become one of the most consequential files on your site for AI search visibility. It's the same simple, decades-old protocol that's always governed search-engine crawling, but a growing list of AI-specific user agents now needs the same deliberate attention Googlebot has always gotten — and a surprising number of sites get it wrong by accident.
This guide covers how to write robots.txt for ai bots correctly: which crawlers to consider, the syntax that actually works, and the decisions worth making deliberately rather than leaving to a default template.
This is one of the highest-leverage, lowest-effort fixes in AI search visibility — a misconfigured file can silently exclude a site from citation eligibility entirely, and fixing it usually takes minutes once you know what to look for.
What Does Robots.txt Do for AI Bots?
Robots.txt is a plain-text file at your site's root (yoursite.com/robots.txt) that tells well-behaved crawlers which parts of your site they may or may not access, using a simple syntax of User-agent and Disallow/Allow directives. AI crawlers — GPTBot, ClaudeBot, PerplexityBot, and others — generally respect this same protocol, which means the file that's always controlled search-engine access now also controls whether your content is even eligible to be read by the systems behind ChatGPT, Claude, and Perplexity.
It's important to be clear about what it isn't: robots.txt is a voluntary, publicly readable request, not a security mechanism. Compliant crawlers respect it; it doesn't prevent anyone from accessing a public page directly. Anyone, including a competitor, can also read your robots.txt directly, so treat it as a public statement of policy rather than a private configuration file.
Why Robots.txt Rules for AI Bots Matter
It's a binary gate before anything else matters
No amount of content quality or structured data compensates for a crawler that's disallowed outright — this is the first thing to get right, not an afterthought.
Default configurations often get this wrong
Many CMS platforms, page builders, and security plugins ship with default rules that predate the AI-crawler landscape, sometimes blocking bots the site owner never consciously decided to exclude.
It lets you separate different AI use cases
You can allow live retrieval crawlers while blocking training-focused ones, or vice versa — robots.txt gives you that granularity if you use it deliberately.
Mistakes here are easy to make and easy to miss
A single misplaced directive can block a bot you meant to allow, and unlike a broken page, a robots.txt error rarely produces a visible symptom you'd notice without checking directly.
How to Write Robots.txt Rules for AI Bots
Step 1: Audit your current file for existing AI-bot rules
Open your live robots.txt and search for user agents like GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and Google-Extended. Many sites are surprised to find rules they never intentionally added, often inserted by a plugin or platform default.
Step 2: Decide your policy per crawler, deliberately
Rather than a single blanket allow-or-block decision, consider each crawler against your actual goals — training exposure versus live citation eligibility are genuinely different trade-offs.
Allowing a crawler explicitly. A block naming the user agent with an Allow: / directive is the clearest way to confirm access, rather than relying on a wildcard rule you haven't checked recently.
Blocking a crawler explicitly. Use a named Disallow: / block for any crawler you deliberately want to exclude — being explicit here is better than assuming an unmentioned crawler defaults to your preference.
Example: A minimal, explicit setup: User-agent: OAI-SearchBot then Allow: /, followed by User-agent: GPTBot then Disallow: / — permitting live ChatGPT citation while opting out of that specific training crawler.
Step 3: Test your syntax carefully
Robots.txt syntax is unforgiving of small mistakes — a misplaced slash or an incorrectly scoped Disallow: / at the wrong point in the file can block far more than intended. Use a robots.txt testing tool before relying on a change in production.
Step 4: Re-check after any platform or plugin update
CMS updates, new security plugins, and CDN changes can silently rewrite or override robots.txt rules — treat this as something to re-verify periodically, not a one-time setup.
Step 5: Coordinate robots.txt with your CDN or WAF settings
If you run a CDN or web application firewall in front of your site, check its own bot-management configuration alongside robots.txt. It's common for one to be permissive while the other silently blocks the same crawler, and only checking both layers together gives you an accurate picture of what a given bot can actually reach.
Robots.txt Policies Compared
Fully open to all AI crawlers
Best For: Sites prioritizing maximum AI search and citation visibility above other considerations.
Watch Out For: This also means allowing training-data collection by any crawler you don't explicitly exclude — make sure that's a deliberate choice.
Allow retrieval, block training bots
Best For: Sites wanting citation eligibility in ChatGPT and Perplexity while declining to have content used for model training.
Watch Out For: Requires knowing which specific user agents correspond to which purpose — this list changes as crawlers evolve.
Fully closed to AI crawlers
Best For: Sites with strong licensing or competitive concerns that outweigh AI-search visibility.
Watch Out For: This forecloses citation-based visibility entirely — a real trade-off, not a neutral default.
Selective by directory or content type
Best For: Sites wanting to expose public marketing and guide content while restricting account areas, internal tools, or paywalled sections.
Watch Out For: More granular rules increase the chance of a syntax mistake — test thoroughly after any change.
Best Practices for AI Bot Access Control
Be explicit rather than relying on wildcards alone
Named rules for specific AI user agents remove ambiguity that a generic wildcard rule can leave open to interpretation.
Document your reasoning internally
A short internal note on why each crawler is allowed or blocked keeps the policy maintainable as your team changes.
Check CDN and firewall layers separately
Robots.txt is necessary but not sufficient — bot-management tools operating at a different layer can override it.
Revisit the policy on a recurring schedule
New crawlers appear; put a periodic review on your calendar rather than trusting a setup from a year ago.
Validate syntax with a real testing tool before deploying
This catches the kind of small mistakes that are easy to make and hard to notice without dedicated testing.
Common Mistakes to Avoid
Accidentally blocking bots via an inherited default
Templates, plugins, and platform defaults sometimes ship with AI-bot restrictions the site owner never consciously chose.
Confusing training-bot rules with retrieval-bot rules
Treating every AI crawler identically misses the ability to make a more precise, useful decision that matches your actual goals.
Broken syntax that blocks more than intended
A misplaced or overly broad Disallow directive can silently exclude far more of your site than planned — always test.
Never revisiting the file after initial setup
The AI crawler landscape changes; a robots.txt written a year ago may be missing newer, relevant user agents entirely.
Frequently Asked Questions
Where does robots.txt need to live?
At the root of your domain — yoursite.com/robots.txt — this is a fixed convention that crawlers expect and won't look elsewhere for.
Do I need separate rules for every individual AI crawler?
Not necessarily — a wildcard rule covers unnamed bots by default, but explicit rules for the specific crawlers you care most about (GPTBot, PerplexityBot, ClaudeBot) give you clearer, more intentional control.
Can I block AI training but still show up in ChatGPT search results?
Often yes, since training-focused and retrieval-focused crawlers are frequently distinct user agents from the same company — verify the specific split for each crawler you're targeting.
Will AI crawlers ignore my robots.txt if they want to?
Reputable AI companies' documented crawlers are built to respect it, the same voluntary protocol search engines have honored for decades — though it remains fundamentally a request, not an enforced restriction.
How do I verify my robots.txt changes actually took effect?
Re-fetch the live file directly after deploying, run it through a testing tool, and where possible check server logs for the crawler's subsequent requests.
Should I put my AI-bot rules in a separate file from my search-engine rules?
No — robots.txt only works as a single file at your site root. All user-agent blocks, whether aimed at search engines or AI crawlers, belong in that same file, organized clearly with one User-agent block per bot.
Key Takeaways
- Robots.txt is the first, binary gate controlling whether AI crawlers can access your site at all.
- Audit your current file for existing AI-bot rules — many exist unintentionally, inherited from platform defaults.
- Training-focused and retrieval-focused crawlers can often be controlled separately — decide deliberately.
- Test syntax carefully; small mistakes can block far more than intended.
ZeroSEO's Agent Readiness Score checks your robots.txt against the current list of major AI crawlers as part of its site audit, flagging accidental blocks before they cost you visibility. Sign up to run the check, or see how it works for more detail.
For the underlying protocol and crawler documentation, see Google Search Central and OpenAI's developer documentation.