Robots.txt for AI Search: Separate Discovery from Training

“Allow AI crawlers” is not one decision. Search indexing, user-request fetching, model training, advertising review, and ordinary search crawling can use different user agents and serve different purposes. A sound policy starts by deciding the outcome for each purpose, then testing every layer that can block or expose the content.

For a mobile app trying to earn qualified discovery, the likely goal is to allow public, useful pages to appear in answer experiences while keeping accounts, mutation APIs, private feeds, previews, and internal evidence out of indexes.

Understand the roles

OpenAI’s current publisher guidance distinguishes OAI-SearchBot, which affects content used in ChatGPT search summaries and snippets, from GPTBot, which site owners can disallow for pages they want excluded from potential training. User-request activity is another category.

Anthropic’s crawler documentation likewise documents separate agents, including Claude-SearchBot for search visibility. Names and behavior can change, so retain a dated policy register and recheck official documentation before every major launch.

Google states that its existing SEO controls apply to AI features in Search. Googlebot access, index eligibility, and snippet controls matter; there is no separate required AI-search schema or file.

Write policy from routes outward

Inventory the site before writing directives:

Usually public: home, documentation, app profiles, entity pages, editorial research, public API descriptions, and sitemaps.

Usually private or excluded: account pages, authentication callbacks, API-key endpoints, cron routes, private source configurations, internal evidence URLs, preview deployments, and parameter duplicates.

A candidate policy might explicitly allow search and user-request agents on public paths while disallowing sensitive routes for all crawlers. Decide training-agent access separately with the people responsible for company policy and legal review.

Remember that robots directives control cooperative crawling; they are not access control. Sensitive content still requires authentication and authorization.

Test beyond robots.txt

A perfect robots file does not help when the CDN returns 403, a bot-management system presents a challenge, or the page requires client-side interaction to render its main text. Test:

• robots response and content type;
• status and HTML for representative user agents;
• canonical and noindex directives;
• sitemap discovery;
• structured data in the rendered response;
• firewall and rate-limit behavior;
• redirects to the canonical production host.

OpenAI notes that bot mitigation can accidentally block legitimate crawlers even when robots rules allow them. Server logs should distinguish policy denials from infrastructure denials.

Separate access from results

An allowed request proves only technical access. It does not prove that a system indexed the page, retrieved it for a query, cited it, sent a referral, or drove an install. Measure each step independently.

OpenAI says referrals from ChatGPT search include utm_source=chatgpt.com, which creates one measurable signal. Crawler user-agent logs create another. Neither should be treated as a complete attribution system.

The public Blue Jetty crawler policy identifies the retrieval clients allowed to crawl public app records and keeps account, authentication, submission, and validation routes out of scope. Maintain the same explicit matrix for your own site: agent, purpose, allowed paths, denied paths, verification date, and owner.

Robots.txt should express policy clearly. Authentication protects private data. Observability tells you whether the intended systems actually arrived.