AI discovery · 7 min read
AI search crawler controls: access, training and measurement
Treat search discovery, model training and user-requested retrieval as separate provider-specific controls. Allowing a crawler does not guarantee a citation, visit or enquiry.
OpenAI uses different crawler roles
OpenAI documents OAI-SearchBot for search, GPTBot for training and ChatGPT-User for user-requested retrieval. The first two can be controlled independently through robots rules. User-initiated requests are a separate case; do not assume a robots block is an access-control system.
Use the provider’s current documentation when choosing controls. Authentication and server access rules protect private content. Public robots rules express crawler preferences; they do not turn confidential content into safe public content.
Choose the control for the actual question
| Question | Relevant control or evidence | What it does not prove |
|---|---|---|
| May OAI-SearchBot crawl this public URL? | Its applicable robots group plus live server access | That ChatGPT will cite it |
| May GPTBot crawl for training? | Its independently applicable robots group | That search visibility is disabled |
| May Google use a property in Search generative AI? | Search Console’s Search generative AI control | That every page will appear |
| Can a person or user-requested fetch access private content? | Authentication and server access controls | That a robots Disallow makes the content private |
Google’s Search AI control is distinct from crawling
Google’s documented Search generative AI control concerns inclusion in AI Overviews and AI Mode. It is a separate question from robots crawl access and from controls for other Google AI uses. Check the setting for the relevant property and its inheritance before making a site-wide decision.
Google’s current AI search guidance still centres on useful, accessible content and established Search requirements. A special AI file or keyword pattern is not a reliable shortcut to discovery. Do not expose private pages or loosen permissions in pursuit of visibility.
A fictional policy to inspect, not a universal recommendation
The following illustrative file permits a search crawler while declining a training crawler. An owner must choose their own policy and verify current provider documentation. Mapseen’s checker evaluates the pasted rules; it does not change your site’s robots.txt.
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /Keep four outcomes separate
Google documents a Search generative AI performance report with impressions for AI Overviews and AI Mode. If that report is absent, review availability and the property setting before concluding there were zero appearances. A public total-clicks comparison does not import that report.
Record appearances or citations, referral visits, enquiries and sales separately. A citation is not a visit, and a visit is not a sale. Referrer information can be incomplete, so label what was observed and avoid assigning every unexplained visit to an AI system.
Review without breaking public and private boundaries
- List the public content you actually want discovered.
- Check each provider’s current purpose and control documentation.
- Test the applicable robots rules and live delivery separately.
- Keep accounts, documents and recovery pages protected.
- Measure the available report signals and real on-site outcomes without promising rankings or citations.
Sources and editorial method
Written and maintained by Mapseen. Guidance is linked to primary documentation. Examples are fictional; they explain a calculation or decision rather than report customer outcomes. Provider behaviour and report availability can change.
- OpenAI: crawler roles and controls
- Google: AI search optimisation guidance
- Google: Search generative AI control
- Google: Search generative AI performance report
Found a mistake? Open an email draft to Mapseen with the guide URL and the detail to review.