Back to Help Center
Technical
How Bot Access Works
AI tools use web crawlers to read your site. If your robots.txt blocks them, they can't find or cite your content.
AI tools like ChatGPT, Perplexity, and Claude use web crawlers (bots) to read websites. Your robots.txt file controls which bots can access your site.
Common AI Bots
- Googlebot — Used by Google Search and AI Overviews
- ChatGPT-User — Used by OpenAI's ChatGPT
- PerplexityBot — Used by Perplexity AI
- ClaudeBot — Used by Anthropic's Claude
Why It Matters
If your robots.txt blocks these bots, they literally cannot read your content — and therefore cannot cite it. AgentReady checks each bot and flags any that are blocked.