Bot Access & Crawlability
Whether AI crawlers like ChatGPT, Perplexity, and Claude can actually reach your content.
What We Test
AgentReady reads your robots.txt file and checks whether major AI crawlers are allowed or blocked. We specifically look for Googlebot, ChatGPT-User (OpenAI's crawler), PerplexityBot, and ClaudeBot (Anthropic's crawler). We also verify that you have an XML sitemap and that it contains valid URLs for AI crawlers to discover.
Why It Matters
AI engines cannot cite content they cannot access. Many sites inadvertently block AI crawlers through overly broad robots.txt rules, or they simply never updated their configuration when AI crawlers emerged. If ChatGPT-User is blocked in your robots.txt, your content will never appear in ChatGPT responses — no matter how relevant it is. This is the simplest and most common reason sites are invisible to AI.
How Scoring Works
Bot Access accounts for 15% of your overall score. You earn +20 points for allowing Googlebot, +20 for ChatGPT-User, +20 for PerplexityBot, and +20 for ClaudeBot. Having an XML sitemap adds +10, and having a valid sitemap (with proper <urlset> or <sitemapindex> tags) adds another +10.
How to Improve
Check your robots.txt file (yourdomain.com/robots.txt) and make sure you are not blocking AI crawlers. If you have a blanket "Disallow" rule for unknown bots, add explicit "Allow" rules for ChatGPT-User, PerplexityBot, and ClaudeBot. Also add a Sitemap directive pointing to your XML sitemap. If you do not have a sitemap, generate one — most CMS platforms have plugins or built-in tools for this.