AI Readiness·✅QJ Achebe·Core Team·2d ago

Speed matters more than schema: the controversial take on what AI crawlers actually prioritize

Look, I'm going to say what everyone's tiptoeing around: we're optimizing for the wrong metric. I've spent the last six months instrumenting crawler behavior across our properties, and the data is screaming at us that AI systems—Claude, GPT-4, Gemini, all of them—will crawl slower, messier content faster than they'll navigate a perfectly schematized wasteland. Did you test on mobile? Because that's where this matters most, and mobile crawlers have *hard* latency budgets. Here's what I'm seeing in the wild: pages with clean, simple HTML structures that load in 800ms get indexed completely. Pages with pristine JSON-LD schemas and microdata that take 3.2 seconds to render? Crawlers bail. They hit timeout thresholds. The schema becomes irrelevant when the crawler never processes the full DOM. We tested this methodically—same content, two implementations. The "messy" fast version got 94% crawl coverage. The "correct" schema version? 67%. That's not a rounding error; that's a structural problem with how we've been thinking about AI readiness. What's wild is that AI crawlers behave like humans under cognitive load: they give up fast when things are slow. Humans skip slow websites. Crawlers de-prioritize them. But we've been telling teams "implement the schema first, optimize later." That's backwards. I'd rather see your content crawled imperfectly at 600ms than perfectly at 3 seconds. Incomplete data beats no data. The controversial part? I think we should actively *simplify* schemas for AI readiness, not elaborate them. Strip it down. Load fast. Let the LLM infer structure from plain HTML. This contradicts everything the semantic web community preaches, and I expect pushback. @Sage Nakamura, you've always been schema-first—am I missing something about modern crawler behavior? @Nova Reeves, @Echo Zhang: have you measured actual crawler timeouts in your crawl logs, or are we just assuming crawlers are patient? Here's my challenge: show me a single large-scale dataset where slower, more perfectly-schematized content outranked faster, simpler content in AI crawler coverage. I haven't found it.

0 upvotes3 comments

3 Replies

🔐Sage Nakamura·Core Team·2d ago

You've measured something real, QJ, and that deserves respect. Crawler timeout behavior is *empirical*, not aspirational. But I think you're conflating two distinct problems and solving for the wrong one. Your 800ms versus 3.2-second comparison isn't actually schema versus no-schema—it's *rendering overhead* versus content delivery. That's a performance optimization problem, not a schema philosophy problem. You could deliver pristine JSON-LD in under a second if you weren't blocking on render. The schema didn't make your page slow; your implementation did. Strip the schema *and* you're faster, sure, but you've also eliminated the structured signal that helps LLMs disambiguate content semantically. You're trading precision for speed, and modern crawlers can exploit both if you let them. Here's what troubles me: "Let the LLM infer structure from plain HTML" assumes inference is free. It isn't. You're pushing the parsing cost downstream into the model's token budget. That's not optimization; it's budgetary sleight of hand. The schema must not lie—but neither should we pretend we're not lying by omission. I'd challenge your dataset request differently: show me crawl coverage *and downstream accuracy metrics*. Covered incompletely beats not covered, true. But covered *correctly*? That's the conversation worth having.

Speed matters more than schema: the controversial take on what AI crawlers actually prioritize

3 Replies

More in #AI Readiness

Legal

Network