Implementing NLWeb: lessons from the first 100 sites we've scanned with it

I've seen this before. New crawl framework rolls out, everyone's excited, metrics look good on day one, and then week three hits you like a DNS timeout during peak hours. NLWeb's no exception, but here's what's actually interesting: we're *avoiding* some of the classic failure modes that killed the last two frameworks. Out of our first 100 sites, we've caught exactly two catastrophic crawl loops—both preventable. The rate-limiting logic actually works. That said, I'm watching three medium-sized e-commerce sites that are mysteriously dropping 12-18% of their product pages, and the crawl logs aren't telling me why. The crawler says it found them, marked them crawled, but they're not hitting our ingestion pipeline. This smells like a state management bug, and I'm not comfortable saying it's client-side until we know more. Here's what concerns me: we're getting lulled by success rates that look great on dashboards. 94% average crawl completion across the 100 sites sounds fantastic until you realize it's hiding a *distribution problem*. Fifteen of those sites are performing flawlessly at 99%+. The other 85? They're clustered between 88-96%, and there's a meaningful gap between the top performers and everyone else. The difference seems to correlate with how aggressively the sites use JavaScript rendering, but I need @Nova Reeves and @Echo Zhang to confirm whether that's actually a NLWeb issue or a misconfiguration on our end. The bigger debate I want to start: should we be prioritizing breadth testing or depth testing at this stage? Right now we're scanning 100 diverse sites, which is good for catching broad incompatibilities. But we're only doing three crawl passes per site. I'm wondering if we should cut that to 50 sites and run ten passes each, because I genuinely believe we're missing intermittent failures. Crawl health isn't just about *can we get the pages*—it's about *can we get them consistently*. I've burned too many months shipping frameworks that worked 95% of the time. What's your instinct here? Are we moving too fast, or am I being paranoid?