Implementing NLWeb: lessons from the first 100 sites we've scanned with it

I've seen this before. And I mean that literally — we're repeating patterns from the Crawler v3 rollout in 2019, except this time we're moving faster and pretending we learned something. Don't get me wrong, NLWeb is solid infrastructure. But I need to call out what the first 100 sites are actually telling us, because the numbers don't match the narrative we're selling. Here's what's keeping me up at night: our redirect handling is clean, yeah, but we're getting false negatives on 8-12% of sites with mixed HTTP/HTTPS configurations. That's not a rounding error. I watched the same issue tank crawler reliability for six months last cycle because nobody wanted to admit the problem existed until it affected enterprise clients. The sites we've scanned so far skew heavily toward modern stacks. Once we hit the long tail — legacy banking sites, government portals, anything built before 2015 — I guarantee this number climbs. @Echo Zhang, your team's error logs show this pattern too, right? I'm not trying to be a pessimist, but we need to surface this before we greenlight the full rollout. Second observation: our resource allocation is too aggressive for sites in certain geographic regions. I'm seeing timeouts spike noticeably on sites served from Eastern Europe and Southeast Asia, even when the sites themselves are snappy. This might be a networking assumption baked into how we're distributing our crawl agents. We optimized for US/Western EU performance, which is exactly the kind of blind spot that creates support nightmares at scale. Here's what I want to know: are we confident enough in our current error recovery logic to handle the inevitable failures at 10,000 sites? Because I've been through three "production-ready" deployments that weren't, and the cost of rolling back is exponential. Before we expand the testing cohort, I want to see our team sit with the actual failure modes we're generating — not just the aggregate success metrics. What specific failure patterns is everyone else seeing in your corners of this scan data?