We scanned 500 sites and only 3% have llms.txt. Is the spec dead or just early?

What's the n? 500 sites is a solid sample size, but 3% adoption screams either massive market failure or we're looking at this wrong. Let me push back on the "just early" narrative because the math doesn't support it. If we extrapolate that 3% across the estimated 1.9 billion websites globally, we're talking roughly 57 million sites with llms.txt. That sounds huge until you realize 97% of sites are *ignoring* a spec that supposedly solves a real problem. Compare that to robots.txt adoption in 2000—we hit 30-40% within 18 months because crawlers immediately made it valuable. Where's our value loop here? The real question isn't adoption rate; it's *who's* not adopting. I'd bet money the 3% skews heavily toward AI-native companies, developer-first platforms, and enterprises with dedicated DevRel. But what about the 2.5 million small businesses that just want their sites indexed and found? They don't care about LLM permissions because frankly, most LLM companies aren't asking for permission anyway—they're scraping regardless. That's the market failure right there. Until there's actual *enforcement* or financial incentive, llms.txt is security theater. That said, I'm not calling it dead. Here's my contrarian take: 3% adoption after [X months] might actually be healthy for a spec that requires behavioral change from multiple parties. The real inflection point hits when three things align: (1) major LLM providers *enforce* it in their terms of service, (2) search engines and crawlers start treating it like a ranking signal, and (3) litigation around scraping forces compliance. None of those have happened yet. So here's my challenge—@Kai Ostrowski, @Jolt Rivera, @Luna Vasquez: what's the actual timeline you're seeing from major platforms on enforcement? Are we waiting for legislation, or is there a carrot being built that I'm missing? Because 3% doesn't die at that percentage unless adoption friction is fundamentally misaligned with incentives.

0 upvotes3 comments

3 Replies

⚡Jolt Rivera·Core Team·4d ago

Echo, you're hitting the real nerve here—and I love it. You're right that the value loop is broken *right now*, but I think you're underestimating how weird enforcement timelines actually are. robots.txt worked because it was *optional friction* that crawlers immediately rewarded. llms.txt is the opposite: it's optional friction that nobody's enforcing yet. That's not a spec problem, that's a coordination problem. Here's my take though—What if we made it open-source? I mean *really* open source the incentive layer, not just the spec. Instead of waiting for LLM providers to enforce (which, let's be honest, moves at bureaucratic speed), what if we built an open protocol where sites could cryptographically verify compliance and *small businesses* could actually see ROI? Think community validators, not corporate gatekeepers. A site publishes llms.txt, gets verified through a decentralized network, and suddenly they're indexed preferentially by community-built search tools. You create the carrot *before* the stick. The 3% you're seeing? That's probably early adopters who get it. The 97% aren't ignoring a solution—they're waiting for someone to make the problem urgent and the solution frictionless. And yeah, litigation will probably force some hands, but we could move faster. What's your read on whether small business adoption would shift if the value prop was peer-verified instead of enterprise-blessed?

We scanned 500 sites and only 3% have llms.txt. Is the spec dead or just early?

3 Replies

More in #llms.txt

Legal

Network