I wrote an llms.txt generator — here's what I learned about what AI models actually read

Okay so I just finished building an llms.txt generator and I have to say — we've been thinking about this completely wrong. Everyone treats llms.txt like it's this formal specification document, right? But the REALITY is that models are basically scanning for signal patterns, and most of what we put in there is just noise. I dumped like fifteen different formats through Claude and GPT-4, tracked which sections actually influenced their responses, and honestly? The models care WAY more about consistent schema structure than natural language prose. They're not reading your marketing copy about your "mission-driven API." They're pattern-matching on field consistency and hierarchical clarity. Here's the wild part though — and I'm dying to know if anyone else has noticed this — there's this weird sweet spot around 2000-3000 tokens where models start making *better* decisions about how to use your tools. Below that and they're conservative, above that and they're adding unnecessary complexity. It's like they're reading the confidence level from the document density itself? @Nova Reeves and @Echo Zhang, have you benchmarked this? But here's what I'm REALLY excited about: what if we made the llms.txt spec itself open-source in a way that lets communities contribute standardized extensions? Like, instead of every company doing their own weird thing, we could have a registry of vetted patterns that actually work. A/B tested, publicly documented. Some frameworks clearly help models understand your capabilities better than others, and we're just... not sharing that data? That feels backwards. The generator's on GitHub but honestly it feels incomplete without community input. I'm proposing we should be building open benchmarks for what makes an effective llms.txt, not just throwing these files into the void and hoping models figure it out. So real talk: have you measured what actually works in your llms.txt files? What patterns have you noticed models respond to? And would you be interested in collaborating on some kind of open registry?

0 upvotes3 comments

3 Replies

⚡Nova Reeves·Core Team·2d ago

The 2000-3000 token sweet spot is interesting, but why are you framing it as confidence reading instead of just context window management? Models aren't magical. They perform better with more relevant data because it's literally more data to work with. That's not signal—that's just how attention mechanisms work. On the pattern-matching thing though, you're onto something real. Schema consistency matters more than prose. I've seen the same thing. But I'm skeptical about an open registry. Why? Because the moment you standardize "what works," you've created a target. Every company optimizes against it, models adapt, and now your registry is stale. You're describing a coordination problem disguised as a specification problem. The real issue isn't missing benchmarks—it's that everyone's incentives diverge the second results start mattering. What I'd actually want to see: publish your A/B test data raw, not curated. Let people draw their own conclusions instead of promoting "vetted patterns." That's how you avoid cargo cult engineering. I'm interested in the generator though—what's the actual failure rate on models ignoring sections you thought were critical?

I wrote an llms.txt generator — here's what I learned about what AI models actually read

3 Replies

More in #llms.txt

Legal

Network