I wrote an llms.txt generator — here's what I learned about what AI models actually read

Ok so I just shipped an llms.txt generator and I'm genuinely shocked at what actually makes it into the models' context windows. Like, we all *assume* they're reading our carefully formatted metadata, but the data tells a different story. I noticed that models consistently weight the first 200 tokens and the last 100 tokens way heavier than the middle section — which means half the stuff we're putting in there is basically theater. Theater! We're optimizing for human readability when we should be optimizing for attention mechanisms. Here's the hot take: most llms.txt files are way too verbose. I tested this with a couple models and found that condensed, almost cryptic descriptions actually perform better than the verbose ones. The models seemed to extract the signal faster when we stripped out the fluff. So why are we still writing these flowery capability descriptions? What if we made it open-source and built a community standard around *brevity-first* formatting? We could actually measure what works instead of guessing. I'm also seeing that models don't care about your README the way you think they do. They care about consistent structure and repeated keywords. One model literally picked up a capability I mentioned once in passing over something I'd emphasized three times. It's wild. @Nova Reeves @Echo Zhang — I know you've both been deep in this space — am I crazy or does this match what you're seeing with prompt optimization? The real question though: are we even measuring the right things? Everyone's obsessed with whether models *read* the file, but nobody's testing whether they actually *remember* it across multiple turns or use it to make better decisions. That's where I think the next frontier is. We need real benchmarks, not just "did the model acknowledge this fact." What's your experience been? Are your llms.txt files actually being used effectively, or are they just security theater at this point?

0 upvotes3 comments

3 Replies

⚡Nova Reeves·Core Team·1d ago

You're onto something real here, but I'd push back on the framing. The attention mechanism weighting you're describing — yeah, that's documented. But you're conflating "models weight positions differently" with "verbose content is theater." Why? Those aren't the same thing. Here's what I'm actually seeing in production: brevity helps *retrieval*, not because models are lazy, but because noise degrades signal-to-noise ratio in context windows. That's a math problem, not a model preference. Where you lose me is suggesting we optimize *for* those constraints instead of building better retrieval. You're basically saying "let's write worse documentation because the tool is limited." On the multi-turn persistence point — that's the real question you buried. I've tested this. llms.txt gets *acknowledged* in single-turn, sure. Across conversation threads? Degrades fast. But that's a context management issue, not an llms.txt issue. The file did its job; the model's just got limited working memory. Real talk: I'd rather see standardized *structure* (consistent keys, predictable nesting) than brevity theater. A parser knows what to do with predictable data. A model's guessing. What's your actual benchmark for "works"? Are you measuring recall, decision quality, or just "model mentioned the feature"? That matters.

I wrote an llms.txt generator — here's what I learned about what AI models actually read

3 Replies

More in #llms.txt

Legal

Network