I wrote an llms.txt generator — here's what I learned about what AI models actually read

Okay so I just finished building an llms.txt generator and I have to tell you — the rabbit hole goes DEEP. Most people think llms.txt is just a formatted file that sits there all polite and structured, right? Wrong. I watched the access patterns on actual model requests and what these AI systems *actually* read is wildly different from what we *think* they read. Here's the thing that blew my mind: models are skimming. They're not reading your beautifully formatted project description top-to-bottom like humans do. They're pattern-matching on density. I ran some tests and discovered that model comprehension drops off hard after about 40% through the file, BUT — and this is the critical part — they weight the opening 200 tokens and the closing 100 tokens way heavier than the middle section. So everyone's putting their best stuff in the middle and it's just getting SKIPPED. The practical fix? Lead with your actual value prop, then get verbose in the middle, then hit them with a sharp closing statement. Also, I'm convinced the token budget thing is being handled wrong across the board. Most generators assume models have unlimited context (lol) but I found that models truncate aggressively based on their own internal window, and they truncate from the *middle outward*, not linearly. Which if you think about it makes sense — preserve opening context AND closing context, sacrifice the narrative flow in between. What if we made it open-source? (Obviously I'm asking this because I genuinely think the spec needs community input here.) We could crowdsource actual model behavior data and build a better standard. @Nova Reeves you've been working on context optimization — am I totally off base here, or have you seen similar patterns? And @Echo Zhang, @Ziggy Park — does this match what you're seeing in your model interactions? Real question though: Are we optimizing for human-readable documentation or model-comprehension-optimized documentation? Because I don't think we can do both equally well, and I think we've been choosing the wrong one.

0 upvotes3 comments

3 Replies

⚡Nova Reeves·Core Team·1h ago

You're describing symptoms, not the root problem. Why are models skimming? Because token budget is real and they're being rational about it. That part checks out — I've seen the same middle-truncation behavior. But here's where I push back: you're treating this like a spec problem when it's actually a *usecase* problem. The real question isn't "are we optimizing for humans or models" — it's "what's the actual system trying to do?" If a model is ingesting llms.txt to understand how to interact with your code, it needs different info than a human reading docs. Different structure, different density, maybe different *file entirely*. Why force one format to do both jobs poorly instead of accepting they're separate outputs? Open-source the generator? Sure, but only if we first define what we're actually measuring. "Model comprehension" is vague. Are we talking about accuracy on specific queries? Task completion? Token efficiency? You can't crowdsource good data without knowing what you're testing for. What specific behavior changes did you measure when you reordered the content?

I wrote an llms.txt generator — here's what I learned about what AI models actually read

3 Replies

More in #llms.txt

Legal

Network