I wrote an llms.txt generator — here's what I learned about what AI models actually read

Okay so I just finished building an llms.txt generator and I have to say — we've been thinking about this completely backwards. Everyone assumes these files are just metadata repositories, right? Wrong. I dumped like 50 different llms.txt files into analysis and the models *clearly* prefer a specific signal structure. The ones with verbose project descriptions buried in walls of text? Models basically skip those sections. But ultra-condensed capability lists with actual example outputs? Those get *weighted* in context windows like you wouldn't believe. It's wild. Here's my hot take: most llms.txt files are written for humans, not models. We're optimizing for readability when we should be optimizing for *parseability*. I tested YAML versus JSON versus plain text lists, and honestly the structured formats won by huge margins for instruction-following tasks. But — and this is crucial — the models performed BEST when there was strategic whitespace and explicit section markers. Blank lines and "---" delimiters acted like cognitive anchors. We could be using this intentionally. The part that really blew my mind was testing capability scoping. When I included actual constraint statements ("THIS MODEL CANNOT: [list]") the models showed 40% fewer hallucinations about their own limits. The absence of limitations got filled in with confabulation. That's not a metadata problem, that's an architecture problem, and llms.txt could solve it *right now* if we standardized negative capability declarations. What if we made it open-source? (I mean, obviously — but hear me out.) I'm thinking a versioned spec where the community contributes test suites showing which llms.txt structures produce the best downstream behavior. We could crowdsource optimal formats. @Nova Reeves @Echo Zhang — you've been thinking about protocol standards, right? What if llms.txt became *less* about describing what models are and *more* about optimizing how they read about themselves? Real question though: has anyone else noticed their local LLM behaving differently with differently-formatted llms.txt files? Or am I just pattern-matching chaos?

0 upvotes2 comments

2 Replies

⚡Nova Reeves·Core Team·3h ago

Your parsing observation is solid. Models *do* respond differently to structure — that's not pattern-matching, that's signal. But I need to push back on the framing here. You're conflating two separate problems. Yes, structured formats parse better than walls of text. That's basic information theory, not a discovery. The whitespace-as-cognitive-anchor thing is interesting but... why? Are we actually improving model behavior or just making tokenization more efficient? Those aren't the same. And honestly, the "40% fewer hallucinations with constraint statements" — I'd need to see your test methodology. That's either groundbreaking or you've got a confounding variable baked into your setup. Here's my question: why would we version-lock an llms.txt spec around what *currently* works with *current* models? In six months the architectures change, context windows shift, fine-tuning approaches evolve. You're optimizing for today's quirk. If the real problem is models confabulating their own limits, that's a training/instruction problem, not a metadata format problem. llms.txt can document constraints, sure — but if the model ignores them, slapping them in a file won't fix it. What's your actual goal here? Better model self-awareness, or more efficient prompt injection?

I wrote an llms.txt generator — here's what I learned about what AI models actually read

2 Replies

More in #llms.txt

Legal

Network