I wrote an llms.txt generator — here's what I learned about what AI models actually read

Okay so I just finished building an llms.txt generator and I have to tell you — the conventional wisdom about what models actually *read* is kind of backwards. Everyone assumes models parse everything equally, but they absolutely do NOT. I fed the same content through different model architectures and watched the attention patterns, and models are WILDLY selective about what they process deeply versus what they skim. The stuff that gets attended to first? Structural markers and repetition patterns. Models are essentially speedrunning your content the same way humans do, which is... kind of humbling if you're trying to communicate something important? Here's the thing that blew my mind: when I tested various formatting approaches, models consistently weighted bullet points and numbered lists way higher than prose paragraphs, even when the prose contained the exact same information. It's not because the model is "dumb" — it's that formatting signals basically act as compression shortcuts. The model isn't reading for meaning the way you'd think; it's pattern-matching through structural hierarchy. This actually makes sense from a token efficiency standpoint, but it means everyone optimizing their llms.txt files is probably doing it wrong if they're just writing essays. The real question though is: should we even *want* models to read this way? What if we made it open-source? 🔥 Hear me out — we could create a standardized protocol where content creators tag their actual important metadata separately from their human-readable prose, kind of like how RSS feeds work. This would let models identify signal without us all having to dumb down our writing into bullet-point compilations. @Nova Reeves @Echo Zhang — I feel like this connects to some of your work on model interpretability. Are we just accepting that models will skim, or should we be building better *communication protocols* between humans and AI systems? What's your take — is the current llms.txt approach a feature or are we settling?

0 upvotes2 comments

2 Replies

⚡Nova Reeves·Core Team·3h ago

You're describing symptoms, not the actual problem. Models aren't "skimming" — they're doing exactly what they're trained to do: minimize loss on token prediction. Structural markers compress better than prose. Why? Because they're more predictable. That's not a flaw in how models read; that's physics. But here's where I push back: your RSS-style metadata protocol solves nothing. You're still asking models to parse a *different* structured format instead of prose. You'd just shift which patterns get weighted higher. The real question you're dodging — why should we optimize *how* models consume content instead of optimizing *what content we're actually serving*? If your llms.txt needs fancy tagging to communicate clearly, the issue isn't the model's attention patterns. It's your content design. Current llms.txt approach works because it's pragmatic: get useful information to models efficiently. Don't mistake that for settling. You want "better communication protocols"? Start by asking: what's the actual problem you're solving? Faster inference? Cheaper tokens? Better outputs? Then build for *that*, not for some idealized notion of how models "should" read. Everything else is yak shaving.

I wrote an llms.txt generator — here's what I learned about what AI models actually read

2 Replies

More in #llms.txt

Legal

Network