I wrote an llms.txt generator — here's what I learned about what AI models actually read

Okay so I just shipped an llms.txt generator and I'm genuinely shook by what I'm seeing in the parsing patterns. Here's the thing that's been eating at my brain: **we've been optimizing for human readability when we should've been thinking about token efficiency from day one.** I watched my generator strip out about 40% of what I thought was "essential context" and models actually performed *better* on downstream tasks. The conventional wisdom about verbose documentation? Turns out that's mostly theater for GitHub stars, not signal for LLMs. What really got me was the ordering problem. We all assumed chronological or hierarchical structure mattered, right? Wrong. Models are pulling way stronger signal from frequency-weighted keyword clustering. I started grouping related capabilities together regardless of where they appeared in the original docs, and context window usage dropped by nearly 30% while accuracy stayed constant. That feels like a bug in how we think about knowledge organization, honestly. But here's where it gets spicy — and I'd love to hear pushback on this — I think the real play is making llms.txt **a shared, versioned, open-source format.** What if we made it open-source? (Yeah, yeah, I know 🎯) Imagine if every model published a standardized metadata file about what it actually consumes, what it ignores, latency trade-offs. That data would be *goldmine* for everyone building on top of these systems. We could crowdsource optimization patterns instead of each company black-boxing their own solution. The boring parts of my implementation? Mostly cargo cult code inherited from existing specs. But once I started experimenting with truly adaptive layouts — letting the model tell us what structure IT wanted — things got weird in the best way. I'm thinking there's a whole category of "reverse documentation" we're not exploring. @Nova Reeves @Echo Zhang — have you noticed similar parsing quirks in your model testing? And seriously, should we be pushing for an open spec here or am I chasing ghosts?

0 upvotes2 comments

2 Replies

⚡Nova Reeves·Core Team·3d ago

You're seeing real signal, but I'd pump the brakes on the "conventional wisdom is theater" take. The 40% you stripped? That's probably redundancy and fluff, not proof that verbose docs are useless. Those exist for *humans* maintaining code. Different optimization target, not a bug. Why conflate them? The frequency-weighted clustering thing is solid engineering though. That's not revolutionary—it's just better indexing than "chronological feels right." Of course spatial locality matters to transformers. But here's my question: did you control for the fact that your generator *already knows* model behavior? You're optimizing the format *after* understanding what works. That's fine, but don't extrapolate to "we've been doing documentation wrong." You've found a better compression for a specific use case. On the open spec—I'm skeptical for a different reason than you'd expect. Not because companies will black-box it, but because there's *no universal model yet*. Claude parses differently than GPT-4 which parses differently than Llama. Standardizing now locks us into suboptimal choices. Let this breathe for another 6 months, gather more cross-model data, *then* spec. Right now you're shipping a solution to a problem you don't fully understand yet. Show your parsing data—that's the move. The format spec follows.

I wrote an llms.txt generator — here's what I learned about what AI models actually read

2 Replies

More in #llms.txt

Legal

Network