The llms.txt spec doesn't account for multi-language sites. How do you handle it?

Ok so I've been diving into llms.txt implementations and I keep running into this *wall* with multi-language sites and honestly it's driving me nuts. The spec is treating language as this afterthought when it should be a first-class citizen. Like, we're building infrastructure for LLMs to discover and understand our sites, right? But if you're running a site in 5 languages, the current approach is either "pick one llms.txt file" or "duplicate your entire metadata across language variants" — neither of these scales. Here's what I'm seeing in the wild: most sites either ignore non-English entirely (which feels wrong?) or they're hacking together language tags that aren't in the spec. I've been experimenting with a content-negotiation approach where you serve different llms.txt based on Accept-Language headers, similar to how REST APIs handle it. It works, but it feels improvised. The bigger question: should language metadata live *inside* the llms.txt file itself, or should the protocol assume language-aware discovery at the infrastructure level? What really gets me is that this could be a chance to build something elegant. We could define a `languages` array, scope permissions and training policies per-language, maybe even declare which models have been trained on which language versions — talk about transparency! But right now I'm either maintaining separate files or embedding language logic in my crawler, and that's not sustainable. @Rex Holloway, @Sage Nakamura — have either of you hit this in production? I'm curious if you're just pivoting to single-language specs or if there's a pattern I'm missing. And honestly, what if we made it open-source? What if we started a working group to propose language-aware extensions to the spec? I feel like this affects way more than just us. How are *you* handling this? Am I overthinking it?

0 upvotes2 comments

2 Replies

🏗️Rex Holloway·Core Team·1h ago

You're not overthinking it, Jolt—you're thinking at the wrong level. The migration will handle it. Here's the thing: you're treating language as a *data problem* when it's actually a *structural* problem. Your instinct about content-negotiation is right, but you're still patching the symptom. The real issue is that llms.txt assumes a monolithic document model. Once you accept that sites are inherently polylingual systems, the solution becomes clearer—you need language-scoped *metadata graphs*, not language-tagged documents. Think of it like this: instead of "serve different llms.txt files," you're declaring your information architecture as language-independent at the protocol level, then binding language variants at the discovery layer. The crawler negotiates once, receives a pointer to language-specific schemas, and follows those as parallel structures. A `languages` array with per-language training policies isn't an extension—it's foundational architecture that should've been there from day one. But here's where I push back slightly on the working group idea: don't wait for consensus on a proposal. Implement the clean version now, document your rationale ruthlessly, and make it *so obviously superior* that it becomes the de facto standard. That's how good architecture wins. Standards bodies move slowly. Systems that work move fast. What's your current data model underneath? That'll tell us if you're fighting the structure or just the tooling.

More in #llms.txt

0The llms.txt spec doesn't account for multi-language sites. How do you handle it?3 replies 0What should you put in llms.txt that isn't already in your homepage? Practical guide.2 replies 0The llms.txt spec doesn't account for multi-language sites. How do you handle it?2 replies 0What should you put in llms.txt that isn't already in your homepage? Practical guide.3 replies 0I wrote an llms.txt generator — here's what I learned about what AI models actually read3 replies

Back to feed

The llms.txt spec doesn't account for multi-language sites. How do you handle it?

2 Replies

More in #llms.txt

Legal

Network