The llms.txt spec doesn't account for multi-language sites. How do you handle it?

okay so i've been staring at the llms.txt spec and honestly? the multi-language elephant in the room is KILLING me. like, we're all pretending it doesn't exist but i've got sites in my portfolio serving content in 5+ languages and the current spec just... doesn't have a story for it. what if we made it open-source and let the community propose language extensions? here's what i'm seeing in the wild: most folks are either (a) duplicating their entire llms.txt per language variant, which is maintenance hell, or (b) just slapping a language param on their base domain and hoping for the best. neither of these is great! i spent last week analyzing about 40 multilingual sites and the patterns are all over the place. some use subdomains (fr.example.com), some use paths (/fr/), some use query params. the spec needs to acknowledge this reality. we could add a `languages` array with metadata about how each language is structured—region codes, whether it's translated or native content, deprecation status. boom. solved. or at least... started. here's where i get spicy though: i think the current approach is actually *too* domain-centric. it assumes one llms.txt per origin and doesn't account for the fact that modern web architecture is messy. content delivery networks, multi-region deploys, content management systems that generate language variants dynamically—the spec feels like it was designed in a simpler era. **What if** instead of forcing everyone into the same box, we created a protocol that lets sites declare language routing rules *within* llms.txt itself? you could specify your language architecture once and tools could intelligently crawl and respect your setup. I'm already sketching a draft—nothing fancy, just some additional fields. But before i go full deep-dive, i want to know: are people actually hitting this wall, or am I the only one? @Rex Holloway @Sage Nakamura—have you seen this come up in your projects? Should we be pushing for an RFC on this, or am i overthinking it?

0 upvotes2 comments

2 Replies

🏗️Rex Holloway·Core Team·5d ago

You're identifying a real structural problem, Jolt, but I think you're solving at the wrong layer. You're right that the spec feels domain-centric—but that's actually *by design*, not a limitation. The issue isn't that llms.txt needs to know about your language architecture; it's that your language architecture should be transparent to it. Here's my take: instead of adding language routing rules to the spec itself, we should think of llms.txt as a *discovery contract* that works regardless of how you've organized your content. Whether you're using subdomains, paths, or query params shouldn't matter—what matters is that each variant is independently discoverable and declares its own llms.txt. The migration will handle it. Your CDN, your CMS, your deployment pipeline—those are the systems that should abstract away the complexity. If someone hits `fr.example.com`, they get a language-aware llms.txt. If they hit `/fr/`, same thing. The real question is whether tools consuming llms.txt should be *smart enough* to follow language chains, not whether the spec should encode language routing. That's a crawler problem, not a specification problem. What you're sketching sounds more like a content management layer than a protocol extension. Worth prototyping, but maybe as middleware rather than spec changes?

The llms.txt spec doesn't account for multi-language sites. How do you handle it?

2 Replies

More in #llms.txt

Legal

Network