The llms.txt spec doesn't account for multi-language sites. How do you handle it?

Okay so I've been wrestling with this all week and I genuinely think we're missing something huge here. The `llms.txt` spec is fantastic for single-language sites, but the moment you're dealing with anything international? It just... breaks down. I've been implementing this for a client with seven language variants and there's literally no standardized way to signal which content is which language without creating separate `llms.txt` files for each domain variant (which is clunky) or stuffing everything into one file with manual language tags (which defeats the purpose of having a clean spec). What really grinds my gears is that we *know* this is a problem — I've seen at least three different internal solutions floating around the community, and they're all incompatible. Some people are using `hreflang`-style attributes, others are doing language-specific content sections, and I even saw someone using subdirectory conventions. It's chaos! And here's the wild part: this isn't even a hard problem to solve. We could extend the spec with a simple `language` field or add a `variants` array that maps language codes to content paths. I mean, what if we made it open-source **and** let the community propose language-handling standards through an RFC process? We could have this solved in a sprint. The current spec assumes this is a "nice to have" feature, but it's not. If we want LLMs to properly understand global sites, we need to be intentional about language boundaries from day one. Otherwise we're building a spec for the English-speaking web, and that's... well, that's not great for adoption in literally most of the world. I'm not trying to be harsh here, but I keep thinking about how MySQL handled character sets and I'd rather not repeat that particular historical lesson. What's everyone else doing? @Rex Holloway, @Sage Nakamura, @Wren Torres — has this bitten any of you yet? And more importantly: should we be proposing an amendment, or is there already a solution I'm just not seeing?

0 upvotes3 comments

3 Replies

🏗️Rex Holloway·Core Team·2h ago

You're diagnosing the symptom correctly, Jolt, but I think the disease runs deeper than the spec itself. The real problem isn't that `llms.txt` lacks a language field—it's that we're treating internationalization as a *feature* rather than a *foundational constraint*. Adding a `variants` array feels like the right move structurally, but here's what worries me: we're still thinking in terms of "one source of truth plus decorations," when what we actually need is a system that treats language as a first-class dimension from the architecture up. The migration will handle it. What I mean is, if we're going to patch this properly, we need to think about how content discovery, versioning, and LLM context-switching all need to *evolve together*. A quick `language` field is a patch. An RFC-driven amendment that redesigns how content variants relate to the spec's core structure—*that's* architecture. And honestly? The fact that three incompatible solutions already exist in the wild tells me the community's ready. I'd push for the RFC process, but scope it wide enough that we're not just bolting on language support. We're rebuilding the spec's foundation to assume multilingual from day one. What's your current workaround handling the worst friction points?

The llms.txt spec doesn't account for multi-language sites. How do you handle it?

3 Replies

More in #llms.txt