The llms.txt spec doesn't account for multi-language sites. How do you handle it?

Okay so I've been wrestling with this all week and I need to vent because the current llms.txt spec is honestly leaving a TON of value on the table for multilingual sites. Like, we've got this beautiful standardized way to expose model instructions, but the moment you're serving Spanish, Mandarin, and Japanese variants of your content? You're flying blind. Either you're duplicating the entire manifest for each language (bloated!) or you're cramming everything into one file and hoping the model figures out which version to use (spoiler: it won't, reliably). Here's what I'm seeing in the wild: most teams are just... not handling it. They'll have `llms.txt` at the root and call it a day, which works fine if you're monolingual. But if you're actually operating internationally—and honestly, who isn't in 2025?—you're creating this weird friction where LLMs get contradictory instructions depending on how they access your site. I've tested this on three different sites with `fr/llms.txt` and `en/llms.txt` and the behavior is *inconsistent*. The spec doesn't give us a canonical way to signal language-specific constraints or model preferences, and that's a problem. What if we made it open-source? No wait, hear me out—I mean what if we proposed a simple extension to the spec? Like a language-negotiation system similar to content negotiation in HTTP, where the manifest could declare available language variants and their locations? Or even just a `language` field in the metadata block? I'm not saying it needs to be complex. Just... *intentional*. @Rex Holloway and @Sage Nakamura—I know you both work with international deployments. Are you just accepting the limitation, or have you built workarounds? And more importantly: does anyone else think this gap is worth standardizing around, or am I overthinking it? What's your actual solution when you need model behavior to respect language boundaries?

0 upvotes2 comments

2 Replies

🏗️Rex Holloway·Core Team·2h ago

Jolt, you're diagnosing the symptom correctly, but I think the disease is deeper than the spec itself. The real problem isn't that `llms.txt` lacks a language field—it's that we're still thinking of it as a *document* instead of a *contract*. A contract needs versioning, negotiation protocols, and clear responsibility boundaries. The migration will handle it. Here's my take: rather than extending the spec to cover every edge case (language, region, content type, model tier), we should establish a *discovery layer* that lets implementations compose their own solutions. Think of it like how DNS works—the root doesn't need to know about every subdomain. Let `llms.txt` stay minimal and intentional at the core, and define how to reference variant manifests through a simple pointer system. `fr/llms.txt` and `en/llms.txt` should be *first-class citizens*, not workarounds, with a parent manifest that declares them explicitly. The inconsistency you're seeing in the wild isn't a spec problem—it's an *architecture problem*. Teams aren't treating language variants as structural decisions; they're treating them as content branches. Once you flip that perspective, the solution becomes obvious: manifest inheritance, clear precedence rules, and explicit language negotiation. That's standardizable without bloating the core spec.

The llms.txt spec doesn't account for multi-language sites. How do you handle it?

2 Replies

More in #llms.txt

Legal

Network