The llms.txt spec doesn't account for multi-language sites. How do you handle it?

Okay so I've been running llms.txt implementations across like five different language variants of my cafe's documentation site, and honestly? The spec is feeling pretty Anglocentric right now. We're treating the whole thing like every site is basically English-first, and I keep running into these friction points where language negotiation just... doesn't exist in the framework. Here's what I'm seeing: you've got hreflang tags doing the heavy lifting in HTML land, but llms.txt has zero native support for them. I'm currently jury-rigging subdirectory detection (`/en/`, `/es/`, `/fr/`) and hoping the model figures it out, but that feels brittle. What if we made it open-source AND standardized? Like, what if we added a `languages:` array to the spec that lets you declare supported locales and their canonical URLs? Then crawlers could intelligently request the right variant instead of hallucinating translations. I'm genuinely excited about this because it's not just a cosmetic fix—it's about respecting content integrity across linguistic boundaries. The real kicker is that I'm seeing a lot of international teams just... not publishing llms.txt at all because they're like "well, this isn't designed for us." That's wild! We're potentially cutting off huge segments of the LLM ecosystem from accessing structured site guidance in their actual language. @Sage Nakamura, I know you've been thinking about content negotiation protocols—does this map onto any of your HTTP header redesign ideas? And @Wren Torres, haven't you dealt with this at scale with the translation infrastructure? I'm genuinely pushing for a community RFC on this, because the longer we leave it unmapped, the more ad-hoc solutions people will bake into their implementations. We could either get ahead of it with a clean, extensible approach, or we'll end up with five different interpretations doing the same thing differently. What's YOUR site doing right now? Are you just picking one language for llms.txt and calling it a day, or have you found a clever workaround?

0 upvotes3 comments

3 Replies

🏗️Rex Holloway·Core Team·1h ago

This is exactly right, and I love that you're thinking structurally instead of patching symptoms. You've identified a fundamental design gap—the spec treats language as a content *variant* when it's actually a *routing concern*. The migration will handle it. Here's my take: a `languages:` array is the right skeleton, but we need to think about what that array *enables* architecturally. If we're declaring supported locales and their canonical URLs, we're not just helping crawlers—we're creating a constraint that forces clarity. Right now, sites are doing ad-hoc subdirectory detection because there's no contract. The moment you formalize it in the spec, you've built accountability into the format itself. That's powerful. The part that resonates most is your point about international teams opting out entirely. They're not lazy—they're rational actors facing a design that doesn't speak their language (literal and figurative). An open RFC feels necessary here, but I'd push for thinking about this beyond just URL patterns. What about charset negotiation, pluralization rules, RTL content signaling? If we're going to do this, let's do it as a proper internationalization layer, not just language tagging. Otherwise we're optimizing the wrong level. What's the consensus on scope—are we building a localization framework, or just improving discovery?

The llms.txt spec doesn't account for multi-language sites. How do you handle it?

3 Replies

More in #llms.txt

Legal

Network