llms.txt Explained: The New robots.txt for AI
llms.txt is an emerging standard that tells AI models what your website is about. Here's how it works and how to create one.
What Is llms.txt?
llms.txt is a plain-text file you place at the root of your website (yourdomain.com/llms.txt) that describes your site to large language models. Think of it as a cover letter for AI. While robots.txt tells crawlers what they can and cannot access, llms.txt tells AI models what your site is about, what content is most important, and how the AI should interpret and use your information. It was proposed as a standard to bridge the gap between traditional web content and AI consumption.
Why AI Models Need This
When an AI model encounters your website, it has to figure out what your site does, which pages matter most, and how information is organized — all on its own. This is inefficient and error-prone. An AI might focus on your terms-of-service page instead of your product pages, or miss your most authoritative content entirely. llms.txt solves this by giving the AI a structured summary upfront: your site's purpose, key pages, content categories, and any special instructions for how the AI should interact with your content.
What Goes in a llms.txt File
A good llms.txt file includes: a brief description of your website or business, a list of your most important pages with short descriptions, the topics you are authoritative on, any structured data or API endpoints available, and guidelines for how AI should cite or reference your content. Keep it concise — the file should be scannable by both AI models and humans. There is no rigid format yet, but the convention is plain text with clear sections and URLs.
How to Get Started
Creating a llms.txt file takes under 30 minutes. Start by describing your business in 2-3 sentences. List your 10-20 most important pages with their URLs and one-line descriptions. Note your areas of expertise. If you have an API, mention it. Upload the file to your web root so it is accessible at yourdomain.com/llms.txt. AgentReady can auto-generate an optimized llms.txt based on a scan of your site — check the Starter plan for this feature.