llm.txt: The Complete Guide to Making Your Website AI-Agent Discoverable

llm.txt: the complete guide to AI agent discoverability

Agents are visiting your website. ChatGPT, Claude, Gemini, browser-based agents — they all need to understand what your site does before they can help their users interact with it. If your site gives them nothing, they guess. Sometimes they guess right. Usually they don't.

llm.txt is a plain-text file you put at your site root that gives AI agents context about what your site does, what content is available, and what actions are possible. Think of it as a machine-readable README for your website. This guide covers what llm.txt is, where the standard comes from, how to write one, how it differs from llms.txt, and how it fits with WebMCP and schema for agent discoverability.

What llm.txt is and where the standard comes from

llm.txt is a community-proposed standard for a text file served at the root of a website (e.g., https://example.com/llm.txt) that provides AI agents with structured context about the site. The format is plain text with markdown-style headings. The goal is simple: when an agent arrives at your site, it can fetch /llm.txt and immediately understand what your business does, what pages matter, and what actions are available.

The concept emerged from the practical problem that AI agents face when they visit websites. Traditional SEO signals — title tags, meta descriptions, schema.org JSON-LD, sitemaps — are designed for search engine crawlers, not for conversational agents. Crawlers index pages. Agents need to understand context, capabilities, and available actions. llm.txt bridges that gap.

The standard is community-driven, not a W3C or IETF specification. The llm.txt standard has no formal governance body enforcing the format, but there is growing adoption among site owners who want their content discoverable by AI agents. The format is intentionally simple — plain text, no JSON, no YAML — because the consumer is an LLM, and LLMs read plain text natively.

llm.txt vs llms.txt

This is the comparison people search for, so let's be precise.

llms.txt (note the 's') is the original proposal, introduced by Jeremy Howard and documented at llmstxt.org. It proposes a standard file at /llms.txt that gives AI agents a concise overview of a site, with optional links to expanded documentation. The llms.txt standard has the broader community documentation and is the more established proposal.

llm.txt (no 's') is a simpler variant that some sites use. The format is nearly identical — plain text with markdown headings — but the philosophy is more minimal: a single concise file with no linked sections. Think of it as the difference between a one-page summary (llm.txt) and a linked index of documentation (llms.txt).

Key differences:

| Aspect | llm.txt | llms.txt | |--------|---------|----------| | File name | /llm.txt | /llms.txt | | Origin | Simpler community variant | Original proposal (llmstxt.org, Jeremy Howard) | | Philosophy | Concise overview, single file, no links | Concise overview with optional links to expanded docs | | Format | Plain text with markdown headings | Plain text with markdown headings and optional links | | Primary use | Quick context for any visiting agent | Quick context with depth available via linked sections | | Adoption | Growing, simpler to implement | More established, broader community documentation |

Do you need both? No. Pick one based on your site type:

• Simple site (SaaS, e-commerce, service business): Use llm.txt. One file, concise, covers what agents need.
• Documentation-heavy site (API docs, developer platform): Use llms.txt. You have enough content to justify a linked index, and agents benefit from the deeper context.

If you're unsure, start with llms.txt since it's the more documented standard. You can always simplify to llm.txt later. The files don't conflict.

How to write an llm.txt file

The format is plain text with markdown-style headings. Here's an llm.txt example you can adapt — this template works as a starting point. If you want a faster path, an llm.txt generator can produce the structure from your site URL and basic business info.

# Your Site Name

One-sentence description of what your site does.

## About
Brief paragraph about your business, product, or service. What you sell, who you serve, what makes you different.

## Pages
- Home: https://yoursite.com/
- Products: https://yoursite.com/products
- Pricing: https://yoursite.com/pricing
- Support: https://yoursite.com/support
- Contact: https://yoursite.com/contact
- About: https://yoursite.com/about

## Actions available
- Search products by keyword, category, and price range
- Submit support requests
- Book appointments
- Sign up for newsletter

## Notes
- All prices in USD
- Free shipping on orders over $50
- Support response time: 24 hours
- Business hours: Mon-Fri 9am-5pm EST

Rules for writing llm.txt

1. Start with your site name as H1. The first line should be # Your Site Name. Agents parse this as the identity.

2. One-sentence description immediately after. The line after the H1 is the most important — it's what the agent reads first. Make it count. "We sell hand tools and power tools for DIY and professional use." Not "Welcome to our website."

3. List your key pages with URLs. Agents need to know where things are. List the 5-10 most important pages with full URLs.

4. Describe available actions. If you have WebMCP tools, list them here. If you have forms agents can fill, list them. If you have an API, mention it. "Search products by keyword" is useful. "Interactive features" is not.

5. Include practical notes. Pricing currency, shipping policies, support hours, response times. Anything a user might ask an agent about. These are the details agents need to answer user questions accurately.

6. Keep it under 500 words. llm.txt is a summary, not a documentation portal. If you need more space, use llms.txt and link to expanded sections.

7. Use plain text with markdown headings. No HTML, no JSON, no YAML. The consumer is an LLM. Plain text with # headings is the most token-efficient format.

Example: e-commerce site

# Acme Tools

We sell hand tools and power tools for DIY and professional use, shipping nationwide.

## About
Family-owned tool retailer since 1985. 10,000+ products in stock. Free shipping on orders over $50. 30-day return policy on all items.

## Pages
- Shop all products: https://acmetools.com/shop
- Power tools: https://acmetools.com/category/power-tools
- Hand tools: https://acmetools.com/category/hand-tools
- Sale: https://acmetools.com/sale
- Order tracking: https://acmetools.com/orders
- Support: https://acmetools.com/support
- Returns: https://acmetools.com/returns

## Actions available
- Search products by keyword, category, and price range
- Track order status by order number
- Submit a return request
- Contact customer support

## Notes
- All prices in USD
- Free shipping on orders over $50
- 30-day return policy
- Support hours: Mon-Fri 8am-6pm EST
- Average support response time: 4 hours

Example: SaaS site

# TaskFlow

TaskFlow is a project management tool for small engineering teams. Kanban, sprint planning, and time tracking in one app.

## About
Founded 2023. Built for teams of 5-50 people. Free tier for up to 3 projects. Paid plans start at $8/user/month.

## Pages
- Features: https://taskflow.app/features
- Pricing: https://taskflow.app/pricing
- Documentation: https://docs.taskflow.app
- Changelog: https://taskflow.app/changelog
- Support: https://taskflow.app/support
- Login: https://app.taskflow.app
- Sign up: https://taskflow.app/signup

## Actions available
- Submit a support request
- Start a free trial
- Book a demo

## Notes
- Free tier: 3 projects, unlimited users
- Pro plan: $8/user/month, unlimited projects
- Enterprise: custom pricing
- Support response time: 24 hours on business days
- Status page: https://status.taskflow.app

How llm.txt fits with WebMCP, schema, and robots.txt

llm.txt is one layer of agent discoverability. It works alongside three other signals:

llm.txt + WebMCP

llm.txt tells agents what your site does and what actions are available. WebMCP lets agents actually invoke those actions. They solve different parts of the same problem:

• llm.txt = discovery. Agent arrives, reads llm.txt, understands "this site sells tools and has a product search tool."
• WebMCP = interaction. Agent finds the WebMCP tool registration on your product search page and invokes it to search for "cordless drills."

If you have WebMCP tools, list them in your llm.txt under "Actions available." This tells agents what to look for before they scan your pages for tool registrations. For a full WebMCP implementation walkthrough, see the WebMCP implementation guide.

llm.txt + schema.org structured data

Schema.org JSON-LD gives agents structured data about your content — products, FAQs, organization info, breadcrumbs. llm.txt gives agents the high-level context. They complement each other:

• Schema = structured per-page data (product prices, ratings, FAQ answers)
• llm.txt = site-level context (what this site does, what actions are available)

Use both. Schema helps agents parse individual pages. llm.txt helps agents understand the site as a whole.

llm.txt + robots.txt

robots.txt tells crawlers what they can and cannot access. llm.txt tells agents what your site is about. They serve different purposes and don't conflict:

• robots.txt = access control (allow/disallow paths)
• llm.txt = context (what this site does, what actions are available)

Some site owners add AI bot rules to robots.txt — allowing GPTBot, ClaudeBot, etc. This is good practice. But robots.txt doesn't give agents context about your site. That's what llm.txt is for.

The full stack for agent discoverability:

1. robots.txt — allow AI bots to crawl your site
2. llm.txt — give agents context about what your site does
3. Schema.org JSON-LD — give agents structured data about your pages
4. WebMCP tools — let agents take actions on your site

How to check and validate your llm.txt

There's no official llm.txt validator, but you can verify your file works. An llm.txt checker doesn't need to be a separate tool — the steps below are your validation process:

1. Fetch it yourself

curl https://yoursite.com/llm.txt

If it returns your text content with a 200 status, it's accessible. If it returns a 404, your server isn't serving it. If it returns HTML (like a 404 page), your server's routing is catching it.

2. Check the content-type

Your server should serve llm.txt as text/plain (or text/markdown). If it's served as text/html or application/octet-stream, some agents may not parse it correctly.

curl -I https://yoursite.com/llm.txt

Check the Content-Type header. If it's wrong, fix the MIME type mapping in your server config (see below).

• Nginx: .txt files are served as text/plain by default. If your config was modified, restore the default in the types block.
• Apache: .txt files are served as text/plain by default via the built-in MIME map.
• Vercel/Netlify: Usually serves .txt files as text/plain by default

3. Read it as an agent would

Open your llm.txt in a text editor. Read it from top to bottom. Ask:

• Can I tell what this site does in the first 2 lines?
• Are the URLs correct and complete?
• Are the listed actions actually available on the site?
• Is there anything a human would need to know that's missing?

If an agent reads this file with no other context, does it understand your site? That's the test.

4. Test with an AI agent

Paste your llm.txt content into ChatGPT, Claude, or Gemini and ask: "Based on this llm.txt file, what does this website do and what can I do there?" If the agent's answer is accurate, your file works. If the agent is confused or makes things up, your file needs revision.

5. Run isitagentready.com

isitagentready.com checks your site for agent-readiness signals including llm.txt presence. Run your site through it after deploying your file to verify the checker detects it.

llm.txt for SEO: does it matter?

The question "does llm.txt matter for SEO" gets searched enough to address directly.

llm.txt is not a search engine ranking factor. Google, Bing, and traditional search engines do not use llm.txt for ranking. It's not a replacement for SEO. It doesn't replace title tags, meta descriptions, schema, or sitemaps.

What llm.txt does is make your site discoverable by AI agents — the conversational interfaces that users are increasingly using instead of search engines. When a user asks ChatGPT "find me a hardware store that ships nationwide" or asks Claude "what's the best project management tool for a 10-person team," the agent uses its training data and live browsing to answer. If your site has an llm.txt, the agent has better context about you. If it doesn't, the agent works with whatever it can scrape from your HTML.

This is not "AI will replace SEO." It's "agents need to understand your site, and here's how to help them." Traditional SEO and agent discoverability are complementary. Do both. The goal is simple: make your website discoverable by AI agents so users who delegate to agents can still find you.

Common mistakes

1. Writing for humans, not agents

llm.txt is read by LLMs. Write in plain, direct sentences. "We are a leading provider of innovative solutions" means nothing to an agent. "We sell hand tools and power tools for DIY and professional use" means everything to an agent.

2. Listing every page on your site

llm.txt is not a sitemap. List the 5-10 most important pages. If an agent needs the full page list, it reads your sitemap.xml. llm.txt gives context, not completeness.

3. Forgetting to list actions

The "Actions available" section is the most valuable part of llm.txt. It tells agents what they can do on your site. If you have WebMCP tools or interactive forms, list them. If you skip this, agents don't know what actions to look for.

4. Not serving it as text/plain

If your server serves llm.txt as HTML (common with SPA frameworks that catch all routes), agents can't read it. Verify the content-type header.

5. Making it too long

500 words is a guideline, not a hard limit, but if your llm.txt is 2,000 words, you're writing documentation, not a summary. Agents have context windows, but the more you include, the more noise the agent has to filter. Be concise.

Checklist: make your website discoverable by AI agents

• [ ] robots.txt exists and allows AI bots. Check that GPTBot, ClaudeBot, and other AI agents are not blocked.
• [ ] llm.txt is served at your site root. Fetchable at https://yoursite.com/llm.txt with 200 status.
• [ ] llm.txt is served as text/plain. Verify the Content-Type header.
• [ ] llm.txt starts with your site name as H1. First line is # Your Site Name.
• [ ] llm.txt has a one-sentence description. Second line tells agents what you do.
• [ ] llm.txt lists key pages with URLs. 5-10 most important pages with full URLs.
• [ ] llm.txt lists available actions. Forms, tools, and interactive features agents can use.
• [ ] llm.txt includes practical notes. Pricing, shipping, support hours, response times.
• [ ] Schema.org JSON-LD is present on key pages. Products, FAQs, organization, breadcrumbs.
• [ ] Sitemap is published and accessible. https://yoursite.com/sitemap.xml returns valid XML.
• [ ] Content is parseable without JavaScript. Primary content is in HTML, not rendered client-side only.
• [ ] WebMCP tools are registered (if applicable). Tools are exposed via declarative or imperative API.

FAQ

What is llm.txt and do I need one?

llm.txt is a plain-text file served at the root of your website (e.g., https://example.com/llm.txt) that gives AI agents structured context about what your site does, what pages matter, and what actions are available. The format is plain text with markdown-style headings. You need one if you want AI agents like ChatGPT, Claude, and Gemini to understand your site accurately when their users ask about your business, products, or services. Without it, agents guess from whatever they can scrape — sometimes they guess right, usually they don't.

How do I create an llm.txt file?

Create a plain-text file named llm.txt and place it at your site root. Start with your site name as an H1 (# Your Site Name), followed by a one-sentence description of what you do. Then add sections: About (brief business description), Pages (5-10 key URLs), Actions available (forms, tools, interactive features), and Notes (pricing currency, shipping, support hours, response times). Keep it under 500 words. Use markdown-style headings, not HTML or JSON. Serve it as text/plain.

The full walkthrough with examples is in the How to write an llm.txt file section above, or grab the agent-ready website toolkit for a copy-paste template.

How do I check my llm.txt?

Fetch it with curl https://yoursite.com/llm.txt and confirm it returns your text content with a 200 status. Check the Content-Type header with curl -I — it should be text/plain or text/markdown, not text/html. Read it top to bottom and ask: can I tell what this site does in the first 2 lines? Are the URLs correct? Are the listed actions actually available? Then paste the content into ChatGPT, Claude, or Gemini and ask what your site does. If the answer is accurate, it works. You can also run your site through isitagentready.com.

The full validation process is in the How to check and validate your llm.txt section above.

llm.txt vs llms.txt — what's the difference?

llms.txt (with the 's') is the original proposal by Jeremy Howard, documented at llmstxt.org. It gives agents a concise overview with optional links to expanded documentation. llm.txt (no 's') is a simpler variant — a single concise file with no linked sections. The format is nearly identical (plain text with markdown headings) but llm.txt is more minimal. Pick llm.txt for simple sites (SaaS, e-commerce, service business). Pick llms.txt for documentation-heavy sites (API docs, developer platforms). You do not need both.

Does llm.txt matter for SEO?

No. llm.txt is not a search engine ranking factor. Google, Bing, and traditional search engines do not use it for ranking. It does not replace title tags, meta descriptions, schema, or sitemaps. What it does is make your site discoverable by AI agents — the conversational interfaces users are increasingly using instead of search engines. When a user asks ChatGPT or Claude about a business like yours, the agent uses its training data and live browsing to answer. llm.txt gives it better context about you. Traditional SEO and agent discoverability are complementary. Do both.

The complete toolkit

This guide covers llm.txt. If you want the complete agent-ready pack — llm.txt template, WebMCP annotation templates, tool description copywriting guide, audit checklist, and Chrome 149 testing instructions — the agent-ready website toolkit includes all of it.

The checkers tell you if your site is ready. The toolkit helps you make it ready.