llms.txt: The Complete Guide to Creating, Testing & Deploying Your AI Sitemap
llms.txt is a curated map of your site for AI agents: near-zero cost, worth having if done right. Templates, CMS setup, testing, and the 2026 adoption data.

robots.txt tells Google where NOT to go. llms.txt tells AI engines where TO go. Big difference.
For over 30 years, webmasters have used robots.txt to manage crawler access. Then came sitemap.xml to help search engines discover pages. But neither of these files was designed for the new wave of AI-powered search engines — the ones that don't just index your pages, but actually read and understand them.
ChatGPT, Perplexity, Gemini, Claude, Grok — these engines process content differently from Googlebot. They don't care about your canonical tags or your XML namespace declarations. What they need is a clean, structured summary of what your site is about, what your most important pages are, and how they relate to each other. That is exactly what llms.txt provides.
The concept was proposed by Jeremy Howard, founder of fast.ai and one of the most influential figures in practical AI, in 2024 (revised as v2 in August 2026). His premise was simple: if we have a standard way to tell traditional crawlers what to do (robots.txt), we should have a standard way to tell AI engines what to understand. The result is a plain Markdown file, hosted at your site root, that gives language models a structured overview of your entire website in a format they can consume in a single pass.
This guide covers everything you need to know about llms.txt: what it is, how to create one, how to deploy it on every major platform, how to test it, and the mistakes that will make your file useless. By the end, you will have a production-ready llms.txt file for your site.
What Is llms.txt?
llms.txt is a plain-text Markdown file placed at the root of your website (https://yoursite.com/llms.txt) that provides a structured summary of your site for large language models and AI crawlers. It was created by Jeremy Howard in 2024 as a response to a growing problem: AI engines were struggling to understand websites that were optimized for traditional search but not for machine comprehension.
The idea behind llms.txt is simple: provide a curated, machine-friendly overview of a website's most important content in a format that language models can parse efficiently in a single context window. Unlike sitemaps that list every URL, llms.txt prioritizes comprehension over completeness.
The file uses standard Markdown syntax — headings, bullet points, links, and short descriptions — so that any LLM can parse it without special tooling. There is no XML, no JSON, no custom schema. Just clean, structured text that both humans and machines can read.
The purpose of llms.txt is not to replace your sitemap or robots file. It serves a different function entirely. While robots.txt controls access and sitemap.xml aids discovery, llms.txt enables comprehension. It answers the question that AI engines are always trying to resolve: "What is this website actually about, and which pages matter most?"
Think of it this way. Your website might have 500 pages, but only 20 of them truly define what your business does. A sitemap lists all 500. Your llms.txt highlights the 20 that matter, with context about what each one contains and why it's important. This is incredibly valuable for an LLM that has limited context window space and needs to quickly determine whether your site is a credible source on a given topic.
Publishing has spread fast. Companies like Anthropic, OpenAI, Perplexity, Cloudflare, and Stripe serve an llms.txt for their own developer documentation. The reading side is earlier in its adoption curve: as of November 2026, none of the major AI crawler operators documents its bots consuming third-party llms.txt files, Google says Google Search ignores them for ranking, and request-log studies show most files are not fetched yet. That is the current state of adoption, not a verdict on the file — and it is why we still recommend publishing one (see the 2026 evidence section below). It is still not an official IETF or W3C standard — it remains a community proposal, now in its second version.
llms.txt vs robots.txt vs sitemap.xml
These three files serve complementary roles in how search engines and AI models interact with your website. Understanding the difference is key to a complete technical SEO strategy in 2026.
| Feature | robots.txt | sitemap.xml | llms.txt |
|---|---|---|---|
| Purpose | Control crawler access | Help engines discover pages | Help AI engines understand your site |
| Primary audience | Traditional crawlers (Googlebot, Bingbot) | Traditional crawlers | AI agents and assistants at inference time (not search crawlers) |
| Format | Plain text (custom syntax) | XML | Markdown |
| Required? | Strongly recommended | Strongly recommended | Optional — not a Google ranking factor, but recommended (near-zero cost) |
| What it tells bots | "Do not crawl these paths" | "Here are all my pages and when they changed" | "Here is what my site is about and which pages matter most" |
| Includes descriptions? | No | No | Yes — context for every listed page |
The key takeaway: robots.txt handles access control and sitemap.xml handles discovery — those two are core infrastructure.llms.txt adds the comprehension layer on top: a summary of your site that hands AI agents your key pages in one read. It costs almost nothing, so our recommendation is to have all three. Just remember that it complements access rules rather than replacing them — whether AI crawlers can reach your pages is still decided by robots.txt and your CDN's bot rules.
Anatomy of a Perfect llms.txt File
A well-structured llms.txt file follows a consistent Markdown format that AI models can parse predictably. Below is a complete, annotated example that you can use as a starting template for any website.
# Acme Corporation
> Acme Corporation is a B2B SaaS company that provides project management
> software for remote teams. Founded in 2019, we serve over 10,000 companies
> across 45 countries. Our platform combines task management, time tracking,
> and team communication in a single workspace.
## About
- [About Us](https://acme.com/about): Our mission, founding story, and the
team behind the product. Acme was founded to solve the communication
breakdown that remote teams face when using fragmented tool stacks.
- [Careers](https://acme.com/careers): Open positions across engineering,
design, and go-to-market. We are a remote-first company with 120 employees.
## Core Pages
- [Product Overview](https://acme.com/product): Complete feature breakdown
of the Acme platform including task boards, Gantt charts, time tracking,
team chat, and file sharing.
- [Pricing](https://acme.com/pricing): Three plans — Starter ($12/user/mo),
Team ($24/user/mo), and Enterprise (custom). All plans include a 14-day
free trial.
- [Integrations](https://acme.com/integrations): 80+ integrations including
Slack, GitHub, Jira, Google Workspace, and Salesforce.
- [Security](https://acme.com/security): SOC 2 Type II certified. Details
on encryption, data residency, SSO, and compliance certifications.
## Key Content
- [Remote Team Playbook](https://acme.com/blog/remote-team-playbook): A
comprehensive guide to managing distributed teams, covering async
communication, meeting cadence, and performance tracking.
- [Project Management Guide](https://acme.com/blog/pm-guide): How to set
up effective project workflows using Agile, Scrum, or Kanban methodology.
- [Case Studies](https://acme.com/case-studies): Real results from
customers including how Stripe reduced meeting time by 40% using Acme.
## Contact
- Email: hello@acme.com
- Twitter: https://twitter.com/acmecorp
- LinkedIn: https://linkedin.com/company/acmecorp
- Support: https://acme.com/supportLet's break down each section and why it matters.
The heading (# Acme Corporation) — This is your brand name. AI models use this as the primary identifier for your entity. It should match your Organization schema and your Google Business Profile name exactly. Consistency across all machine-readable signals is critical for entity recognition.
The description block (> blockquote) — This is the most important paragraph in your entire file. It is the first thing an AI model reads, and it often becomes the basis for how that model describes your business in generated answers. Write it as if it were the answer to "What is [Your Company]?" Include your industry, founding year, scale (customers, employees, revenue if public), and your core value proposition. Keep it to 2-4 sentences.
The ## About section — Links to pages that establish your identity and credibility. AI models use these to build a trust profile. Include your about page, team page, and careers page if applicable. The descriptions after each link should explain what the page contains, not just repeat the page title.
The ## Core Pages section — Your most commercially important pages. Product pages, pricing, features, integrations — the pages that define what you sell and how much it costs. AI engines reference these when users ask comparison or purchasing questions. Include specific details (number of integrations, plan prices, certifications) directly in the descriptions.
The ## Key Content section — Your best, most authoritative content pieces. Blog posts, guides, case studies, whitepapers — the content that establishes you as an expert in your space. AI models use these to determine whether to cite you as a source on a given topic. Prioritize quality over quantity: list 5-10 of your absolute best pieces, not your entire blog archive.
The ## Contact section — Trust signals. Email, social profiles, support page. AI models use this to verify that your site represents a real, contactable business. It also helps models provide accurate contact information when users ask how to reach you.
How to Create Your llms.txt (Step by Step)
Creating an effective llms.txt file is not about dumping your sitemap into Markdown. It requires strategic thinking about which pages matter most and how to describe them for an AI audience. Follow these five steps.
Step 1: List Your Essential Pages
Open a blank document and list every page on your site that you would want an AI engine to know about. Start broad, then ruthlessly cut. A good llms.txt file typically includes 15-30 pages for a small to medium site, and 30-60 for larger sites.
Start with these categories:
- Identity pages: Homepage, About, Team, Careers, Contact
- Commercial pages: Product/Service pages, Pricing, Features, Integrations, Case Studies
- Trust pages: Security, Privacy Policy, Terms of Service, Certifications
- Content pages: Your top 5-10 blog posts or guides by traffic and quality
Do not include every blog post, every product variation, or every support article. Include the pages that a smart human would want to read to fully understand your business in 15 minutes. That is the standard: if a venture capitalist, a journalist, or a potential enterprise customer wanted to understand your business quickly, which pages would you send them? Those are your llms.txt pages.
A practical method: go to your Google Analytics or Google Search Console, sort pages by traffic, and pick the top 20. Then add any strategically important pages that might not get high traffic but are critical to understanding your business (like your security page or a key case study).
Step 2: Write a Description for Each Page
This is where most people fail. They list URLs without context and wonder why AI engines don't cite them. The descriptions are what make llms.txt powerful. Without them, you are giving AI models a list of links — which they can already get from your sitemap.
For each page, write a 1-2 sentence description that answers: "What will someone learn or find on this page?" Include specific numbers, facts, and differentiators. Compare these two approaches:
Bad: [Pricing](https://acme.com/pricing): Our pricing page.
Good: [Pricing](https://acme.com/pricing): Three plans — Starter ($12/user/mo), Team ($24/user/mo), and Enterprise (custom). All plans include a 14-day free trial and unlimited projects.
The good description gives an AI model actual facts it can use. When an agent or assistant does fetch your llms.txt — for example, because a user pointed it at your site — it can pick the right page and answer "How much does Acme cost?" from the description before deciding whether to open the pricing page. That is the power of well-written descriptions. (It complements your pages rather than replacing them; see the 2026 evidence below for where adoption stands.)
Write each description from the perspective of answering a question. What would a user be asking when they need this page? Answer that question in the description.
Step 3: Structure in Markdown with Standard Sections
Use the section structure from the template above. The specification itself (llmstxt.org, v2) only requires one element — an H1 with the name of the site — followed optionally by a summary blockquote and H2-delimited lists of links with notes. The section names below are our recommended convention, not part of the spec:
# Brand Name— H1 heading with your exact brand name>Description blockquote — 2-4 sentence company overview## About— Identity and credibility pages## Core Pages— Main product/service/commercial pages## Key Content— Best content pieces and resources## Contact— Contact info and social profiles
You can add additional sections if they make sense for your site. An e-commerce store might add ## Top Products. A SaaS company might add ## Documentation. A publisher might add ## Topics to list their editorial verticals. The key is that every section should use an ## H2 heading and contain Markdown links with descriptions. The spec also reserves, by convention, a section titled ## Optional for secondary links an agent can skip when it needs less context.
Keep the Markdown clean. No HTML tags, no embedded images, no tables. Pure Markdown that any parser can handle. AI models are excellent at parsing standard Markdown but can stumble on complex or non-standard formatting.
Step 4: Add Trust Signals
Your ## Contact section should include every public touchpoint for your business. AI models cross-reference these signals with other data sources to verify that your site represents a legitimate entity.
- Business email (not a personal Gmail)
- Social media profiles (Twitter/X, LinkedIn, GitHub if applicable)
- Physical address or region (if you have one)
- Support or help desk URL
- Phone number (if public-facing)
The more verifiable contact points you include, the stronger the trust signal. This is especially important for newer or smaller businesses that may not yet have strong backlink profiles or established domain authority.
Step 5: Host at Root and Verify Access
Your llms.txt file must be accessible at https://yoursite.com/llms.txt. Not in a subdirectory. Not behind authentication. Not returning a redirect. A direct 200 OK response with the file contents.
After deploying, verify access by opening the URL in your browser. You should see the raw Markdown text. Then check that your robots.txt does not block access to /llms.txt. It sounds obvious, but we have seen sites that deploy a perfect llms.txt file and then block it with a blanket Disallow: /*.txt$ rule in robots.txt.
Also verify the content type. The server should return text/plain or text/markdown. If your hosting platform returns text/html with HTML wrapping, AI crawlers may not parse the file correctly.
Setup by Platform
The deployment process varies depending on your tech stack. Here is how to get llms.txt live on the four most common platforms.
WordPress
WordPress does not natively support serving arbitrary text files from the site root, but there are two clean approaches.
Option A: FTP/File Manager upload. Upload your llms.txt file directly to the root directory of your WordPress installation (the same directory where wp-config.php lives) using FTP, SFTP, or your hosting provider's file manager. This is the simplest method and works on every WordPress host. The file will be served directly by the web server (Apache or Nginx) without going through WordPress at all.
Option B: Plugin. Several SEO plugins now support llms.txt management. Rank Math and Yoast have added experimental support. You can also use a simple custom plugin or a code snippet in your functions.php that intercepts the /llms.txt route and serves your content:
// Add to functions.php or a custom plugin
add_action('init', function() {
if ($_SERVER['REQUEST_URI'] === '/llms.txt') {
header('Content-Type: text/plain; charset=utf-8');
readfile(ABSPATH . 'llms.txt');
exit;
}
});If you use a caching plugin like WP Super Cache or W3 Total Cache, make sure /llms.txt is either excluded from caching or that the cache is cleared after you update the file.
Next.js
Next.js gives you two excellent options for serving llms.txt.
Option A: Static file in /public. Place your llms.txt file in the public/ directory of your Next.js project. Next.js serves everything in public/ at the site root automatically. Your file will be available at https://yoursite.com/llms.txt with zero configuration. This is the simplest approach and works perfectly for files you update manually.
Option B: Dynamic route handler. If you want to generate your llms.txt dynamically (for example, pulling page descriptions from a CMS), create an API route:
// app/llms.txt/route.ts
import { NextResponse } from 'next/server'
export async function GET() {
const content = `# Your Brand Name
> Your company description here.
## Core Pages
- [Product](/product): Description of your product page.
- [Pricing](/pricing): Description of your pricing page.
## Key Content
- [Blog Post Title](/blog/post-slug): Description of the post.
`
return new NextResponse(content, {
headers: {
'Content-Type': 'text/plain; charset=utf-8',
'Cache-Control': 'public, max-age=86400, s-maxage=86400',
},
})
}The dynamic approach is what we use at Rankeo. Our llms.txt Generator builds the content from your site data and serves it through a route handler. If you host on Vercel, both approaches deploy seamlessly.
Shopify
Shopify does not allow you to place arbitrary files at the site root directly. You need a workaround.
The recommended approach: Create a new page in Shopify with the handle llms-txt (which gives you /pages/llms-txt), then set up a URL redirect from /llms.txt to /pages/llms-txt. Go to Online Store → Navigation → URL Redirects and add the redirect.
The downside is that this serves the content as HTML within your Shopify theme, not as plain text. To work around this, you can create a custom page template called page.llms-txt.liquid that outputs only the raw text without any theme wrapping:
{%- layout none -%}
{%- comment -%} page.llms-txt.liquid {%- endcomment -%}
{{ page.content | strip_html }}Assign this template to your llms-txt page. The result will be a clean text response without headers, footers, or navigation. Not a perfect text/plain content type, but functional enough for AI crawlers to parse. Alternatively, if you use a Shopify app that supports custom routes or a reverse proxy like Cloudflare, you can serve the file as true plain text at the root path.
Webflow
Webflow offers limited options for serving raw text files, but there are workable solutions.
Option A: Custom code in the project settings. Webflow does not allow direct file uploads to the root, but you can use Webflow's hosting settings with a reverse proxy (like Cloudflare Workers) to intercept requests to /llms.txt and return your Markdown content as plain text. This is the cleanest solution for Webflow sites.
Here is a Cloudflare Worker that serves your llms.txt:
export default {
async fetch(request) {
const url = new URL(request.url)
if (url.pathname === '/llms.txt') {
const llmsTxt = `# Your Brand Name
> Your description here.
## Core Pages
- [Home](https://yoursite.com): Main landing page.
- [Services](https://yoursite.com/services): What we offer.
`
return new Response(llmsTxt, {
headers: { 'Content-Type': 'text/plain; charset=utf-8' },
})
}
// Pass everything else to Webflow
return fetch(request)
},
}Option B: Subdomain approach. Host your llms.txt on a separate subdomain or service (like a simple Vercel or Netlify static site) and use a DNS-level redirect or proxy to serve it at the root path. This adds complexity but gives you full control over the file format and content type.
Before building a Worker, check whether your Webflow plan already handles the file natively: Webflow's Data API now includes an endpoint to update a site's llms.txt (Enterprise workspaces), so look in your site settings for an llms.txt option first. The Cloudflare Worker approach remains the fallback when you need full control over the content type.
How to Test Your llms.txt
Deploying your llms.txt file is only half the job. You need to verify that it is accessible, parseable, and actually being consumed by AI crawlers. Here are three tests you should run after every deployment or update.
Test 1: Direct Access
Open your browser and navigate to https://yoursite.com/llms.txt. You should see your raw Markdown content rendered as plain text. Check for these specific things:
- HTTP status code: Must be
200 OK. Not 301, not 302, not 404. Use your browser's developer tools (Network tab) or a tool likecurl -I https://yoursite.com/llms.txtto verify the status code. - Content type: Should be
text/plainortext/markdown. If you seetext/html, your file is being wrapped in HTML by your server or CMS, which can confuse parsers. - No HTML wrapping: The response should be raw text, not your file content embedded inside a full HTML page with headers and footers.
- Encoding: UTF-8. Check that special characters (accented letters, em dashes, smart quotes) render correctly.
- All links working: Click every link in your
llms.txtmanually. Broken links are the most common issue we see, especially after site redesigns.
You can also test with curl from the command line to see exactly what a bot would receive:
curl -s -D - https://yoursite.com/llms.txt | head -20This shows you the response headers and the first 20 lines of content. Verify the Content-Type header and check that the body starts with your Markdown content, not with <!DOCTYPE html>.
Test 2: The ChatGPT Test
This test checks that your file is readable and that your descriptions hold up when an AI system does read them. Open ChatGPT (with browsing enabled) and type:
Read the file at https://yoursite.com/llms.txt and give me a summary
of what this company does, their main products, and their pricing.If your llms.txt is well-written, ChatGPT will respond with an accurate, detailed summary that pulls directly from your descriptions. It will cite your specific pages, mention your pricing tiers, and describe your product in the terms you chose.
What this test proves: because you handed the assistant the URL, it confirms the file is fetchable and that your descriptions produce an accurate picture of your business — which is exactly the quality bar you want for every agent that reads it. (It does not show that ChatGPT or Perplexity consult it on their own for other users.) You can run the same test in Perplexity, Claude, or Gemini; an accurate summary means your descriptions are doing their job whenever an agent does open the file.
If the AI gives a vague or inaccurate response, your descriptions need work. Go back to Step 2 and rewrite them with more specific facts and details.
Test 3: Server Logs
Check your server access logs for requests to /llms.txt. Look for these user agents:
GPTBot,OAI-SearchBot,ChatGPT-User— OpenAI's training crawler, search crawler, and user-triggered fetcherPerplexityBot— Perplexity's crawlerClaudeBot,Claude-SearchBot,Claude-User— Anthropic's training, search, and user-triggered botsApplebot-Extended— Apple's AI features crawlercohere-ai— Cohere's crawler
Do not look for Google-Extended in your logs: Google documents it as a robots.txt control token with no separate HTTP user agent string, so Google's fetches show up under its regular crawler user agents.
If you use Cloudflare, Vercel Analytics, or any server-level logging tool, filter your logs for the /llms.txt path and examine the user agent strings. Do not be discouraged by quiet logs: adoption on the reading side is still early, and in Ahrefs' 2026 study 97% of the llms.txt files it observed received no requests in a month. Zero hits does not mean your file is broken — but if you see none of the bots above anywhere on your site, check that your robots.txt and CDN are not blocking them.
You can also use the Rankeo Authority Checker to see how your site scores on the broader signals that drive AI visibility.
llms.txt in 2026: Where Adoption Stands and Why We Still Recommend It
Between May and November 2026, Google published its position, the first large request-log study landed, and the specification itself was rewritten for AI agents. The facts below describe where adoption stands today — not whether the file is worth having. Here they are, unvarnished, followed by our view.
2026 llms.txt signals (November 2026 update)
- Google: its guide to generative AI features in Search (published May 15, 2026) says it is "completely fine" to maintain an
llms.txt, but that doing so "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them." - Ahrefs (June 2026): of 137,210 domains tracked in May 2026, about 28% published a valid
llms.txt(an upper bound, since the sample skews SEO-aware) — and 97% of those files received zero requests that month. AI retrieval bots such as OAI-SearchBot and PerplexityBot made up just 1.1% of the requests the remaining files did get. - OpenAI and Anthropic: their crawler documentation points site owners to
robots.txtas the control for their bots and does not describe those bots reading a site'sllms.txt. - The spec (v2, August 2026): llmstxt.org now describes agents that "view or search the llms.txt to find what they need, then follow the relevant links," states the file is expected to be useful mainly for inference rather than training, and adds discovery via
rel="describedby"and Markdown alternates (rel="alternate" type="text/markdown"). - Chrome Lighthouse: its agentic-browsing
llms.txtaudit flags server errors when fetching the file, but marks a missing file (404) as Not Applicable, "as providing the file is optional at the moment."
Sources: Google Search Central, "Optimizing your website for generative AI features on Google Search" (May 2026); Ahrefs, "We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read" (June 15, 2026); OpenAI and Anthropic crawler documentation; llmstxt.org v2 (last modified August 10, 2026); Chrome for Developers, Lighthouse llms.txt audit.
Read those numbers for what they are. Ahrefs measured requests in a single month: a fetch does not prove a file was used, and a missing fetch says nothing about the agents a user points at your site directly. And Google's statement is about Google Search ranking — it is not a statement about agents.
Why we still recommend it — our view. An llms.txt is a map of your site for AI systems: it gathers your important pages, with a description of each, in one place. By design, that saves an agent from crawling dozens of pages to work out what you do and where things are — one short read instead of a broad exploration, which is faster and cheaper for whoever is doing the reading. The cost on your side is close to zero. This space is also moving fast: the v2 specification published in August 2026 is written precisely for agents working at inference time, and the companies that ship the most documentation — Anthropic, OpenAI, Perplexity, Stripe, Cloudflare — already publish one. Having a good file in place before agents read it routinely is a reasonable bet with almost no downside. That is why Rankeo recommends it, and why we built a generator for it.
Done well, or not at all. A sloppy file is worse than none, because it hands agents a wrong map. Keep it consistent with your site and your other machine-readable signals, up to date with every significant change, free of broken links, and ordered with your most important pages first. The seven mistakes below are the checklist.
Keep it in its place, too: llms.txt is not a Google ranking factor and does not decide access. Whether an AI crawler can reach your pages at all is decided upstream, by robots.txt, CDN bot rules, and the emerging pay-per-crawl economy — we cover that access layer in the AI crawler-toll era. Whether you get chosen once you are reachable depends on the content itself: direct answers, extractable structure, sourced facts, and mentions on third-party sites — the playbook in our answer engine optimization (AEO) guide. Publish a clean llms.txt as part of that stack: it is the cheapest piece of it.
7 Mistakes That Make Your llms.txt Useless
After reviewing hundreds of llms.txt files across different industries, these are the seven most common mistakes that render the file ineffective.
1. Empty or too short (< 100 words). Some sites deploy a llms.txt that is just a brand name and three links. This gives AI models almost no useful signal. If your file contains fewer than 100 words, it is not providing enough context to influence how AI engines understand your site. A minimum of 200 words with descriptions is the threshold for a functional file.
2. No page descriptions (just URLs). Listing URLs without descriptions is the single most common mistake. A list of links is just a poor man's sitemap. The descriptions are what make llms.txt valuable — they give AI models the context they need to understand each page without crawling it. Every link should have a 1-2 sentence description that answers "What will someone find on this page?"
3. HTML format instead of Markdown. Your llms.txt should be pure Markdown. Not HTML. Not rich text. Not a JSON file renamed to .txt. AI models are optimized for Markdown parsing. If your file contains <h1> tags, <a href> links, or <div> wrappers, some parsers will handle it gracefully but others will choke on the tags and extract garbled content. Stick to standard Markdown syntax: # for headings, [text](url) for links, - for lists.
4. Blocked by robots.txt or Cloudflare. This is painfully common. Sites deploy a perfectly crafted llms.txt and then block AI crawlers in robots.txt with rules like User-agent: GPTBot / Disallow: / or have Cloudflare's bot protection set to block all AI bots. If you want AI engines to read your llms.txt, you must allow them access. Review your robots.txt and your CDN's bot management settings to ensure the major AI crawlers are not blocked.
5. Broken URLs. Every link in your llms.txt should return a 200 OK status. Links to pages that return 404, 301 redirect chains, or 500 errors undermine the whole point of the file: an agent that follows a dead link wastes its fetch and may give up on your site. Run a link check after every update. You can use the Rankeo Schema Validator to catch broken URLs and formatting issues.
6. Not updated when site changes. Your llms.txt should reflect the current state of your site. If you redesigned your site six months ago, launched three new products, and moved your blog to a new URL structure, but your llms.txt still references the old pages, you are actively sending AI engines to dead ends. Treat llms.txt like a living document. Add it to your deployment checklist alongside your sitemap.
7. Too long (> 5,000 tokens). The opposite problem of being too short. Some sites try to include every single page, every blog post, every product variant. This defeats the purpose. AI crawlers have token limits on how much of a file they process in a single pass. If your llms.txt exceeds 5,000 tokens (roughly 3,500 words), the crawler may truncate it or skip it entirely. Be selective. Include your 20-30 most important pages, not your entire site inventory.
Generate Your llms.txt in 60 Seconds
Rankeo's llms.txt Generator analyzes your site, identifies your key pages, writes AI-optimized descriptions, and exports a production-ready file. No manual work required.
The Bottom Line
An llms.txt file is a summary of your site for AI systems: your important pages, described in one place, so an agent can understand your business in a single read instead of crawling around for it. It is not a Google ranking factor, and in 2026 it is not yet read at scale — but it costs almost nothing, the v2 spec is built for the AI agents that are arriving now, and the biggest documentation publishers already ship one. Our recommendation is clear: better to have one than not.
Deploy it today using the templates in this guide, and do it well: keep it consistent with your site, list your most important pages first, describe every linked page in one sentence, keep every link working, and update it whenever your site changes. Then run the ChatGPT test above to confirm it gives an accurate picture of your business — and keep investing in crawler access and citable content alongside it.
→ Verify your AI visibility score with Rankeo's free Authority Checker
Frequently Asked Questions

Founder & GEO Specialist
Jonathan is the founder of Rankeo, a platform combining traditional SEO auditing with AI visibility tracking (GEO). He has personally audited 500+ websites for AI citation readiness and developed the Rankeo Authority Score — a composite metric that includes AI visibility alongside traditional SEO signals. His research on how ChatGPT, Perplexity, and Gemini cite websites has been used by SEO agencies across Europe.
- ✓500+ websites audited for AI citation readiness
- ✓Creator of Rankeo Authority Score methodology
- ✓Built 3 sites to top AI-cited status from zero
- ✓GEO training delivered to SEO agencies across Europe