llms.txt Optimization for AI Search Visibility: Setup Guide
TL;DR: llms.txt is a plain markdown file at your site's root that gives AI systems a curated map of your most important pages, in a format built for language models rather than browsers. It sits alongside your sitemap.xml but serves a different purpose: instead of listing every URL, it lists the pages worth reading first, each with a one-line description. A well-formed llms.txt starts with a single H1 for your site name, a short blockquote summarising what the site does, then H2 sections that group links by topic. The highest-leverage move is generating the file automatically from the same content source that builds your site, so it never drifts out of date the way a hand-maintained file inevitably does. Treat it as part of your deploy pipeline rather than a one-off task, and keep it short enough that an LLM reading it in one pass comes away with a genuinely useful map, not a dump of every page you have ever published.
What Is llms.txt and Why Does It Matter for AI Search?
llms.txt is a proposed convention, similar in spirit to robots.txt, that lives at the root of a domain (/llms.txt) and gives large language models a concise, structured summary of a site's content. Where robots.txt tells crawlers what they can access, llms.txt tells any AI system that fetches it what the site actually contains and which pages matter most.
The reason this matters now is straightforward: a growing share of how people find and evaluate content happens through AI assistants that browse, summarise, and cite sources, rather than through a traditional search results page. When one of those systems needs to describe your site, answer a question about your product, or decide whether to link to you, it works from whatever context it can retrieve quickly. A full HTML page, with navigation, ads, cookie banners, and scripts, is expensive and noisy for a model to parse. A short, curated markdown file that states exactly what the site is and where the important pages live is far easier to use correctly.
This is not a replacement for good on-page SEO or a solid sitemap. It is an additional, purpose-built entry point aimed specifically at the systems that now sit between a lot of readers and your content.
What Should You Actually Include in an llms.txt File?
The temptation is to treat llms.txt like a sitemap and list everything. Resist it. A sitemap's job is completeness, so search engines can discover and index every URL. An llms.txt file's job is curation, so a language model reading it once gets a genuinely useful sense of the site without wading through years of minor updates.
A reasonable structure:
- One line identifying the site and what it does.
- A short blockquote, two or three sentences, giving context a model would otherwise have to infer.
- Sections grouped by topic (documentation, guides, product pages, key reference material), each with a handful of links.
- A description on every link, written the way you would describe the page to a colleague, not the way you would write a meta description for a search snippet.
Durable, canonical content belongs here: setup guides, core documentation, pricing, and reference pages that stay accurate for months. Time-sensitive posts and minor updates usually do not, unless the file is regenerated often enough that stale entries never accumulate.
How Do You Structure an llms.txt File Correctly?
The emerging convention uses plain markdown with a specific shape: an H1 with the site name, a blockquote summary, then H2 sections containing a bullet list of links.
# Yanni.uk
Technical documentation and build logs covering Claude-based automation, deployment pipelines, and SEO tooling for a portfolio of independently run sites.
## Documentation
- [MCP Server with Python FastMCP](https://yanni.uk/technical-documentation/build-mcp-server-python-fastmcp/): Build and deploy a Python MCP server using the FastMCP framework.
- [Astro on Cloudflare Pages](https://yanni.uk/technical-documentation/deploy-astro-cloudflare-pages/): Deploy an Astro site to Cloudflare Pages with correct build settings.
## Guides
- [IndexNow and Sitemap Automation](https://yanni.uk/technical-documentation/indexnow-sitemap-automation/): Automatically ping search engines when new pages publish.
- [CI/CD Pipelines for Claude Projects](https://yanni.uk/technical-documentation/cicd-pipelines-claude-projects/): Wire a build and deploy pipeline around Claude-assisted projects.
Keep the sections few and the descriptions honest. A model reading this should be able to answer "what is this site, and what are its three most important pages" from the file alone.
How Do You Generate llms.txt Automatically from Your Content?
A hand-maintained llms.txt goes stale the same way a hand-maintained sitemap does: someone ships a new guide, forgets to update the file, and within a month the two disagree about what the site contains. The fix is the same in both cases: generate it from the source of truth at build time rather than editing it directly.
If your content lives as markdown files with front matter, an Astro content collection, for example, a small Node script can read every entry, filter to what belongs in a curated file, and write llms.txt as part of the build:
import { readFileSync, writeFileSync, readdirSync } from "fs";
import matter from "gray-matter";
const CONTENT_DIR = "src/content/technical-documentation";
const SITE_URL = "https://yanni.uk";
function loadEntries() {
return readdirSync(CONTENT_DIR)
.filter((file) => file.endsWith(".md"))
.map((file) => {
const raw = readFileSync(`${CONTENT_DIR}/${file}`, "utf-8");
const { data } = matter(raw);
return {
title: data.title,
slug: data.slug,
description: data.metaDescription,
featured: Boolean(data.featured),
};
});
}
function buildLlmsTxt(entries) {
const featured = entries.filter((entry) => entry.featured);
const lines = [
"# Yanni.uk",
"",
"> Technical documentation and build logs covering Claude-based automation, deployment pipelines, and SEO tooling.",
"",
"## Documentation",
"",
];
for (const entry of featured) {
lines.push(
`- [${entry.title}](${SITE_URL}/technical-documentation/${entry.slug}/): ${entry.description}`
);
}
return lines.join("\n");
}
const entries = loadEntries();
writeFileSync("public/llms.txt", buildLlmsTxt(entries));
console.log(`llms.txt written with ${entries.filter((e) => e.featured).length} entries.`);
Mark the handful of posts that deserve a place in the curated file with a featured: true field in front matter, run the script as a build step, and the file updates itself every time you publish. This pairs naturally with the same content pipeline covered in astro-cloudflare-pages-guide, where the build step already has access to every content entry before the site deploys.
How Do You Keep llms.txt in Sync with Your Sitemap?
sitemap.xml and llms.txt describe the same underlying site to two different audiences, and when only one gets updated after a content change, they start disagreeing about what actually exists. The most common failure mode is a new section of the site launching, landing in the sitemap automatically because the build process generates it, while llms.txt sits untouched because someone wrote it by hand months earlier.
The fix is treating both as generated artefacts from the same content source, covered for the sitemap side in indexnow-sitemap-automation. If you already run a periodic SEO audit, extend it to diff the URLs in llms.txt against the sitemap's URL list and flag anything present in one but not the other; the audit pattern for this is covered in automated-seo-audits-claude-skills.
Where Do You Host llms.txt, and Does the Platform Matter?
No. Any static host serves a plain text file the same way it serves any other file in the build output; llms.txt just needs to sit in the root of the site so it resolves at /llms.txt. On an Astro project, that means placing the generated file in the public directory. The deployment side works the same as any other static asset, whether the target is deploy-static-site-cloudflare-pages or another static host.
Once deployed, confirm it actually resolves and is served as plain text rather than wrapped in an HTML error page:
curl -sI https://example.com/llms.txt | grep -i "content-type\|http/"
curl -s https://example.com/llms.txt | head -20
The first command checks the response headers; the second confirms the actual content looks like the file you built, not a 404 page or a redirect target.
How Do You Automate llms.txt Regeneration on Every Deploy?
Running the generator script by hand works until it does not, usually the week you are too busy to remember. Wire it into the build pipeline instead, so it runs automatically before every deploy rather than depending on someone remembering a manual step. The general pattern for connecting a build step to a deploy trigger is covered in cicd-pipelines-claude-projects; the llms.txt generator slots in as one more step alongside the sitemap and any other build-time file generation.
For a portfolio with more than one site, it is worth packaging the generator as a small internal tool rather than copying the script into every repository. A lightweight service built with mcp-server-python-fastmcp can expose "regenerate llms.txt for site X" as a callable tool, which keeps the generation logic in one place and makes it easy to trigger from a chat interface or a scheduled job rather than a repository-specific script.
How Do You Manage llms.txt Across a Multi-Site Portfolio?
The same problem that shows up in DNS management across a portfolio shows up here: without a shared approach, each site ends up with a slightly different llms.txt format, maintained inconsistently by whoever last touched that repository. Centralising the generator, one script or package reused across every site's content collection, keeps the format consistent, and it means an improvement to the generator (a better description format, a fix to how featured entries are chosen) benefits every property at once rather than needing to be copied around manually. The same "one source of truth per portfolio" thinking that applies to dns-management-multi-site-portfolios applies just as well to a shared llms.txt generator.
How Do You Verify AI Crawlers Are Actually Reading It?
Check server or edge logs for requests to /llms.txt and look at the user agents making them. Several AI systems identify themselves in the request, which lets you confirm the file is being fetched rather than assuming it is, based on the fact that it deploys correctly.
As a second check, paste the file's contents into a conversation with an AI assistant and ask it to summarise what the site does and list its most important pages. If the summary matches what you intended, the file is doing its job. If it misses the point or fixates on the wrong section, that is a sign the curation needs tightening before the format matters at all.
Frequently Asked Questions
Is llms.txt an official standard?
No, not in the sense of being ratified by a standards body. It is a widely adopted convention that a growing number of AI systems and tools recognise, similar to how robots.txt became a de facto standard long before anyone formalised it. Adopting it costs very little, one small file, and the downside of skipping it is that AI systems have to work harder, with less accurate context, to describe your site.
Do I need llms.txt if I already have a good sitemap.xml?
Yes, because they solve different problems. A sitemap is exhaustive and built for search engine crawlers that index every page. An llms.txt file is curated and built for language models that need a fast, accurate summary rather than a full page list. Keeping both, generated from the same content source, covers both audiences properly.
How often should llms.txt be regenerated?
Every time the site builds and deploys, if it is wired into the build step as described above. There is no reason to regenerate it less often once the generator exists; the marginal cost of running it on every deploy is a fraction of a second, and it guarantees the file never falls out of sync with what is actually published.
Will a bad or outdated llms.txt hurt my search rankings?
It will not directly affect traditional search engine rankings, since llms.txt is not a ranking signal for conventional search. What it affects is how accurately AI systems describe and cite your site when a user asks them a question your content could answer. An outdated or poorly curated file means a model working from it gives an outdated or incomplete answer, which is a visibility problem even if it never touches your position in a search results page.