llms.txt: What It Is, What the Data Says, and Whether You Need One
“Should we add llms.txt?” has probably been sitting unresolved on your backlog since a client brought it up or a LinkedIn post made it sound like the next must-have for AI search.
Until recently, there wasn’t much evidence to move that decision one way or the other. Now there is. Several studies, including SE Ranking’s analysis of almost 300,000 domains, have found no measurable link between having an llms.txt file and getting cited more often by AI systems.
But the industry is sending mixed signals. OpenAI, Anthropic, and Gemini publish llms.txt files of their own. Some platforms generate them automatically. Chrome’s Lighthouse now checks for them. At the same time, Google Search says the file isn’t required for visibility in Search or its AI features.
That makes the decision less straightforward than “llms.txt works” or “llms.txt is useless.”
This article takes a more practical approach: we’ll look at what the evidence tells us and break down when adding llms.txt makes sense, when it probably doesn’t, and when it’s simply not worth prioritizing.
llms.txt is a proposed file listing key pages for AI crawlers, and three independent studies plus Google’s own guidance find no measurable citation benefit. This is the extraction target — it must survive being lifted out of the page with zero surrounding context.
1. What is llms.txt, and what problem is it trying to solve?
llms.txt is a proposed Markdown file that lives at a website’s root, typically at example.com/llms.txt. Jeremy Howard of Answer.AI introduced the format in 2024 as a way for website owners to give LLMs and AI agents a curated, machine-readable overview of their most important content.
The idea is not simply to create another file for crawlers. It is meant to solve a specific problem: websites are designed for people, not for language models trying to quickly understand what information matters.
A typical page can contain navigation, scripts, ads, widgets, repeated elements, and other content surrounding the information an AI system actually needs. Large websites create another challenge: an LLM or agent cannot necessarily process every page—or fit an entire site into its context—before deciding what is relevant.
llms.txt is meant to reduce that work. Instead of forcing an AI system to discover a site’s structure and key resources on its own, the website owner can provide a concise description of the site along with links to the pages they consider most useful.
That is also what separates llms.txt from robots.txt. Robots.txt controls crawler access; llms.txt is intended to provide context and point AI systems toward useful content. It does not grant access, block crawling, or tell an AI system what it must cite.
And importantly, llms.txt is still a proposed convention, not an established web standard. Platforms can choose to publish, generate, read, or ignore it.
That gap between what llms.txt is designed to do and what AI systems actually do with it is where things get interesting.
2. Is llms.txt actually used, and by how many sites?
Yes, real websites use llms.txt. But the rates are still relatively low, and the exact percentage changes significantly depending on which part of the web you measure.
In SE Ranking’s study, 10.13% had an llms.txt file. In other words, roughly nine out of ten domains in the dataset did not.
More interestingly, adoption did not increase with traffic:
- Low-traffic websites: 9.88%
- Mid-traffic websites: 10.54%
- High-traffic websites: 8.27%
The largest sites in the dataset were actually slightly less likely to use llms.txt than mid-tier sites.
That matters because it weakens one of the common arguments for implementation: that llms.txt is already becoming an insider best practice among well-established websites.
Other studies land in the same general range, although their samples differ. Trakkr found llms.txt on 13.3% of 37,894 AI-cited domains in March 2026. Casey Burridge’s analysis of HTTP Archive data found a valid file on 5.61% of top-10,000 websites in June 2026.
So, is llms.txt used? Yes. Is it becoming universal? Not yet.
Adoption numbers also need some context. An llms.txt file on a domain does not necessarily mean the site owner made a deliberate decision to optimize for AI. Some CMSs, documentation platforms, and SEO plugins now generate the file automatically.
So even the 5–13% adoption range may overstate how many teams are actively investing in llms.txt. In some cases, the file exists simply because the platform created it.
That makes presence a weak proxy for perceived value. To understand whether llms.txt matters, we need to look beyond who has one and ask whether it changes what AI systems actually retrieve or cite.
3. Does llms.txt actually move your AI citations?
So far, the evidence says no. Across three separate analyses using different methodologies, llms.txt has shown no meaningful predictive signal for how often a domain gets cited in AI answers.
The clearest evidence comes from SE Ranking’s own modeling. The team built an XGBoost model to predict citation frequency across the same 300,000-domain dataset, testing llms.txt presence alongside other factors using Spearman correlation and SHAP analysis. When they removed the llms.txt variable entirely, the model’s accuracy improved. The file wasn’t just failing to help. It was adding noise that made the prediction worse.
SE Ranking’s separate work on optimizing for ChatGPT reached the same result independently: dropping llms.txt from the predictive model improved its accuracy there too.
The obvious counterargument, and one worth addressing directly, is that Google itself maintains llms.txt files on some of its own properties. If Google runs the file internally, doesn’t that mean something?
Not according to Google’s own team. John Mueller has been consistent that these files showed up through an internal CMS update, not as an endorsement of the format. And he’s also gone further on the mechanics: on Google’s Search Off the Record podcast, he argued that a self-reported file can’t help an LLM differentiate between competing sites for discovery, since every site would simply claim to be the best match.
According to him, a ranking signal that any site can claim about itself, without a way to check the claim, stops functioning as a signal once enough sites use it.
4. Why Google says skip it while Chrome checks for it
This is where the story gets confusing.
In May 2026, Chrome’s Lighthouse tool added an llms.txt check to its Agentic Browsing audit category. The audit flags whether a machine-readable summary exists at the domain root, on the reasoning that without one, an AI agent may spend more time crawling a site to understand its structure.
Then, 10 days later, Google Search Central published its official guide to optimizing for generative AI features in Search. Its advice on llms.txt was much more direct: Google Search does not use the file, and creating one will neither help nor hurt a site’s visibility or rankings in Google Search, including its generative AI features.
So within the same company, Chrome is checking whether the file is available while Google Search is telling site owners they do not need it.
The two positions are not necessarily technically incompatible. Chrome is looking at how browser agents interact with and understand websites, while Search Central is talking specifically about retrieval and visibility in Google Search.
And neither position gives us a universal verdict on llms.txt.
For now, that contradiction is probably more useful than trying to resolve it. llms.txt can be irrelevant to search visibility and still have a possible role in agentic browsing. Those are two different questions, and the evidence for each needs to be evaluated separately.
5. Do ChatGPT, Claude, and Gemini treat llms.txt the same way?
Not necessarily, and the honest answer here is that nobody outside those companies can confirm how any of them actually use the file for retrieval.
What we do know: OpenAI currently publishes an llms.txt file for its developer documentation. Anthropic does the same for Claude’s developer platform, and Google’s Gemini also publishes one.
But there’s a big difference between publishing an llms.txt file and using other websites’ files.
OpenAI having an llms.txt file does not mean ChatGPT checks yours before choosing sources. The same goes for Claude and Gemini.
So far, none of the major AI companies has publicly confirmed that third-party llms.txt files are used as a signal for deciding which websites to retrieve or cite.
That doesn’t mean the file is useless. Some agents or documentation tools may read it because it gives them a cleaner map of a site. But that is a very different use case from improving your chances of appearing in an AI answer.
And this is where a lot of llms.txt coverage gets misleading: support for the format is often treated as proof that it improves AI visibility. It isn’t.
For now, the distinction is simple:
llms.txt may help some AI agents navigate a site. There is still no evidence that it helps ChatGPT, Claude, or Gemini cite that site more often.
6. Does your site actually need an llms.txt file?
There is no useful universal yes-or-no answer.
A better rule is to route the decision based on how much work the file creates and who you expect to use it.
If your site already has llms.txt
Leave it as is.
Platforms and plugins including Wix, Mintlify, GitBook, Yoast, and AIOSEO now support or automatically generate llms.txt in some form.
If your file costs you effectively nothing to maintain, deleting it probably accomplishes just as little as adding it.
Instead, check that it is accurate. Make sure it does not contain stale URLs, outdated product information, or pages you would rather not actively point agents toward.
Then move on.
If your site would need a manual llms.txt setup
This is where llms.txt becomes much harder to justify.
According to different research studies, the number of domains using llms.txt varies from 5.61% to 13.3%. None of those datasets suggests that llms.txt is essential for AI visibility.
If creating it means auditing your site, manually curating URLs, building Markdown versions, and maintaining another technical asset every time the website changes, it probably isn’t where your optimization hours should go.
There are higher-impact things to work on first.
If your site targets a specific AI platform
This is the one case where a deliberate implementation may make sense.
If your audience heavily uses a particular AI tool (especially for technical documentation or agent workflows), you may want to test whether giving that system a cleaner map of your content helps.
But treat it as an experiment, not an SEO requirement.
Start by measuring your current AI visibility. SE Visible’s Brand Visibility Dashboard lets you track how often your brand appears across AI engines before deciding whether the implementation is worth maintaining. The platform tracks visibility trends, prompts, topics, competitors, and cited sources across ChatGPT, Gemini, AI Mode, AI Overviews, and Perplexity.
Then add llms.txt and watch what happens.
That gives you a much better rule than “everyone says we should have one”:
If llms.txt is free, keep it. If it costs real time to maintain, skip it. If you have a specific platform and a measurable reason to test it, experiment.
7. How do you test whether llms.txt is doing anything on your site?
If you decide to implement or remove the file, do not judge the result by checking a handful of prompts manually a week later.
Start with a baseline.
In SE Visible, use your existing prompt set and Prompt & Topic Insights to record:
- overall brand visibility;
- the percentage of tracked prompts where your brand appears;
- the specific prompts and topics producing those citations;
- performance by AI engine.
SE Visible tracks these metrics over time and lets you filter results by dates, topics, and AI platforms.
Ideally, collect at least four to six weeks of baseline data. Then add (or remove) llms.txt and monitor the same prompt set for another four to six weeks.
Keep everything else as stable as realistically possible.
If you launch a major content campaign, earn 50 new backlinks, rewrite the pages being cited, change your prompt set, or run a large PR push during the test, you will not know what caused the change.
The same applies to major platform updates. AI results are inherently variable, so look for a sustained shift rather than one strong week.
For a stronger test, keep a control group of prompts or pages that should not be affected by the URLs prioritized in llms.txt. If visibility changes equally across both groups, the file is unlikely to be the reason.
And remember what you are testing.
A crawler requesting /llms.txt proves the file was fetched. It does not prove the system used it to choose your page, understand your brand differently, or generate a citation. The outcome that matters is whether your visibility changes.
FAQ
Is llms.txt a real thing?
Yes. llms.txt is a real proposed convention created by Jeremy Howard and used by thousands of websites and major AI companies. The proposal was first published in September 2024 and was updated to version 2 in August 2026.
But it is still a proposed convention rather than a universally adopted web standard, and there is no confirmed general citation benefit from implementing it.
How do you add an llms.txt file?
Create a Markdown-formatted text file named llms.txt and make it accessible from your website, commonly at: https://example.com/llms.txt.
This file should start with the name of the site or project, followed by a short description and organized lists of links to important resources. A software company, for example, might point agents toward its documentation, API references, tutorials, and key product pages.
The site owner decides what to include, which is part of both the appeal and the limitation of the format: llms.txt can give AI systems a cleaner route through a website, but the information inside it is still curated by the website itself.
You can build the file manually, as well as use a dedicated llms.txt generator or CMS plugin. Many platforms now handle generation automatically, so check whether your website already has one before creating another.
So, should you add llms.txt?
llms.txt is not a decision every site needs to spend time on.
If your CMS already generates the file, keep it, make sure it stays accurate, and move on.
If you would need to build and maintain it manually, the effort is difficult to justify when there is still no proven visibility or citation lift.
And if you are targeting a specific AI platform or agent workflow where llms.txt may actually be used, treat it as a controlled experiment rather than a new SEO requirement.
That is ultimately the useful way to think about llms.txt in 2026.
Not as a box every website needs to tick.
Not as something that must be removed because Google Search ignores it.
But as an optional technical layer whose value depends on who is actually using it.
Before spending time maintaining one, use the Brand Visibility Dashboard in SE Visible to establish what your AI visibility looks like today. Then, if you decide to test llms.txt, you will have something much more useful than speculation: a baseline to compare it against.