1,227 requests to llms.txt, zero AI bots: there is no file to place
AI crawlers take thousands of pages for every visitor they return, and the file meant to speak to them is read by none of them. What the published measurements show.
0 of 1,227
requests to llms.txt coming from a major-lab crawler, in a server-log study
In this article
Short answer: you cannot configure your way into an AI answer. Two measurements say it together. On one side, the crawlers of the large models take hundreds to thousands of pages for every visitor they send back. On the other, the llms.txt file the industry started placing at the root of websites is requested by no verified crawler from the major labs, and Google states that it ignores it. There is no side door: what counts happens in the corpus.
3,389 : 1
pages taken by Mistral's crawler per visitor returned (Cloudflare Radar, July 2026)
none
measurable correlation between publishing llms.txt and being cited, across some 300,000 domains
There is no file to place at your root to exist inside an AI answer. Visibility is earned in the corpus; it is not declared.
The exchange is lopsided, and it can be counted
Cloudflare publishes a simple and brutal indicator: the ratio between the number of pages an AI platform comes to read on a site and the number of visitors it sends back. The definition is mechanical. Count the HTML requests coming from a platform's agents, and divide them by the HTML requests whose referring header carries that platform's name.
| Platform | Pages taken per visitor returned |
|---|---|
| Mistral | 3,389 : 1 |
| Anthropic | 2,237 : 1 |
| OpenAI (GPTBot) | 217 : 1 |
| DuckDuckGo | 2.5 : 1 |
The order of magnitude matters more than the decimal, and it moved a great deal in 2026: Anthropic's ratio was counted in the tens of thousands in the first quarter before falling to a few thousand by the summer. The direction is right, the imbalance remains whole. For a website it means machine reading is already the norm, while the human visit that used to pay for it becomes the exception.
One clarification Cloudflare makes itself and that coverage often drops: traffic referred by Claude's native application carries no referring header. Those referrals are therefore invisible in the denominator, and Cloudflare writes that its ratios may overstate the imbalance, without being able to say by how much. The direction of the finding holds; its precision does not.
The file nobody reads
Faced with this imbalance, the industry proposed a simple answer: llms.txt, a file placed at the root of a site to tell the models what to read and how. The idea is appealing. It has one flaw: the crawlers do not request it.
A server-log study run from September 2025 to April 2026 across roughly 900 domains recorded 1,227 requests to those files, and not one from a verified crawler of the major labs - no GPTBot, no ClaudeBot, no PerplexityBot, no Google-Extended. The requesters were a commercial data aggregator for 794 requests, or 64.7%, and human browsers for 392, or 31.9%. In other words: the people reading the file are the ones selling tools around it, and the curious.
The adoption rate, meanwhile, depends entirely on who you look at, which is why you can read one thing and its opposite.
| Sample measured | Adoption |
|---|---|
| Panel of 218 developer-oriented hosts (August 2026) | 51.8% |
| Tranco top 1,000, reachable roots (June 2026) | 15.8% |
| About 300,000 domains (November 2025) | 10.13% |
| Tranco top 1,000, all (June 2026) | 8.7% |
And on the question that settles everything, the correlation study run across roughly 300,000 domains finds no measurable association between publishing this file and being cited by the models. Removing the variable even improved the statistical model's accuracy, which is the polite way of saying it was contributing noise.
What Google says, officially
The position is not ambiguous. Google's documentation, updated in June 2026, states that the file has no effect, positive or negative, on Search rankings or on AI Overviews, and that Search simply ignores it. The Chrome team, for its part, added an llms.txt check to Lighthouse in May 2026, but filed under agentic browsing audits rather than SEO audits, and it flags only a server error, never a missing file.
That filing says a great deal: the file is treated as an object meant for agents browsing the web on behalf of a user, not as a ranking signal. Placing it costs almost nothing and does no harm. You simply should not expect visibility from it.
Why there is no file to place
The two measurements answer each other. The models read the open web massively, your pages included, and they do so without passing through any door you opened. What they retain about you therefore does not come from a declaration made at the root of your domain, but from what is said about you across a corpus you do not control.
This extends exactly what the comparison between engines shows: they do not read the same sources, and yet they agree far more on the names they recommend. A configuration file can do nothing about that, in either direction. What weighs is the consistency of what is written about you, everywhere, and a machine's ability to attribute it.
A crawl ratio is a snapshot, not a constant.
Cloudflare's figures move from month to month and fell by an order of magnitude over 2026. Comparing a July reading with a March one makes no sense, and we did not measure them ourselves.
Cloudflare warns that its own ratios may overstate the imbalance.
Referrals from Claude's native application send no referring header and are therefore missing from the denominator. The company says so and states it does not know the size of the bias: we carry the warning with the figure.
The server-log study is small.
1,227 requests across roughly 900 domains: enough to observe that no major-lab crawler shows up, not enough to claim none ever will, nor to speak for the whole web.
Adoption rates are not comparable with each other.
From 8.7% to 51.8% depending on the sample, for the same object: the difference is the population observed, not a trend over time. An article quoting only one of them, without saying which, tells you nothing.
What this changes
Do not count on a file.
Placing llms.txt is cheap and harmless, but no published measurement links it to a citation. It is not a visibility lever, at best a courtesy towards agents that do not yet exist.
Read your own logs.
One query tells you which crawlers come to your site, how often, and what they ask for. It is the only data in this file you can produce yourself, and it beats a global average.
Decide what you let them take.
The ratio between pages read and visitors returned is a negotiation, not a fate. Allowing, slowing or blocking an agent is a commercial decision that belongs to you, taken crawler by crawler.
Measure citation, not compliance.
Having ticked the technical boxes says nothing about your presence in an answer. The useful question stays: does your name come out, on which engine, and next to whom.