Nobody fetched my llms.txt

A while back I added two things to this site to make it legible to AI. Per-page Markdown copies of every post, and an llms.txt file at the root — a plain-text index that points at all of them. This is roughly what you're told to do if you want your writing to turn up when someone asks a chatbot a question. I did it, felt vaguely productive, and moved on.

Then it occurred to me that I had no idea whether anything had ever asked for either file. Everyone writing about this recommends it. Almost nobody seems to check their own server afterwards. So I did.

Two files that are not the same bet

Here's the distinction I didn't know about: those two files pay off in opposite directions.

The Markdown twin only matters once something has already found one of my posts — a person pasting the link, a crawler following it. At that point it hands over a clean copy of the text, stripped of the navigation and the Macintosh skin around it. It's a convenience for a visitor that already arrived.

The llms.txt index is the reverse. It does nothing unless a crawler decides, on its own, to go looking for a file at /llms.txt and follow the list inside. The file can be flawless and still sit untouched, because it only pays off if something comes hunting for it first.

So: one serves what already found me, the other has to be found first. Same afternoon of work, wildly different odds — and I'd been filing them under a single tactic.

What Google actually does

So I went digging into which of the two files actually did anything, and the first thing I found was Google contradicting itself.

Its own guidance says, plainly, that Google Search doesn't use llms.txt, and that you don't need one to show up in its AI features. People quote that line and stop there. But look at what Google itself ships: every page of its developer docs has a Markdown twin (add .md.txt to a docs URL and clean text comes back), while there's no llms.txt anywhere on the site. It built the twin and skipped the index.

Which is the same call I'd made by accident, except Google made it on purpose, running the largest crawler on the web. That was the moment the distinction stopped being a hunch.

Worth being fair about the scope, though. Google aimed that statement at Google Search and said so. It isn't claiming the file is useless everywhere, only that its own crawler ignores it. A lot of write-ups flatten that into "llms.txt is dead," which is more than anyone can actually show.

I wanted to know, so I logged it

I put a small logger in front of both files. Every time something fetches /llms.txt or one of the Markdown twins, it writes down what asked: the user-agent, the path, nothing about who the visitor is. It's been running for about eight days.

Here is what came back.

The Markdown twins were fetched thirty times by things claiming to be real AI crawlers. Meta's crawler did most of it, at twenty-four. Amazon's bot managed three, Anthropic's ClaudeBot two, OpenAI's GPTBot one. Around that, plain browser visits ran to roughly two dozen, about the same as Meta, plus one search engine.

The llms.txt index was fetched four times in the same window. Two were reconnaissance scanners, the kind of thing that pokes every site for known filenames. The other two were people in browsers, almost certainly me and whoever I sent the link to. Genuine AI crawlers asking for it: none.

The one other log study I could find, run on someone else's site, reported the same nothing for llms.txt. It didn't run the two files side by side, so it couldn't see the other half of the picture: that the same crawlers skipping the index were instead reading the pages.

The caveats, because they matter

This is eight days of data in one small site. It is not a law of the internet. A bigger site, a longer window or a different crawler mix could easily tell another story.

Also, a user-agent is only ever a claim. Anything at all can stamp GPTBot into a header, and my log believes whatever it's told. The counts aren't tamper-proof either: nothing stops a bored person hammering the file to bend the numbers. Nothing in mine looks staged, the pattern is dull and steady with no suspicious spike, but I can't prove a negative, so I'd rather say it than let the table imply more than it earned.

And "AI crawlers read the twins" is mostly one company. Twenty-four of those thirty fetches are Meta. Take it out and the rest thins to a handful of visits. It's a signal, not a stampede. And it's worth noticing which crawlers never showed up at all: the on-demand ones that fetch a page because someone just asked ChatGPT or Perplexity about it. Everything in my log is a bulk crawler, working the web on its own schedule. Nothing arrived because a person was mid-conversation about my site — which might simply mean nobody is.

I did guard against one failure mode: a logger that quietly breaks looks precisely like "nobody ever asked." So a scheduled job fetches the file once a day on purpose and leaves a heartbeat in the log. The heartbeat shows up every single day. Whatever else my log missed, it wasn't asleep during the days the crawlers stayed away.

What I'd tell myself before I started

If you're trying to make your site more AI friendly, serve the Markdown twin. It's cheap, real agents fetch it, and the cost is a slightly cleaner copy of something you already wrote.

The llms.txt index is a cheap bet on a future that hasn't turned up on my site. I'm leaving mine in place; it does no harm, and the day it's useful might come. But I've stopped treating it as the same move as the twin, and I've stopped trusting a recommendation I never once measured. The measuring took an afternoon. Odd how rarely that's the step anyone writes about.