We Logged Every AI Crawler on a Small Blog for 37 Days

The numbers, up front

• 6,399 AI-bot page requests over 37 days, from 16 distinct crawlers.
• ClaudeBot made 2,415 of them, about 38 percent of all AI-bot traffic. ChatGPT-User made 1,387. Bytespider, ByteDance’s crawler, made 984. Those three bots were 74 percent of everything.
• GPTBot made 1 request in 37 days. ClaudeBot made 2,415 times as many. The single most-blocked AI user agent on the web barely visited.
• The crawlers that fetch a page in direct response to a live user question, the ones that can actually put you in front of a reader, were 1,572 of the 6,399, about a quarter. ChatGPT-User alone was 1,387 of that 1,572, or 88 percent.
• PerplexityBot made 10 requests in 37 days. The fear that Perplexity is quietly scraping small blogs did not show up in our log at all.
• robots.txt was fetched 1,381 times. OAI-SearchBot, OpenAI’s search indexer, checked robots.txt 221 times and fetched 7 actual pages. It reads the rules far more than it reads the site.
• We are not reporting an llms.txt number. On this site llms.txt is a static file served without touching WordPress, so our logger cannot see requests to it. A complete llms.txt was in place the whole time. We would rather report nothing than a figure we did not measure.

What we actually did

On 10 August 2026 we added a small logger to this site. It runs on every request, checks the user agent against about 35 known AI-crawler signatures, and records three things per bot per day: how many pages it fetched, how many times it read robots.txt, and how many times it requested llms.txt. Nothing about the reader is stored. The bot names it watches for include GPTBot, OAI-SearchBot and ChatGPT-User from OpenAI, ClaudeBot and Claude-User from Anthropic, PerplexityBot and Perplexity-User, Google-Extended and GoogleOther, Bytespider, Amazonbot, Meta-ExternalAgent, Applebot, CCBot and DuckAssistBot, among others.

One technical note that matters for anyone trying to repeat this. This site sits behind a page cache, and a cached page is served without the request ever reaching the logging code. So we told the cache to let every one of those bot user agents bypass the cache and hit the application directly. Without that step most bot visits are invisible to the site itself, which is one reason so few bloggers have any idea what is crawling them.

The honest limitation is llms.txt. The file exists and is comprehensive, but it is served as a flat file that never runs through WordPress, so we cannot count requests to it. Every llms.txt figure in this post is therefore blank on purpose. We describe below what that does and does not let us conclude.

The bot everyone tells you to block is the one that never came

Search “how to block AI crawlers” and almost every result opens with the same instruction: add GPTBot to your robots.txt. It is the user agent that launched the whole block-the-bots conversation in 2023, and it has become shorthand for keeping ChatGPT off your site.

In 37 days, GPTBot requested one page from us.

This is not because we blocked it. Our robots.txt let it through the entire time. GPTBot is OpenAI’s training crawler, the one that gathers text to train future models, and on a small blog it simply does not come around often. Meanwhile ClaudeBot, which does the equivalent job for Anthropic, fetched 2,415 pages. If your mental model is that GPTBot is the AI crawler hammering your server, the log says the opposite. The bot doing the most crawling by a wide margin belongs to Anthropic, and the OpenAI crawler people actually block is close to absent.

Here is the part that turns this from a curiosity into a mistake worth avoiding. When a person is inside ChatGPT and asks it about something on your site, the fetch that pulls your page comes from a different user agent entirely, called ChatGPT-User. It ignores a robots.txt rule written for GPTBot, because a rule naming one bot does not bind another. ChatGPT-User fetched 1,387 pages from us. So the standard advice, block GPTBot to keep ChatGPT out, blocks the crawler that made one request and leaves untouched the crawler that made 1,387. That is the GPTBot decoy. You feel like you have closed the door on ChatGPT, and the door ChatGPT actually uses is the one you left open.

Training crawlers and retrieval crawlers are not the same animal

The single most useful way to read this log is to split the bots into two groups, because they do different jobs and blocking them has completely different consequences.

Training and indexing crawlers walk your site in bulk to build a dataset or a search index for later. They do not send you a reader today. ClaudeBot, Bytespider, CCBot, Applebot and PetalBot are in this group. Retrieval crawlers fetch a specific page in the moment, because a user just asked a question the page might answer. These are the visits that can end with your link in an answer. ChatGPT-User, Claude-User, Perplexity-User, PerplexityBot, OAI-SearchBot and DuckAssistBot are in this group.

Read down that table and the block-the-bots advice starts to look backwards. The heavy crawlers are all training bots that do not send you anyone, so blocking them protects nothing you can measure and costs you presence in the next generation of models. The crawlers that can actually deliver a reader are a quarter of the traffic, and almost all of that quarter is one bot, ChatGPT-User, which the popular robots.txt recipe does not even name.

Perplexity barely crawled us

Perplexity has a reputation, in some corners, as an aggressive scraper. On our log it was one of the quietest names on the list. PerplexityBot made 10 page requests in 37 days, and Perplexity-User, the live-question fetch, did not register enough to rank. That fits how Perplexity works: it leans on a search index and fetches at the moment of a query rather than crawling everything in advance. For a blog our size, the practical amount of Perplexity crawling to worry about was almost none.

We hold this one at the scale we measured it. A larger or more newsworthy site may see far more Perplexity activity. What we can say is that on one small blog over five weeks, the crawler people name as the scraping threat was outnumbered 241 to 1 by ClaudeBot.

So should you block AI crawlers?

On this evidence, for a small blog, blocking is the weaker move, and it is worth being precise about why.

Blocking the training crawlers, which are most of the volume, removes you from the datasets that train the models people will ask questions of next year. You get no measurable benefit in return, because a training crawl was never sending you a visitor. Blocking the retrieval crawlers is worse, because those are the fetches that can end with your link inside an answer, which on our own earlier research is one of the few surfaces where a small site can appear next to a big one. We argued the visibility side of this case in full when Google shipped its AI Overviews opt-out toggle, in why opting out of AI Overviews is a visibility strike small blogs cannot win. This log is the crawler-level version of the same conclusion: the door you are tempted to close is the one worth keeping open.

There is a real cost question underneath the visibility one, and we are not waving it away. Training crawlers use your bandwidth and give you nothing back, and 2,415 ClaudeBot requests is not nothing on a small host. If your server is straining, rate-limiting the heavy training bots is defensible. But that is a hosting decision about load, not a visibility strategy, and it is a different action from the blanket “block AI” advice that circulates. Blocking for load and blocking to “protect your content” point at different bots.

If you still want to block, block the right names

Say you have decided to keep training crawlers out while staying eligible for citation. The robots.txt that matches what is actually crawling looks nothing like the one-line GPTBot rule. You would name the training bots you saw, and deliberately leave the retrieval user agents alone:

User-agent: ClaudeBot
User-agent: Bytespider
User-agent: CCBot
User-agent: GPTBot
User-agent: Google-Extended
Disallow: /

# Left deliberately allowed, these bring readers:
# ChatGPT-User, Claude-User, PerplexityBot, OAI-SearchBot

Two honest caveats on that block. User agents are self-reported and can be spoofed, so a robots.txt rule is a request, not a wall, and the badly behaved crawlers are exactly the ones that ignore it. And blocking Google-Extended removes you from Gemini training while keeping normal Google Search indexing, which is a real and separate lever most “block AI” posts blur together. The point is that the decision deserves the names in front of you, not a copied line aimed at the bot that made one request.

What this log cannot tell you

This is one small blog in the blogging and SEO niche, over 37 days. A news site, a store or a bigger domain will see a different mix, and probably far more retrieval traffic. Treat the shape of the finding as portable and the exact counts as ours.

We logged crawler requests, not readers. A ChatGPT-User fetch means a page was pulled into an answer’s working set. It does not prove your link was shown, clicked, or credited. Measuring that end of the pipe is a separate study, and one we care about, because our tracing of viral AI-traffic conversion claims found the reader numbers are routinely inflated.

The llms.txt gap is the one we most wish we could close. Because the file is served statically, we cannot say whether a single AI bot ever requested it during these 37 days. Anyone selling you an llms.txt file should be asked for exactly that evidence, and it is the question we are turning into our next test.

Finally, user agents can lie. We report what the crawlers called themselves. A determined scraper can wear any name it likes, which cuts both ways: some of the ClaudeBot volume could be impostors, and some traffic wearing an innocent browser string could be a bot we never counted.

Why we can publish this and most sites cannot

The reason this data is rare is not that it is hard to gather. It is that a cached page never reaches the code that could count the visit, so the average WordPress blog has no record of any of this. We had to deliberately route every AI user agent around the cache to see it at all. The wider point runs through most of our research: the AI-search advice with the most confidence behind it, block this bot, add that file, is usually written by people who have never watched their own logs. We watched ours, and the first thing it did was contradict the most repeated instruction in the genre.

@misc{bloggingtitan_ai_crawler_log_2026,
  title  = {AI Crawler Log: 37 Days of Bot Traffic to a Small Blog},
  author = {{Blogging Titan}},
  year   = {2026},
  url    = {https://bloggingtitan.com/blog-seo/ai-crawler-log-small-blog/},
  note   = {Original first-party dataset, CC BY 4.0}
}

Related guides

• Don’t opt out of AI Overviews: the visibility strike small blogs can’t win
• The AI citation league table: what ChatGPT and Perplexity actually cite
• Does schema markup help you get cited by AI? We tested it
• We traced the viral AI-traffic conversion claims to their source

Written by

Blogging Titan Team

Blogging Titan is an independent team of bloggers documenting what actually grows a blog in the AI search era. We have been building, ranking, and monetizing WordPress sites since 2017, and every guide on this site is based on strategies and tools we have tested ourselves. Want a second pair of eyes on your blog? Request a free blog audit or start with the 2026 playbook.

Summarize with: ChatGPT Perplexity Claude Grok