
We Logged Every AI Crawler on Our Blog for 104 Days. Search Bots Took Over in October
Between 10 August and 21 November 2026 we logged every AI crawler request that reached WordPress on bloggingtitan.com: 28,850 page fetches from 25 named bots. The headline is simple. OpenAI’s and Perplexity’s search crawlers went from almost nothing to the busiest AI bots on the site in about six weeks. The crawlers that collect training data did not change at all.
The short version
- In our first 30 days, AI bots fetched 4,967 pages. In the latest 30 days (23 October to 21 November) they fetched 12,462, two and a half times as many.
- Almost all of that growth came from two search crawlers. OpenAI’s OAI-SearchBot went from 5 fetches in the first 30 days to 1,837 in the last 30. PerplexityBot went from 10 to 3,388.
- The biggest training crawler, ClaudeBot, did not move: 1,847 fetches in the first 30 days, 1,827 in the last 30.
- OpenAI’s training crawler, GPTBot, fetched 11 pages in 104 days. ChatGPT-User, the agent that fetches a page when a person’s ChatGPT conversation needs it, fetched 4,841.
What changed, and when
OAI-SearchBot first fetched 10 or more pages in a day on 2 October. PerplexityBot first did so on 15 October. Before those dates both were close to silent. By November they were the two busiest AI bots on the site.
Average AI bot page fetches per day, by month
| Bot | Type | Aug* | Sept | Oct | Nov* |
|---|---|---|---|---|---|
| PerplexityBot | Search index | 0.5 | 0.0 | 51.7 | 123.4 |
| OAI-SearchBot | Search index | 0.1 | 0.6 | 39.9 | 64.4 |
| ChatGPT-User | User-triggered fetch | 37.3 | 39.2 | 53.6 | 56.4 |
| ClaudeBot | Training | 61.1 | 65.2 | 69.2 | 56.0 |
| Applebot | Search index | 8.3 | 33.6 | 57.1 | 42.3 |
| Bytespider | Training | 26.6 | 28.0 | 32.6 | 37.9 |
| Claude-User | User-triggered fetch | 1.9 | 5.0 | 2.7 | 7.0 |
| GPTBot | Training | 0.0 | 0.1 | 0.1 | 0.3 |
Three things the log says about what AI systems want from a site
1. They want to index it, not just train on it
The training crawlers were flat for three and a half months. The growth was entirely in crawlers that build search indexes for answer engines. For a small publisher, that is the part that can send a visitor back.
2. Real demand shows up as user-triggered fetches, and it is growing slowly
ChatGPT-User fetches rose 58% between the first and last 30 days (1,083 to 1,715). That is the closest thing in the log to a person asking a question our pages answer. It grew far less than the indexing did.
3. Perplexity reads a lot and fetches live almost never
PerplexityBot fetched 4,205 pages. Perplexity-User, its live-fetch agent, fetched 4 in 104 days. Perplexity appears to answer from its own index, while ChatGPT sends a live fetch for many answers. If you want to know whether Perplexity can see your page, the index crawl is the signal to watch.
What we cannot tell you yet
- Which pages. Our logger keeps daily totals per bot and only the most recent few hundred requests with URLs. We cannot yet say which pages each bot preferred. We plan to add a per-page counter and report it in a follow-up.
- Why October. The site changed a great deal over this period: dozens of new research posts, new pricing and comparison pages, and a new site structure. We cannot separate those changes from changes on OpenAI’s and Perplexity’s side. This is one site, not a sample.
- llms.txt. Our llms.txt is a static file served without WordPress, so fetches of it are invisible to this logger. A zero in our data means “not measurable”, not “never fetched”.
- Bot identity. Bots are matched by user-agent string. We did not verify them against published IP ranges, so some traffic could be spoofed.
- Cached pages. We told our page cache not to serve these bots, so their requests reach WordPress. Googlebot is not in this log.
Method
A small logger on bloggingtitan.com matches about 35 AI bot user-agent strings on every request and stores daily counts per bot (pages and robots.txt) in the WordPress database. It has run continuously since 10 August 2026. Figures in this post were read on 21 November 2026 at 14:15 UK time. “First 30 days” is 10 August to 8 September; “last 30 days” is 23 October to 21 November.
This is the third report from the same log. The first covered our first 37 days; the second split the data at Cloudflare’s 15 September crawler change.
Cite this data
Blogging Titan, “AI crawler log, 10 Aug to 21 Nov 2026” (28,850 AI bot page fetches, 25 bots, one site), bloggingtitan.com, 21 November 2026. Free to cite under CC BY 4.0 with a link to this page.
Key figures: OAI-SearchBot 5 to 1,837 fetches and PerplexityBot 10 to 3,388 (first vs last 30 days); ClaudeBot 1,847 to 1,827; GPTBot 11 in 104 days; ChatGPT-User 4,841; Perplexity-User 4.