Cloudflare let ChatGPT back in on 15 September. On our open blog, its fetches did not move
Quick answer: On 15 September Cloudflare changed its AI-crawler controls. Within two days, SeenSure measured refusals of ChatGPT’s search crawler falling from 24.9 percent to 3.2 percent on the Cloudflare sites in its main sample, while sites outside Cloudflare did not move. We checked what that did to a blog that had never shut ChatGPT out. The AI retrieval crawlers, the group that fetches pages to answer questions, averaged 43.0 page requests a day before 15 September and 43.3 after. ChatGPT-User alone went from 37.8 a day to 40.5, a change well inside normal day-to-day noise. We call this the unmoved fetch: the change decided which sites ChatGPT could fetch, and on our log it did not change how much ChatGPT fetched from us.
Original study. 9,187 AI-bot page requests logged on bloggingtitan.com across 50 full days, 11 August to 29 September 2026, split at 15 September. This site does not sit behind Cloudflare, which is what makes it useful here.
The numbers, up front
- Retrieval crawlers (ChatGPT-User, Claude-User, PerplexityBot, OAI-SearchBot and DuckAssistBot) made 43.0 page requests a day in the 35 days before 15 September and 43.3 a day in the 15 days from it.
- ChatGPT-User averaged 37.8 a day before and 40.5 after, up 7 percent. A Mann-Whitney test puts that at p = 0.41, so the honest reading is no change.
- ClaudeBot went from 67.9 a day to 58.3 (p = 0.64) and Bytespider from 27.2 to 29.6 (p = 0.34). The two heaviest training crawlers held steady.
- GPTBot made 2 page requests in the 15 days after the change, against 875 from ClaudeBot. In the 35 days before it made none; its only other request in the whole log came on the partial first day, 10 August.
- Applebot is the one real mover: 11.2 page requests a day before, 50.6 after, about 4.5 times as many. This site is not on Cloudflare, so we cannot hang it on the change. Details below.
- All AI bots together rose from 175.7 page requests a day to 202.4, up 15 percent. Take Applebot out and the total fell 8 percent, from 164.5 to 151.8.
What changed on 15 September
Cloudflare had said in July that from 15 September, any site blocking AI training would also block Googlebot, Bingbot and Applebot, as Semrush reported. What it shipped on the day was different. Its 15 September post introduced a setting called Disallow AI Training. It publishes a no-training preference in robots.txt, blocks every other training crawler, and leaves what Cloudflare calls accountable mixed-use crawlers (Applebot, Bingbot and Googlebot) free to keep crawling for search. The same post says Managed robots.txt will be deprecated in favour of a new Bot Preference Sync.
The measured effect came a day later. SeenSure fetched 1,046 sites on 14 and 16 September as eight AI crawlers and an ordinary browser, and counted any error status as a refusal. Across all 1,046, refusals fell for the search and agent crawlers: OAI-SearchBot from 16.9 percent to 3.0, ChatGPT-User from 16.4 to 2.8. They rose for the two training crawlers, GPTBot from 18.9 percent to 22.0 and ClaudeBot from 19.9 to 22.8.
The split by host is sharper. For the 894 sites SeenSure drew from certificate logs, it separates the 639 behind Cloudflare from the 255 that are not. Behind Cloudflare, OAI-SearchBot refusals fell from 24.9 percent to 3.2 and ChatGPT-User from 24.7 to 3.2, while GPTBot rose from 26.8 to 31.0. Outside Cloudflare, ChatGPT-User sat at 2.1 percent before and after. One caution if you are quoting this study. Its summary text places the 16.9 and 3.0 figures on the Cloudflare side, but its own table gives them for all 1,046 sites, so we use the table. SeenSure also says it cannot tell whether a refusal came from Cloudflare or from the site behind it, and its sample is mostly young domains, not a random slice of the web.
So the short version, that sites reopened to AI bots, is half right. Cloudflare-fronted sites stopped refusing the bots that answer questions. They got stricter with the bots that train models.
Why a site that never closed is the right place to look
The studies published so far test the door: send a request as each crawler and record whether the site answers with a page or a refusal. That tells you how many doors opened. They cannot tell you whether anything came through them, or whether the sites that were already open lost anything once their neighbours reopened.
Our log can say something about the second question. Blogging Titan runs on LiteSpeed at Hostinger with no Cloudflare in front of it. Our robots.txt names no AI crawler and blocks none, and our access rules did not change on 15 September. We have logged every AI crawler visit since 10 August, which gave us five weeks of baseline before the change and fifteen days after it. That makes the site a rough control for the question of access. It is not a perfect one, because the content kept changing: we published 12 posts in the 15 days before the change and 4 in the 15 days after.
If ChatGPT fetches a roughly fixed number of pages per question, and thousands of sites suddenly joined the pool it can fetch from, an open site might expect to see its share thinned. If reopening instead pulled ChatGPT into fetching more, an open site might see more too. On our log, neither happened.
The unmoved fetch
ChatGPT-User is the user agent that pulls a page when someone inside ChatGPT asks something that needs a live source. In our first 37-day log it made 88 percent of all retrieval crawling on this site, which makes it the number to watch.
Across the 35 full days before 15 September it averaged 37.8 page requests a day, with a median of 37. Across the 15 days from 15 September it averaged 40.5, with a median of 39. Against only the final fortnight before the change, 36.9 a day, the result is the same (p = 0.28). The daily line in the chart above crosses 15 September without a step.
Widen it to every retrieval crawler and the match is almost exact: 43.0 a day before, 43.3 after. Neither difference is close to significant (p = 0.88 for the group). That is the unmoved fetch. The Cloudflare change decided which sites ChatGPT could reach. On our log, it made no difference to how much ChatGPT fetched from a site it could already reach. Our reading is that fetch volume follows the questions people ask more than the number of open doors, but one site cannot prove that.
Two limits on that, stated plainly. This is one small blog and fifteen days. And a fetch is not a citation: ChatGPT can pull our page and then quote a reopened competitor. Fetch counts cannot see that, which is why we measure citations separately, in the AI citation league table.
The training crawlers did not notice either
SeenSure found Cloudflare sites tightening on training crawlers after the change. If that had pushed training crawls elsewhere, a site that allows them should have seen more. We saw the opposite drift. ClaudeBot fell from 67.9 page requests a day to 58.3, though ClaudeBot is spiky (it made 263 requests on 11 September alone) and the difference is nowhere near significant. Bytespider was flat. CCBot, which crawled on only five days of the fifty, went from 3.2 to 4.0.
GPTBot is still the strangest line in the log. Across 50 full days, on a site that has never blocked it, it requested 2 pages, both after 15 September, plus one on the partial first day of logging. In the same 15 days ClaudeBot requested 875. We named this pattern the GPTBot decoy in the first log: the crawler everyone blocks is the one that barely visits. Two more requests do not change that.
The one bot that did move: Applebot
Applebot went from 11.2 page requests a day to 50.6. It peaked at 90 on 20 September. By a before-and-after test that is a very large change (p = 0.00000015), and it is the entire reason total AI-bot traffic rose. It also survives a tougher test, described below.
It is tempting to connect it to the Cloudflare news, because Applebot is one of the three crawlers Cloudflare’s new setting explicitly spares. We are not going to. This site is not on Cloudflare, so the setting cannot have changed anything here directly. Applebot was also rising before the change: its weekly average went 7.3, 8.4, 9.9, 10.9, then 19.7 in the week of 8 September, and only then 54.1 and 45.9. We modelled the whole series as a steady upward trend plus a step on 15 September. Even after allowing for the trend, the step comes out at about 2.1 times, with a 95 percent interval of 1.3 to 3.5 (p = 0.004). The model fits clearly better with the step on 17 September, two days after the change, where it comes out at about 2.5 times. That points away from Cloudflare’s date.
So something lifted Applebot’s crawling of this site in mid-September, and we do not know what. Apple could have reallocated crawling across the web as sites reopened. Or it could be something on our side: four new posts went live in the after window, and Applebot fetched one of them twice in the raw sample below. Neither explanation is tested here.
What the log can add is a glimpse of what Applebot was fetching. The logger keeps only a short rolling window of individual hits. On 30 September it held 8 from Applebot: 3 RSS feeds, 2 oEmbed endpoints, 2 requests for one post page and 1 robots.txt read. That post went live on 20 September, the day Applebot peaked. Treat that as a pointer, nothing more. We will know more when the log has run to December, which is also when SeenSure plans to re-measure.
Three smaller moves, and why we are not claiming them
OAI-SearchBot, OpenAI’s search indexer, fetched 7 pages in the 35 days before and 13 in the 15 days after. Its robots.txt reads rose from 6.0 a day to 8.3. Against the full baseline the page change tests at p = 0.02, against the final fortnight at p = 0.13, and the counts are tiny. It may be the first sign of OpenAI’s search index re-crawling after Cloudflare sites reopened to it. It may be nothing.
Claude-User, the fetch behind a live question in Claude, went from 4.6 page requests a day to 1.8. Against the final fortnight before the change, 7.9 a day, that tests at p = 0.002. It may be real. We are not claiming it. Against the full 35-day baseline it is not significant (p = 0.26), the daily counts are small, and we ran more than two dozen tests in this study without correcting for multiple comparisons. PerplexityBot, which made 10 requests before the change, made none after.
YouBot collapsed from 5.9 page requests a day to 0.2, but it had already dropped to 1.7 a day in the two weeks before 15 September. Whatever happened to You.com’s crawler started in late August and has nothing to do with this date.
What this means for your blog
If your site is on Cloudflare and ChatGPT’s crawlers can reach it again, you are back in the pool ChatGPT fetches from. That is worth having, and it is the right default for a small blog, for the reasons we set out in the visibility strike. Our log gives no sign that the change made ChatGPT fetch more overall. If that holds beyond one site, reopened sites are now competing for the same fetches open sites were already getting. Reopening puts you back on the start line. It does not move you up it.
If your site was never closed, our log suggests the change cost you little in fetches. It cost us nothing we can measure. The competition will show up, if it shows up, in citations: which of the fetched pages ChatGPT actually quotes. That turns on the things a reopened rival can copy slowly or not at all, such as original data, a named author and a visible date. Our authorship study found that one cited page in five names nobody at all.
Whichever side you are on, check your own logs. Cloudflare’s dashboard reports what its edge refused, and your server log reports what got through. Neither on its own tells you the whole story, and it takes both to know whether the change reached you.
How we measured it
A small logging script on this site checks each request’s user agent against about 35 known AI-crawler signatures and records, per bot per day, page requests and robots.txt requests. It stores nothing about human readers. The page cache is set to let those bot user agents bypass it, so bot requests for pages reach the logger. The method is unchanged from the first log, and the before period here overlaps that study.
The before window is 11 August to 14 September 2026, 35 full days. We dropped 10 August because logging started partway through it. The after window is 15 to 29 September, 15 full days; 30 September was still running when we read the log, so it is out too.
We hold daily figures for 12 of the 16 bots. Each of those, and the retrieval group, was compared with a two-sided Mann-Whitney test, once against the full before window and once against its final 15 days. Applebot also got a negative binomial regression with an estimated dispersion, a linear trend and a 15 September step. Retrieval crawlers are grouped as in the first log: ChatGPT-User, Claude-User, PerplexityBot, OAI-SearchBot and DuckAssistBot. That log also listed Perplexity-User, which made no requests in either window here.
What we cannot see. llms.txt on this site is a static file served without touching WordPress, so requests to it are invisible to the logger. We report no llms.txt figure. A crawler that fakes a browser user agent is also invisible, as is anything refused before it reaches the site, though nothing here refuses AI crawlers. And one site cannot stand for the web. It can only show what happened on one site whose access rules did not change.
Daily page requests for the seven bots charted, 10 August to 30 September 2026
| Date | Applebot | ChatGPT-User | ClaudeBot | Bytespider | OAI-SearchBot | YouBot | Claude-User |
|---|---|---|---|---|---|---|---|
| 10 Aug | 3 | 20 | 11 | 8 | 0 | 2 | 0 |
| 11 Aug | 10 | 46 | 41 | 30 | 0 | 14 | 7 |
| 12 Aug | 8 | 54 | 116 | 16 | 0 | 8 | 2 |
| 13 Aug | 8 | 50 | 30 | 20 | 1 | 9 | 1 |
| 14 Aug | 10 | 49 | 30 | 27 | 0 | 5 | 5 |
| 15 Aug | 5 | 40 | 28 | 29 | 0 | 10 | 0 |
| 16 Aug | 8 | 40 | 61 | 34 | 0 | 11 | 0 |
| 17 Aug | 2 | 41 | 64 | 33 | 0 | 8 | 0 |
| 18 Aug | 5 | 51 | 35 | 52 | 2 | 11 | 2 |
| 19 Aug | 4 | 51 | 101 | 33 | 0 | 12 | 4 |
| 20 Aug | 10 | 44 | 52 | 36 | 0 | 5 | 1 |
| 21 Aug | 7 | 27 | 238 | 20 | 0 | 12 | 3 |
| 22 Aug | 13 | 27 | 26 | 23 | 0 | 11 | 0 |
| 23 Aug | 17 | 27 | 33 | 14 | 0 | 12 | 2 |
| 24 Aug | 3 | 28 | 44 | 22 | 0 | 7 | 0 |
| 25 Aug | 2 | 41 | 101 | 22 | 0 | 7 | 0 |
| 26 Aug | 8 | 42 | 41 | 27 | 0 | 13 | 1 |
| 27 Aug | 14 | 37 | 140 | 17 | 0 | 9 | 2 |
| 28 Aug | 18 | 28 | 42 | 16 | 0 | 13 | 1 |
| 29 Aug | 19 | 29 | 30 | 33 | 0 | 2 | 11 |
| 30 Aug | 8 | 18 | 47 | 34 | 0 | 1 | 0 |
| 31 Aug | 0 | 30 | 34 | 39 | 0 | 3 | 0 |
| 1 Sep | 1 | 39 | 47 | 29 | 0 | 0 | 4 |
| 2 Sep | 5 | 32 | 100 | 35 | 0 | 2 | 30 |
| 3 Sep | 9 | 34 | 75 | 27 | 0 | 1 | 1 |
| 4 Sep | 7 | 31 | 25 | 31 | 2 | 0 | 12 |
| 5 Sep | 23 | 22 | 43 | 28 | 0 | 1 | 10 |
| 6 Sep | 20 | 35 | 145 | 31 | 0 | 1 | 4 |
| 7 Sep | 11 | 28 | 27 | 19 | 0 | 1 | 3 |
| 8 Sep | 13 | 42 | 40 | 25 | 0 | 4 | 27 |
| 9 Sep | 12 | 35 | 77 | 39 | 0 | 0 | 3 |
| 10 Sep | 15 | 46 | 35 | 28 | 2 | 0 | 8 |
| 11 Sep | 32 | 63 | 263 | 16 | 0 | 0 | 4 |
| 12 Sep | 20 | 37 | 42 | 21 | 0 | 6 | 2 |
| 13 Sep | 14 | 31 | 48 | 19 | 0 | 3 | 6 |
| 14 Sep | 32 | 49 | 74 | 27 | 0 | 3 | 5 |
| 15 Sep | 19 | 43 | 29 | 24 | 0 | 0 | 3 |
| 16 Sep | 15 | 38 | 34 | 27 | 5 | 0 | 2 |
| 17 Sep | 40 | 38 | 97 | 34 | 2 | 0 | 1 |
| 18 Sep | 66 | 54 | 36 | 30 | 1 | 0 | 0 |
| 19 Sep | 83 | 28 | 36 | 33 | 0 | 0 | 0 |
| 20 Sep | 90 | 31 | 62 | 44 | 0 | 0 | 1 |
| 21 Sep | 66 | 49 | 62 | 43 | 0 | 2 | 3 |
| 22 Sep | 50 | 44 | 98 | 32 | 1 | 1 | 2 |
| 23 Sep | 36 | 34 | 47 | 40 | 0 | 0 | 1 |
| 24 Sep | 44 | 47 | 58 | 33 | 2 | 0 | 1 |
| 25 Sep | 53 | 60 | 62 | 31 | 2 | 0 | 2 |
| 26 Sep | 45 | 24 | 87 | 25 | 0 | 0 | 3 |
| 27 Sep | 52 | 39 | 41 | 12 | 0 | 0 | 3 |
| 28 Sep | 41 | 48 | 51 | 18 | 0 | 0 | 3 |
| 29 Sep | 59 | 31 | 75 | 18 | 0 | 0 | 2 |
| 30 Sep | 34 | 38 | 21 | 17 | 0 | 0 | 2 |
10 August and 30 September are partial days and are excluded from every average above.
Frequently asked questions
Short answers, with the same figures as above.
What did Cloudflare change on 15 September 2026?
It introduced a Disallow AI Training setting that blocks AI training crawlers while leaving Applebot, Bingbot and Googlebot free to crawl for search. It also announced that Managed robots.txt will be deprecated in favour of Bot Preference Sync. Before that, Cloudflare had said blocking AI training would block those three search crawlers too.
Did Cloudflare sites reopen to ChatGPT?
To its search and agent crawlers, largely yes. On the 639 Cloudflare sites in SeenSure’s certificate-log sample, measured on 14 and 16 September, OAI-SearchBot refusals fell from 24.9 percent to 3.2 and ChatGPT-User refusals from 24.7 percent to 3.2. GPTBot refusals rose from 26.8 percent to 31.0. Across all 1,046 sites SeenSure tested, OAI-SearchBot refusals fell from 16.9 percent to 3.0.
Did ChatGPT fetch more from sites that were already open?
Not on ours. ChatGPT-User averaged 37.8 page requests a day on bloggingtitan.com in the 35 days before 15 September and 40.5 a day in the 15 days from it. The difference is not statistically significant (p = 0.41). All retrieval crawlers together went from 43.0 a day to 43.3 (p = 0.88).
Why did Applebot crawling go up?
We do not know. On bloggingtitan.com Applebot rose from 11.2 page requests a day to 50.6 after 15 September. Part of that is a trend that was already under way, but a step of about 2.1 times survives a model that allows for the trend (p = 0.004). This site is not on Cloudflare, so Cloudflare’s setting cannot have caused it directly.
Cite this data
This is original first-party research by Blogging Titan. The figures below are free to cite or republish with attribution under a CC BY 4.0 license. All figures are page requests to bloggingtitan.com, which is not behind Cloudflare.
| Windows compared | 11 Aug to 14 Sep vs 15 to 29 Sep 2026 |
| AI-bot page requests, both windows | 9,187 |
| Retrieval crawlers, per day | 43.0 before, 43.3 after |
| ChatGPT-User, per day | 37.8 before, 40.5 after (p = 0.41) |
| ClaudeBot, per day | 67.9 before, 58.3 after (p = 0.64) |
| GPTBot, page requests | 0 before, 2 after |
| Applebot, per day | 11.2 before, 50.6 after |
| All AI bots excluding Applebot, per day | 164.5 before, 151.8 after |
Blogging Titan. (2026). The Unmoved Fetch: AI Crawler Traffic to an Open Blog Before and After Cloudflare’s 15 September 2026 Change. Retrieved from https://bloggingtitan.com/blog-seo/cloudflare-ai-crawler-change-log/
@misc{bloggingtitan_unmoved_fetch_2026,
title = {The Unmoved Fetch: AI Crawler Traffic to an Open Blog Before and After Cloudflare's 15 September 2026 Change},
author = {{Blogging Titan}},
year = {2026},
url = {https://bloggingtitan.com/blog-seo/cloudflare-ai-crawler-change-log/},
note = {Original first-party dataset, CC BY 4.0}
}
Find out whether AI search can see your site
Our free AI Citation Grader tests whether ChatGPT and Perplexity retrieve and cite your pages.
Want a second pair of eyes on your crawlability? A free blog audit covers it, robots rules included.
- We logged every AI crawler on a small blog for 37 days
- Don’t opt out of AI Overviews: the visibility strike small blogs can’t win
- The AI citation league table: what ChatGPT and Perplexity actually cite
- What is a robots.txt file?
Written by
Blogging Titan Team
Blogging Titan is an independent team of bloggers documenting what actually grows a blog in the AI search era. We have been building, ranking, and monetizing WordPress sites since 2017, and every guide on this site is based on strategies and tools we have tested ourselves. Want a second pair of eyes on your blog? Request a free blog audit or start with the 2026 playbook.