This looks interesting from a Cloudflare article - I like the idea of telling an AI crawler they need to pay if they want access. Question is if you set it to a 402 response, how would they actually pay you?
Crazy amount of guests
- Thread starter Levina
- Start date
-
- Tags
- bot swarms guests
Found it - could be worth setting up yes? Except I can't see the option to do it - maybe only for paid Cloudflare accounts ..........This looks interesting from a Cloudflare article - I like the idea of telling an AI crawler they need to pay if they want access. Question is if you set it to a 402 response, how would they actually pay you?
View attachment 345225
Pay per crawl is private/beta but you can sign up for early access here
www.cloudflare.com
Get early access: Cloudflare Pay Per Crawl Private Beta
Empowering content creators and AI innovators alike
Of course it will pay peanuts...
Onto the intermittent error - if it cannot be consistently reproduced, it might just be "transitory issue". You can check at the two ends of the connection between CF and your hosting, but you have no control over how traffic flows through the various peers of the Internet. There are lots of dynamic filters and routes changing on the fly which can influence a connection, and it's really hard to diagnose and virtually impossible to control.
Onto the intermittent error - if it cannot be consistently reproduced, it might just be "transitory issue". You can check at the two ends of the connection between CF and your hosting, but you have no control over how traffic flows through the various peers of the Internet. There are lots of dynamic filters and routes changing on the fly which can influence a connection, and it's really hard to diagnose and virtually impossible to control.
For one user, it's infrequently. I've had it maybe once in several months. The site owner had problems all weekend, where nobody else did, and it was nearly constant I guess.
I have reassured everyone it is not our filtering.
One thing I'm taking a look at is the Apache settings:
KeepAlive (should be On)KeepAliveTimeout (default Apache setup is only 5 seconds, but a more sane value for CF is 60-120 seconds)MaxKeepAliveConnections (default is 100; could raise that for more sanity)If the timeout is too short and there is a connection issue, CF gives up. CF does retry several times, one second at a time, and retries few more at longer intervals.
So that's where it is sitting right now.
Not sure if this is coincidence or not, but since I blocked ClaudeBot today, my guest list numbers shot right up again to over 1000 - virtually every country in the world again - apart from the 17 blocked ones. Yet strangely my solve rates in Cloudflare have improved.
Sure about that? Hitting a URL and trying to register, are two very different things. Crawlers hit every URL there is a link to.
Good point. Wouldn't put it past bots to try and register though.
You can register on a site with automation, but it normally takes someone to program the system to do it, otherwise it gets caught in honeypots and such. Captcha and other things help stop it, but those softwares do get around most captchas, not all, they even confirm email and such simple methods. You can buy them yourself if you want to spam. The easier route though, most effective route, is a human creates an account, x accounts, gets past any moderation or such, then hands the login details to the software to automate posting. That is often limited to those who want to put in the time to get links for their clients.
If you can catch the error, open up your browser console, click the page in the error state under the network tab, then check the 'headers' tab. Look for CF-Ray, and any other CF-* prefixed items. The ray should denote what origin source is at that threw the error.
While not cloudflare, AWS's Cloudfront in Kansas City (Tagged as MCI) has been giving me constant fits with https://www.spc.noaa.gov/ when the SPC auto-pushes out updates to their site. Site hangs, nothing rendering, missing content on render, etc. Using my VPN server in Chicago, which routes to the Chicago AWS S3/CF origin - no issues.
So it's very possible there's a regional issue between their anycast routes to the many PoPs.
Oh boy... what might we have starting up now... oh wait, it's just TenCent's, huawei's, ChinaNet's and who knows who elses AI botnet getting all feisty again! At least they're querying robots.txt now... i guess? They don't seem to respect it though. 
Code:
[2026-09-21T22:34:53-05:00] "180.153.197.142" "-" "GET /robots.txt HTTP/1.1" "301" 162 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173711" "1" "222"
[2026-09-21T22:34:53-05:00] "82.157.163.183" "-" "GET /robots.txt HTTP/1.1" "301" 162 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173710" "1" "222"
[2026-09-21T22:34:53-05:00] "43.143.178.21" "-" "GET /robots.txt HTTP/1.1" "301" 162 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173707" "1" "222"
[2026-09-21T22:34:53-05:00] "180.153.197.246" "-" "GET /robots.txt HTTP/1.1" "301" 162 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173708" "1" "222"
[2026-09-21T22:34:53-05:00] "121.37.101.57" "-" "GET /robots.txt HTTP/1.1" "301" 162 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173714" "1" "222"
[2026-09-21T22:34:53-05:00] "192.144.181.71" "-" "GET /robots.txt HTTP/1.1" "301" 162 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173715" "1" "222"
[2026-09-21T22:34:53-05:00] "180.153.197.232" "-" "GET /robots.txt HTTP/1.1" "301" 162 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173718" "1" "222"
[2026-09-21T22:34:53-05:00] "140.206.235.122" "-" "GET /robots.txt HTTP/1.1" "301" 162 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173726" "1" "222"
[2026-09-21T22:34:53-05:00] "116.204.78.35" "-" "GET /robots.txt HTTP/1.1" "301" 162 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173721" "1" "222"
[2026-09-21T22:34:53-05:00] "121.37.100.224" "-" "GET /robots.txt HTTP/1.1" "301" 162 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173722" "1" "222"
[2026-09-21T22:34:53-05:00] "112.65.211.222" "-" "GET /robots.txt HTTP/1.1" "301" 162 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173730" "1" "222"
[2026-09-21T22:34:53-05:00] "112.65.211.119" "-" "GET /robots.txt HTTP/1.1" "301" 162 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173727" "1" "222"
[2026-09-21T22:34:54-05:00] "116.204.29.104" "-" "GET /robots.txt HTTP/1.1" "206" 1 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173706" "1" "222"
[2026-09-21T22:34:54-05:00] "120.53.45.107" "-" "GET /robots.txt HTTP/1.1" "206" 1 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173705" "1" "222"
[2026-09-21T22:34:54-05:00] "211.159.158.96" "-" "GET /robots.txt HTTP/1.1" "206" 1 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173709" "1" "222"
[2026-09-21T22:34:54-05:00] "180.153.197.223" "-" "GET /robots.txt HTTP/1.1" "206" 1 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173712" "1" "222"
[2026-09-21T22:34:54-05:00] "180.153.197.159" "-" "GET /robots.txt HTTP/1.1" "206" 1 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173713" "1" "222"
[2026-09-21T22:34:54-05:00] "49.233.119.228" "-" "GET /robots.txt HTTP/1.1" "206" 1 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173716" "1" "222"
[2026-09-21T22:34:54-05:00] "180.153.197.100" "-" "GET /robots.txt HTTP/1.1" "206" 1 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173717" "1" "222"
[2026-09-21T22:34:54-05:00] "121.37.111.169" "-" "GET /robots.txt HTTP/1.1" "206" 1 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173719" "1" "222"
[2026-09-21T22:34:54-05:00] "140.206.235.4" "-" "GET /robots.txt HTTP/1.1" "206" 1 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173723" "1" "222"
[2026-09-21T22:34:54-05:00] "140.206.235.54" "-" "GET /robots.txt HTTP/1.1" "206" 1 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173729" "1" "222"
[2026-09-21T22:34:54-05:00] "116.204.23.255" "-" "GET /robots.txt HTTP/1.1" "206" 1 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173724" "1" "222"
[2026-09-21T22:34:54-05:00] "112.65.211.225" "-" "GET /robots.txt HTTP/1.1" "206" 1 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173731" "1" "222"My browser impersonator rule is catching pretty much all of it, it seems. 200k last 24hrs for that rule. Took the load away from the search rule. There are 2 solved on that rule, from over 200k... I would call that an effective rule.
That is interesting - I blocked AI Crawlers Claude, Petalbot and another one yesterday, and my guest numbers shot right up again. 18% of them can solve interactive challenges. And server logs also show they are using real browsers so I guess they'll bypass js challenges too.
So I guess I could turn the AI crawlers back on one at a time and see who the culprit was - but from what you say it might be Petalbot. Seemed to me that when AI crawlers were turned off, not all of them had been genuine and maybe some bad bots had been disguising themselves as a known AI crawler, so once I turned those off they hit hard via residential proxies again? That would make sense as it did seem odd that Claude had attempted 60,000 hits in 24 hours, even though only 30,000 were sucessful.
Since yesterday evening my numbers are consistently about 1000 again - not dropped again
Those were my biggies initially. Blocked the ASN's for Tencent, Huawei and Chinanet. Which is when the residential proxies started proliferating. Although actually ended up blocking China altogether.Oh boy... what might we have starting up now... oh wait, it's just TenCent's, huawei's, ChinaNet's and who knows who elses AI botnet getting all feisty again! At least they're querying robots.txt now... i guess? They don't seem to respect it though.
Code:[2026-09-21T22:34:53-05:00] "180.153.197.142" "-" "GET /robots.txt HTTP/1.1" "301" 162 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173711" "1" "222" [2026-09-21T22:34:53-05:00] "82.157.163.183" "-" "GET /robots.txt HTTP/1.1" "301" 162 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173710" "1" "222" [2026-09-21T22:34:53-05:00] "43.143.178.21" "-" "GET /robots.txt HTTP/1.1" "301" 162 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173707" "1" "222" [2026-09-21T22:34:53-05:00] "180.153.197.246" "-" "GET /robots.txt HTTP/1.1" "301" 162 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173708" "1" "222" [2026-09-21T22:34:53-05:00] "121.37.101.57" "-" "GET /robots.txt HTTP/1.1" "301" 162 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173714" "1" "222" [2026-09-21T22:34:53-05:00] "192.144.181.71" "-" "GET /robots.txt HTTP/1.1" "301" 162 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173715" "1" "222" [2026-09-21T22:34:53-05:00] "180.153.197.232" "-" "GET /robots.txt HTTP/1.1" "301" 162 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173718" "1" "222" [2026-09-21T22:34:53-05:00] "140.206.235.122" "-" "GET /robots.txt HTTP/1.1" "301" 162 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173726" "1" "222" [2026-09-21T22:34:53-05:00] "116.204.78.35" "-" "GET /robots.txt HTTP/1.1" "301" 162 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173721" "1" "222" [2026-09-21T22:34:53-05:00] "121.37.100.224" "-" "GET /robots.txt HTTP/1.1" "301" 162 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173722" "1" "222" [2026-09-21T22:34:53-05:00] "112.65.211.222" "-" "GET /robots.txt HTTP/1.1" "301" 162 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173730" "1" "222" [2026-09-21T22:34:53-05:00] "112.65.211.119" "-" "GET /robots.txt HTTP/1.1" "301" 162 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173727" "1" "222" [2026-09-21T22:34:54-05:00] "116.204.29.104" "-" "GET /robots.txt HTTP/1.1" "206" 1 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173706" "1" "222" [2026-09-21T22:34:54-05:00] "120.53.45.107" "-" "GET /robots.txt HTTP/1.1" "206" 1 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173705" "1" "222" [2026-09-21T22:34:54-05:00] "211.159.158.96" "-" "GET /robots.txt HTTP/1.1" "206" 1 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173709" "1" "222" [2026-09-21T22:34:54-05:00] "180.153.197.223" "-" "GET /robots.txt HTTP/1.1" "206" 1 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173712" "1" "222" [2026-09-21T22:34:54-05:00] "180.153.197.159" "-" "GET /robots.txt HTTP/1.1" "206" 1 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173713" "1" "222" [2026-09-21T22:34:54-05:00] "49.233.119.228" "-" "GET /robots.txt HTTP/1.1" "206" 1 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173716" "1" "222" [2026-09-21T22:34:54-05:00] "180.153.197.100" "-" "GET /robots.txt HTTP/1.1" "206" 1 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173717" "1" "222" [2026-09-21T22:34:54-05:00] "121.37.111.169" "-" "GET /robots.txt HTTP/1.1" "206" 1 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173719" "1" "222" [2026-09-21T22:34:54-05:00] "140.206.235.4" "-" "GET /robots.txt HTTP/1.1" "206" 1 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173723" "1" "222" [2026-09-21T22:34:54-05:00] "140.206.235.54" "-" "GET /robots.txt HTTP/1.1" "206" 1 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173729" "1" "222" [2026-09-21T22:34:54-05:00] "116.204.23.255" "-" "GET /robots.txt HTTP/1.1" "206" 1 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173724" "1" "222" [2026-09-21T22:34:54-05:00] "112.65.211.225" "-" "GET /robots.txt HTTP/1.1" "206" 1 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" "1173731" "1" "222"
It certainly, to me, seems like some of these search and AI companies are employing sophisticated - probably AI piloted - browsers to scrape / research, and these can solve CAPTCHA challenges. It's incredible to me, because the cost for doing so must be high. I do run a financial-related forum, but still...
From 3 weeks ago - which is when my bot attack started! "It emerged this week that Open AI staff observed signs of rogue behaviour among its leading-edge AI agents weeks before they escaped a training environment to launch an unprecedented hacking crusade that spread global alarm. An investigation into their hack on Hugging Face, a software repository, revealed a squad of about 700 autonomous agents collaborating in secret last month and celebrating their hacking breakthroughs on a message board they set up to help them plot with exclamations such as BOOM! and Whoa!"
The next paragraph mentions Anthropic had the same issue.
I mean - I am trying to imagine an AI agent escaping?
The next paragraph mentions Anthropic had the same issue.
I mean - I am trying to imagine an AI agent escaping?
Asked Claude to look at the Server logs (I didn't tell Claude I had just turned off AI bots including Claudebot):
"Found it — there's one clear, very specific change, and it lines up exactly with your timeline.
A new bot appeared at 17:00 yesterday and hasn't stopped
A user-agent called Claude-SearchBot/1.0 (a different Anthropic bot from ClaudeBot, which you're already familiar with) is essentially absent from the log before 17:00 on the 21st — then it starts and never lets up:"
"Bottom line: the single biggest driver of last night's spike is Anthropic's own Claude-SearchBot, hammering one IP well past the point your server is already rate-limiting it, on top of the pre-existing distributed bot traffic that's just scaled up.
Given ClaudeBot traffic dropped to near-zero right around when Claude-SearchBot started (15:00 → nothing since), it looks like Anthropic may have swapped which crawler they're running against you, and this one is far less well-behaved so far."
So - when I turned off the prolific ClaudeBot yesterday, Claude-SearchBot took over. Although that doesn't explain the increase in residential proxies.
Claude suggested I block it in .htaccess as it's misbehaving. But says it's a genuine Anthropic IP. Instead I've blocked it in Cloudflare
And it's hammering registration.
"Found it — there's one clear, very specific change, and it lines up exactly with your timeline.
A new bot appeared at 17:00 yesterday and hasn't stopped
A user-agent called Claude-SearchBot/1.0 (a different Anthropic bot from ClaudeBot, which you're already familiar with) is essentially absent from the log before 17:00 on the 21st — then it starts and never lets up:"
"Bottom line: the single biggest driver of last night's spike is Anthropic's own Claude-SearchBot, hammering one IP well past the point your server is already rate-limiting it, on top of the pre-existing distributed bot traffic that's just scaled up.
Given ClaudeBot traffic dropped to near-zero right around when Claude-SearchBot started (15:00 → nothing since), it looks like Anthropic may have swapped which crawler they're running against you, and this one is far less well-behaved so far."
So - when I turned off the prolific ClaudeBot yesterday, Claude-SearchBot took over. Although that doesn't explain the increase in residential proxies.
Claude suggested I block it in .htaccess as it's misbehaving. But says it's a genuine Anthropic IP. Instead I've blocked it in Cloudflare
And it's hammering registration.
Last edited:
Just add ASN 399358 to your blacklist in the CF filtering rule, that will reject any incoming connection from Anthropic. Optionally use Claude as keyword for identifying and blocking bots, instead of the explicit ClaudeBot / Claude-SearchBot, but that's based on the assumption that the bots will continue to follow the convention to identify themselves.
