I don't block them... no need.
Crazy amount of guests
- Thread starter Levina
- Start date
-
- Tags
- bot swarms guests
When you IP ban CIDRs or wildcard ranges on XenForo, XenForo is configured to throw a 403 to the client. On the web server daemon, it should be "seeing" that being sent from XenForo's PHP code, and throws that 403 Forbidden HTTP header code to the client. In most configurations, the modified phrase/text isn't even needed.
When there is no IP ban:
Code:
root@box006:~# curl -s -o /dev/null -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64)" -w "%{http_code}\n" [...]
200When the IP ban is applied and active:
Code:
root@box006:~# curl -s -o /dev/null -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64)" -w "%{http_code}\n" [...]
403In common web server daemon configurations (nginx:
proxy_intercept_errors, apache: ProxyErrorOverride On), you can actually intercept HTTP codes of your choosing to replace it with a custom override page. Which further reduces all bandwidth & CPU processing requirements at the XenForo level, as it is now handled directly at the web server and not PHP. For example, on my testing environment, I applied an IP ban on XenForo to my IP address, and nginx intercepted the 403 status header from the backend server. Nginx at that point, sent this error to the browser... nothing from XenForo was reached.
Last edited:
When you IP ban CIDRs or wildcard ranges on XenForo, XenForo is configured to throw a 403 to the client. On the web server daemon, it should be "seeing" that being sent from XenForo's PHP code, and throws that 403 Forbidden HTTP header code to the client. In most configurations, the modified phrase/text isn't even needed.
I don't think that is the same on cloud, I banned an IP from my phone and got the Oops IP banned message from my forum and I added the 403 error.
If the site that you manage is what you have set on your profile... it is indeed sending 404s and 403s correctly to the browser via the status code header.
Since you're on the XF Cloud service, you're a bit limited regarding how to add overrides/interception of certain status codes.
A little bit of insight regarding user agent rotation by the bots: I had two guests IPs in the list that came from Lithuania from an ASN that I had recognized as a bot source earlier. They showed of repeatedly within a couple of minutes, using quite a bunch of different user agents. I can clearly exclude the possibility that these were genuine users or that this was a genuine proxy:
I've been debating whether or not to block AI crawlers in the cloudflare AI crawling options. Hadn't done anything as numbers are ok right now so it was more a case of - should I let them crawl or not? ClaudeBot has been crawling excessively however, and just now I saw it trying to register! So I am thinking I will block the AI crawers (not the AI Assistants or AI search) in Cloudflare. That is very cheeky of ClaudeBot hitting registration.
That is how these cloud servers are set up. The database and Elasticsearch servers only connect via non-public IP addresses in the 10.0.0.0/8 block, and their firewalls reject everything else except the other two servers' IP addresses, my own IP address, and from a couple other servers at other hosts, so I always have a backdoor to get in if my home IP changes. The Apache server connects to the Internet, but only CF's IP addresses are allowed, and the small number of other ports I need like SSH are again permitted only via just a couple of IP addresses. (The hosting company also has their own console through my account.) I like the networking end of all this--it is actually fun to set up and test.
Regarding Amazon's bots...
I whitelist two of Amazon's bots. We are Amazon affiliates and two of their crawlers might be checking to see if the links are valid and within their terms (not verified that they do this, but why take the chance?). Amazon is flaky in that regard, and will send warnings if anything varies from normal. These would also apparently be used if a site hosted their dynamic widgets that update based on site content.
Amazon Contxbot. User-Agent: contxbot
Amazon AdBot. User-Agent: AmazonAdBot
I whitelist two of Amazon's bots. We are Amazon affiliates and two of their crawlers might be checking to see if the links are valid and within their terms (not verified that they do this, but why take the chance?). Amazon is flaky in that regard, and will send warnings if anything varies from normal. These would also apparently be used if a site hosted their dynamic widgets that update based on site content.
Amazon Contxbot. User-Agent: contxbot
Amazon AdBot. User-Agent: AmazonAdBot
I'm also allowing Amazonbot for that reason. Even though it shows up as an AI crawler in Cloudflare.
Overnight update. On the busiest forum, our origin loads are still on a very even keel, and I've been logging server loads for years, once a minute, so I can look back at high traffic events. I'd say on average, our server during normal times has a 25% to 33% load, and this has held steady for three days now. I don't start analyzing traffic again unless I see spikes in requests on the origin server. Life's too short to worry about smacking down every bot out there. (And I love smackin' them down! 
)
Yet the forum owner sent an email saying he's been getting CF errors all weekend, and seemingly "every other time" he tries to access us. Another pal on the staff says he sees it occasionally but nowhere near as often
CF 522 error. Can't reach the origin server.
The server's firewall has all the correct IPv4 and IPv6 addresses.
I haven't seen it myself, and I'm often monitoring things during my workday. Unknown: the hosting company might be doing something? It's not like our loads are high--I've posted the plots above, and origin traffic is smooth sailing.
I don't know if there's any sort of log in our accounts to see what might be causing the 522 errors. Or if there's something in the CF DNS settings that should be adjusted.
Any ideas?
)Yet the forum owner sent an email saying he's been getting CF errors all weekend, and seemingly "every other time" he tries to access us. Another pal on the staff says he sees it occasionally but nowhere near as often
CF 522 error. Can't reach the origin server.
The server's firewall has all the correct IPv4 and IPv6 addresses.
I haven't seen it myself, and I'm often monitoring things during my workday. Unknown: the hosting company might be doing something? It's not like our loads are high--I've posted the plots above, and origin traffic is smooth sailing.
I don't know if there's any sort of log in our accounts to see what might be causing the 522 errors. Or if there's something in the CF DNS settings that should be adjusted.
Any ideas?
I mean, where would we even find the results of this setting?
CF's menus have gotten too dense and unorganized, and full of too many clickbait menu items that try to get us to upgrade to other plans. If we have a free plan, we shouldn't see any additional settings or options not available at our account level. I don't have an hour to dig through everything to find some obscure set of logs that they don't make easy to find.
Then there's this:
Yet another clickbait option, which sort of implies we can't even view the logs.
CF's menus have gotten too dense and unorganized, and full of too many clickbait menu items that try to get us to upgrade to other plans. If we have a free plan, we shouldn't see any additional settings or options not available at our account level. I don't have an hour to dig through everything to find some obscure set of logs that they don't make easy to find.
Then there's this:
Yet another clickbait option, which sort of implies we can't even view the logs.
Could it be an ASN challenge somewhere?
I've seen these here on the forums as well from time to time and also more or less often on other websites - with some of them relatively frequently, with most rather rarely.Yet the forum owner sent an email saying he's been getting CF errors all weekend, and seemingly "every other time" he tries to access us. Another pal on the staff says he sees it occasionally but nowhere near as often
CF 522 error. Can't reach the origin server.
The server's firewall has all the correct IPv4 and IPv6 addresses.
I haven't seen it myself, and I'm often monitoring things during my workday.
BTW, a helpful page if you want to see how your CF security rules are working:
Go to Investigate -> Trace, and you can enter a handful of parameters to see if your rulesets are working as expected.
Go to Investigate -> Trace, and you can enter a handful of parameters to see if your rulesets are working as expected.
It's a connection error between CF and the origin, so no rules/challenges apply. Basically the origin isn't responding as quickly as CF would like.
I'd assume that these three are not the same bot. As many of the big techs Amazon has more than one bot, serving different purposes - i.e. here they name three:
About AmazonBot
Customer facing page of Amazonbot crawler which all web content publishers can refer to.
developer.amazon.com
Amazonbot is clearly an AI training bot:
So in fact a case for the big middle finger.
I'm tracking down a few server settings, as I don't know if Keep-Alive is set properly (which CF recommends). This is a handy checklist to work through:
Also:
CF isn't showing them in any of the AI categories. If it is tagged "Training" or "Agent" it requires more investigation.
Our other staffer says it happens infrequently.
I don't know the full details of CF's network, but I'm wondering if CF has hundreds of different access points from their servers to the origins. The problem seems to be Los Angeles, while the other staffer is in Iowa.
