Crazy amount of guests

[Anthony Parsons]

I enabled the Managed Challenge for parts of the world and my members are complaining about the constant "checking if you're human" warnings.

In all honesty, I would not enable any firewall features yet, until you learn and understand their implications. You can block all good crawlers too, just by manage challenging US for example, you block all crawlers from the US.

Use chatgpt or claude free versions to help you, tell them to be cautious with recommendations, instead help them talk you through command line log analysis, copy paste to them, let them do everything, and find the real bad IP's and then decide whether you take out the CIDR or ASN.

You are better to always start small. An ASN with just 256 IP blocks is usually a bad actor, they flip them as they burn them out. But an ASN with 64K blocks... you have to make informed decisions.

You can take out Amazon servers, for example, they're not an ISP, however, they are to large companies. Amazon provide all the IT infrastructure needed that instead of buying, companies rent and write-off the cost, whilst Amazon maintains everything. Companies get internet, security, compliance, email systems, reliability, redundancy, backup, cooling, electricity, file storage, etc etc etc, Amazon do 80% of the hard work, the company only needs IT for setup, coding, etc. You can take out hundreds of thousands of legitimate users, all just thinking you're blocking Amazon servers, who aren't an ISP, but they are in a different way, just not to homes, but businesses.

My point is, use AI chat to help you understand things before you do them.

Oh... and you want to be using this: https://xenforo.com/community/resources/digitalpoint-app-for-cloudflare-r.8750/

Enable guest caching, that will take a huge hit off your server. Use chatgpt to help you setup the free caching and get it right for your images, being a photography forum, basically, ensure the CDN and caching component of CF is functioning correctly for your site type, as you will get significant benefit from this, especially bots hitting the content. No longer loading from your site, instead loading from CF cache.

 

[Levina]

In all honesty, I would not enable any firewall features yet, until you learn and understand their implications. You can block all good crawlers too, just by manage challenging US for example, you block all crawlers from the US.

Use chatgpt or claude free versions to help you, tell them to be cautious with recommendations, instead help them talk you through command line log analysis, copy paste to them, let them do everything, and find the real bad IP's and then decide whether you take out the CIDR or ASN.

You are better to always start small. An ASN with just 256 IP blocks is usually a bad actor, they flip them as they burn them out. But an ASN with 64K blocks... you have to make informed decisions.

You can take out Amazon servers, for example, they're not an ISP, however, they are to large companies. Amazon provide all the IT infrastructure needed that instead of buying, companies rent and write-off the cost, whilst Amazon maintains everything. Companies get internet, security, compliance, email systems, reliability, redundancy, backup, cooling, electricity, file storage, etc etc etc, Amazon do 80% of the hard work, the company only needs IT for setup, coding, etc. You can take out hundreds of thousands of legitimate users, all just thinking you're blocking Amazon servers, who aren't an ISP, but they are in a different way, just not to homes, but businesses.

My point is, use AI chat to help you understand things before you do them.

Oh... and you want to be using this: https://xenforo.com/community/resources/digitalpoint-app-for-cloudflare-r.8750/

Enable guest caching, that will take a huge hit off your server. Use chatgpt to help you setup the free caching and get it right for your images, being a photography forum, basically, ensure the CDN and caching component of CF is functioning correctly for your site type, as you will get significant benefit from this, especially bots hitting the content. No longer loading from your site, instead loading from CF cache.

Yes, I'm talking to ChatGPT at this very moment. I've turned off managed challenges and am blocking countries at the moment.

Traffic has normalised, the good bots are back. And I am cautiously moving forward now, guided by AI. It gets that I need to be careful not to affect genuine visitors (and potential new members) and also make sure the good bots face no obstacles whatsoever. I'll get it right I'm sure.

I came here to look at digital point's add-on.

I will look at guest caching. Although "your server", it's Xenforo's server as I am on Cloud hosting.

 

[Anthony Parsons]

Although "your server", it's Xenforo's server as I am on Cloud hosting.

I'm not certain how XF allocate their server, but normally, that space is allocated x amount of cpu and ram. So guest caching affects how those allocated resources to you are used, and gives your users a better experience, snappier page loads.

 

[ES Dev Team]

Looks like cloudflare's captcha is getting defeated. Not good. ASN blocks are necessary for this case.

 

[puterfixer]

It was never a question of "if" but "when". sigh

10+ years ago I was abruptly introduced to a server pulled out of a rack after disks failed unnoticed one by one, putting all 4 mirrors in single-disk mode, and finally the 5th failed drive taking down a mirror completely, crashing the operating system. The previous "admin" has not performed any maintenance on it for years - no updates, no checks, no backups, nothing - and also bailed when failed disks called for a disaster recovery. It was blind luck that the server was not compromised in the years before it arrived in my livingroom for data recovery.

Even back then it was no longer a safe internet, "set it and ignore it" was like playing russian roulette with a single barrel shotgun. I had to step in and take the admin role for a replacement server, which was loaded with a hardened OS with some intrusion detection tools. I was shocked of how many bots continuously scan and probe for any vulnerability or open service, to brute force their way through any login, and prove how absolutely mad the previous admin's negligence has been. Fail2ban became my all-seeing friend, working in tandem with the almighty dynamic firewall, with the only limit being the processor. The blocked IPs would get into the hundreds per day, and I started publishing occasionally my list of top offending subnets that were permanently banned on my server.

Then came cost saving by moving to a shared hosting, where the server management is no longer my problem and I no longer have visibility on all those intrusion attempts; it's enough work to focus on the web application level security. Then CloudFlare was brought in to filter stuff based on rules that the webhost was not (easily) capable of, and now I had the power to manage ASNs entirely, not just subnet after subnet. So that list of subnets went dormant.

It's probably time to start maintaining (public) lists of ASNs that might be useful to be blocked. Join the forces of CI - Crowdsourced intelligence. Hehe

On that note, Tencent has the ASN 132203 but also 45090.

 

[Anthony Parsons]

Fail2ban became my all-seeing friend, working in tandem with the almighty dynamic firewall, with the only limit being the processor.

Yup. I have a server outside professional infrastructure, its locked up tighter than a fishes butt hole, with minimal essential ports, locked to only country of use, and it still blocks over 1000 IP's daily just scanning ports and response. Only end to end encryption can access it... but its nuts to watch the blocking tail when bored. The internet is a wild wild place.

 

[Wildcat Media]

You can take out Amazon servers, for example, they're not an ISP, however, they are to large companies. Amazon provide all the IT infrastructure needed that instead of buying, companies rent and write-off the cost, whilst Amazon maintains everything.

The problem is that some VPNs also use Amazon, and we had numerous complaints when I initially blocked Amazon's servers via their ASN. They're on a managed challenge now.

 

[Wildcat Media]

Looks like cloudflare's captcha is getting defeated. Not good. ASN blocks are necessary for this case.

And if they are defeating it, it's only a matter of time before they either sell/give this knowledge to others, or other bots learn to bypass the challenges also.

I did block both of Tencent's ASNs but I assume there may be others. We'll probably get complaints, but I told staff that full stop, anyone on Tencent is not getting in anymore. At least until Cloudflare eventually decides to fix this.

 

[puterfixer]

The problem is that some VPNs also use Amazon, and we had numerous complaints when I initially blocked Amazon's servers via their ASN. They're on a managed challenge now.

I'm sick and tired of people trusting the misleading marketing around VPNs and their false claims for privacy, then routing their entire internet traffic through a VPN company from a Banana Republic which decrypts and fingerprints their entire traffic for commercial purposes. Basically they're being scared about a potential man-in-the-middle attack, and then they are happily paying a company that's legally unreachable to do exactly the same. WTF!!!

If they want to shoot themselves in the foot, that's fine, but I feel no obligation to enable their ill advised decisions by allowing traffic from some servers in a datacenter, from companies with whom I have no agreement of any kind, and when exactly the same VPN endpoints are used by others to attack, scan, scrape, brute force etc. my domain. Sorry buddy, you put your trust in some scammers, suffer the consequences.

Renting out a virtual server in AWS is apparently reasonably cheap for people using it for vulnerability probing or running bots. The ratio of AWS employees vs. bots coming from AWS is zero.

The only legitimate reason I found to allow AWS is that some of my forum users have been using for years the Tapatalk client on their mobiles, and that traffic seems to come through AWS. Tapatalk used to offer a far better navigation experience back in the day when forums needed to have explicit themes/skins for mobile browsers, sometimes multiple ones for various screen resolutions, and they all looked like crap. To have a single mobile client with a consistent interface that can connect to the plugins installed for the multiple different forums was nothing short of a blessing. But then responsive interfaces came around for a much better mobile experience in the browser, and Tapatalk fell out of grace. It doesn't help that they caused some beef with CloudFlare and they do not disclose their IPs to be whitelisted; right now they're with a foot in the grave, no announcements in nearly a year, no updates to their forum plugins, their support forum is dead... Good riddance.

 

[Wildcat Media]

I'm sick and tired of people trusting the misleading marketing around VPNs and their false claims for privacy, then routing their entire internet traffic through a VPN company from a Banana Republic which decrypts and fingerprints their entire traffic for commercial purposes. Basically they're being scared about a potential man-in-the-middle attack, and then they are happily paying a company that's legally unreachable to do exactly the same. WTF!!!

There are a few trusted VPNs out there, but there are so many others with shady backgrounds and I agree that so many Internet users are believing the marketing without knowing the truth behind them. They don't have the technical expertise either to know what some of these VPNs are doing with their data behind the scenes.

My task in the next 4-5 weeks is to write up something for the non-technical forum members explaining all this bot traffic, and all the issues we are having in doing so, along with some of our methods to try and combat them. The staff gets asked these questions and I need to have something out there for them, in a way they can understand. I'm pretty much unavailable until about mid-July so it will have to wait. But I don't think anyone not on our technical side of the forum can ever understand what we are all going through fighting these AI slop bots.

Tapatalk used to offer a far better navigation experience back in the day when forums needed to have explicit themes/skins for mobile browsers, sometimes multiple ones for various screen resolutions, and they all looked like crap.

Tapatalk also shot itself in the foot with the way they accessed the forum (in XF, if I recall, their new account registration process circumvented what was built into XF, and allowed spammers to register which would normally have been trapped by XF's spam detection...and that's just one example of their unorthodox methods), as well as being very secretive with everything. Not to mention other transgressions like introducing new features full of bugs that never got fixed, and syndicating some forums' content via their own interface without permission. When we dumped vBulletin for XF, we let users know Tapatalk was going away. (That was 15 years ago!) I don't understand, nor trust, any forum that still uses it. When I briefly tried monetizing a few of my forums through them, I had to get a third party involved to get the money, as they were not sending payments nor were they responding to any emails (which was common with them for everything--very uncommunicative).

TL;DR--no love lost between myself and Tapatalk.

 

[smallwheels]

The problem is that some VPNs also use Amazon, and we had numerous complaints when I initially blocked Amazon's servers via their ASN.

I do have one user where this was the case after I blocked the Amazon ASNs. Whitelisted his VPN IP and we were good to go. Obviously depends from how many of these users you have if that is a valid solution but given the amount of IPs that Amazon has within it's various ASNs (151.446.600 V4 IPs alone in AWS) most of them should be easy to block (the more as the ASNs are geographically distributed) and given the amount of crap that comes from Amazons IPs it is more than justified. According to Cloudflare's Radar 9% of all bot traffic worldwide is coming from one Amazon ASN (and another 6,1% from a second one):


In other words: According to Cloudflare Amazon is the worst offender worldwide when it comes to bots. By far. If you want to block anyone Amazon should be on your list.

Companies do use Amazon and AWS for their infrastructure because it works. Well, possibly time to show them, that it does not work, as their own web traffic now comes from the same source as the scumbags of the internet. If enough website owners block Amazon ASNs this will lead to either companies leaving Amazon, the baddies leaving Amazon or Amazon caring about wiping out the scumbags from their ASNs. All three options seem equally preferable. Bending towards Amazon on the other hand will make matters worse and cause nothing but trouble for your own infrastructure and forum.

 

[Anthony Parsons]

My task in the next 4-5 weeks is to write up something for the non-technical forum members explaining all this bot traffic, and all the issues we are having in doing so, along with some of our methods to try and combat them.

And this is case in point about this thread, and sometimes the validity of allowing some AI bots. I asked ChatGPT 5.5 to scan this thread, all current 30 pages, understand the issues, sort all the noise, add solutions not mentioned, and it spat out a novice guide to help block things. Now... I didn't ask it to add examples, I'm sure it could, but it outlined things fairly well.

 

[eva2000]

The problem is that some VPNs also use Amazon, and we had numerous complaints when I initially blocked Amazon's servers via their ASN. They're on a managed challenge now.

Yup had one service I use email me saying they have issues contacting my server. They used Amazon AWS

And this is case in point about this thread, and sometimes the validity of allowing some AI bots. I asked ChatGPT 5.5 to scan this thread, all current 30 pages, understand the issues, sort all the noise, add solutions not mentioned, and it spat out a novice guide to help block things. Now... I didn't ask it to add examples, I'm sure it could, but it outlined things fairly well.

Ask it for a 2nd and 3rd pass with each pass it elaborates and 'levels up' the doc

 

[Anthony Parsons]

Ask if for a 2nd and 3rd pass with each pass it elaborates and 'levels up' the doc

gonna run that right now, thanks for the tip.

 

[Anthony Parsons]

Put into pdf this time... second pass. Yep, it found a few more things and improved some previous aspects.

 

[BrettC]

Looks like cloudflare's captcha is getting defeated. Not good. ASN blocks are necessary for this case.

View attachment 339082

I'm not saying "told you so" (Fall 2025), but well... we all knew something was going on. It's well known that a third party has been mass-scraping the internet for just about anything and everything.

A multi-layered approach is now required if you care about user privacy (fingerprinting), and if you value your community's content/data on your websites. Indexing and ranking is one thing, but to fully scrape and store the content - no thanks... And that's barring bandwidth/resource costs, if that is a thing for many of you folks out there.

It's probably time to start maintaining (public) lists of ASNs that might be useful to be blocked. Join the forces of CI - Crowdsourced intelligence. Hehe

On that note, Tencent has the ASN 132203 but also 45090.

I've been keeping a list of them at https://asn.potatoforinter.net/ (there are netblock overlaps in this and the aggregated lists might be too aggressive) - updates every few weeks in the event they acquire more or release IPv4 and IPv6 netblocks. I am sure someone out there has this automated and posts to Github with a cleaner presentation.

 

[Anthony Parsons]

@eva2000 you are bang on mate... v3 refined it. Always learning. Also add a laymans version, for the less tech savvy forum owners.

 

[BrettC]

@eva2000 you are bang on mate... v3 refined it. Always learning. Also add a laymans version, for the less tech savvy forum owners.

Fairly decent overview/summary of the thread. Some 2c's worth of feedback...


Slight oversight(?) on the technical summary, which might exist on the laymans PDF as well, but this part:

Self-hosted forums have more options, but also more responsibility. The thread repeatedly shows that server-level controls can outperform simple CDN settings when tuned well, but they require Linux, web server and log-analysis skills.

Linux isn't a 100% requirement, but it is indeed where most of the 'good stuff' happens at. Some maniacs out there still cling to IIS/Windows or other OS-based software that isn't 100% linux.

Section 10,

Do not treat all bots equally. Search crawlers can be valuable. Some AI crawlers may be acceptable depending onthe forum's goals. Abusive scrapers, spoofed bots and residential proxy networks are the problem.

'Spoofed Bots' isn't exactly the 'best of terminology' to utilize in this example. Maybe 'Bots utilizing falsified user agents' would suffice better?



Overall, this is a nice start. Getting your parser/AI-organizer to add in sources to the posts and depicting examples apart from the basis of Cloudflare that I seen in the PDF would be greatly beneficial.

 

[smallwheels]

Overall, this is a nice start.

Yes and no. It makes it easy to get an overview about the thread quickly, but it is typical AI in the sense that it looks nice and uses elaborate language, creating the impression it would be a high quality "tutorial" around the topic, while it even misses out relevant parts of the thread content, let alone ignoring other things that have not been explicitly or loudly mentioned in this thread and partly covers gaps with slop. The Linux example that you mentioned is one good example, but I see a couple more shortcomings, in both, v1 and v3 (did not read v2). v1 seems in parts even better to me than v3.

Examples:

• in v3 it mentions the rising relevance of protecting the privacy of forum users against scraper bots - but irritatingly limits it to pictures in photo forums. This misses the point entirely.
• in v1 it mentions the option to limit guest view of content (which seems to have gone void in v3) but states that would lower registrations. The opposite is the case as I can confirm from experience.
• it does not distinguish between shared hosting and XF cloud while this can make a world of a difference regarding the options one has
• it does not at all consider or mention the classical "it depends" approach: What measures are appropriate heavily depends from your audience, the size of the forum, the geographical distribution of the users, the time you are willing to invest and foremost what you want to achieve. Instead, it gives flat out advice what to do w/o considering any of that.

etc. etc.

So while it looks convincing on first spot (especially if you do not know what really has been written in the thread) it is in fact typical AI: Overconfidential, missing out the fine grain and thus becoming possibly even misleading - and does not replace actually reading the thread, if you really want to know what the aspects are.

 

[Mr Lucky]

So while it looks convincing on first spot (especially if you do not know what really has been written in the thread) it is in fact typical AI:

This is so crucial. If we use ai to summarise something we can't be bothered to read, it is very obvious that we are in no position to verify its accuracy. And yet so many use ai in that way, and have total confidence in it because it's, er, ai.

It comes down to the old fallacy I read it in the paper (or saw it on telly) so it must be true.