Crazy amount of guests

[Levina]

It takes time to chase nasties around the internet, and even when you think you caught them, they change IP's / ASN's / use residential proxies. You can tune and forget, or spend time daily / weekly trying to stop it all. There is a healthy middle ground IMO, takes maybe 5 minutes weekly, but they still shift faster than you will ever block.

After spending a few days with the IP Threat Monitor this makes a lot of sense to me.

I have countries blocked, a few ASN providers blocked. And now I have genuine guests plus a fair number of what I believe to be residential proxies. I can blacklist those IP addresses but it's like playing Whac-A-Mole, you kill one, another pops up. I would be doing it all day every day. With 400-500 guests now I'm pretty happy, even if there are bots in there. It pales to the 30k+ I had all week.

 

[Anthony Parsons]

it's like playing Whac-A-Mole, you kill one, another pops up

Yep. In all honesty, your best bet is to tune your server well and just let it go. Honestly, you don't even need XTR if you take that approach. Cloudflare is much better than any server tool, because they aren't reaching your server then. Check your logs weekly or fortnightly, find any massively abusing IP, block that CIDR at CF, clear your logs and walk away until next week/fortnight. Initially you will have several abusing, but you quickly revert to seeing your largest use is good bots or your own IP/server.

 

[smallwheels]

In all honesty, your best bet is to tune your server well

Levina is still on XF Cloud (as written a couple of times). So how would she tune her server well?

and just let it go.

Not everyone is fine with AI bots scraping the content of his or her forum.

Cloudflare is much better than any server tool

Levina has said a couple of times in this thread that she does not want to use Cloudflare.

While I agree that the whack-a-mole game with single IPs does not make sense at all I have my guest count down reliably to typically below 50 w/o using Cloudflare through IP-Threat Monitor and country and ASN blocking and building up the ruleset over time. Does not make much sense to give advice and completely ignoring the situation and needs of the person you give advice to.

 

[Levina]

Yep. In all honesty, your best bet is to tune your server well and just let it go. Honestly, you don't even need XTR if you take that approach. Cloudflare is much better than any server tool, because they aren't reaching your server then. Check your logs weekly or fortnightly, find any massively abusing IP, block that CIDR at CF, clear your logs and walk away until next week/fortnight. Initially you will have several abusing, but you quickly revert to seeing your largest use is good bots or your own IP/server.

As smallwheels says, I'm on Cloud hosting so I have no access to the server. I do plan to switch to self hosting by the end of this year and then I might decide to go with Cloudflare after all. It depends on how I would fare on my own. At least I will have access to the .htaccess file.

Levina is still on XF Cloud (as written a couple of times). So how would she tune her server well?

Not everyone is fine with AI bots scraping the content of his or her forum.

No, I am not, especially since we are a photography forum and I would prefer to not have my members' images stolen, but I don't know how to battle these individual IP addresses. Yes I can add a ton of ASN providers but that would also cut off potential new members so I can't do that. Instead I just settle for a low amount of bots. I still blacklist IPs when I have a bit of time but I try not to worry about them too much.

Levina has said a couple of times in this thread that she does not want to use Cloudflare.

While I agree that the whack-a-mole game with single IPs does not make sense at all I have my guest count down reliably to typically below 50 w/o using Cloudflare through IP-Threat Monitor and country and ASN blocking and building up the ruleset over time. Does not make much sense to give advice and completely ignoring the situation and needs of the person you give advice to.

The problem is that I have a photography forum and my members e.g. come from America, Canada, Europe, Australia, New Zealand. We also have someone from Japan and someone from Taiwan. So I can't go blocking every ASN out there. I want my forum to grow. I could block Japan and Taiwan for sure because although we have the odd member from those countries, I don't expect there to be a lot of potential new members. But Europe? America? Canada? Australia and New Zealand? Most definitely.

Right now I have exactly 84 guests in the house and they are all legit, browsing the forum, viewing threads, discussions, images in the gallery, none go where they are not allowed. It's a normal I would expect for my small forum. Earlier we had around 400 guests and I estimated that about 300 were bots. So apparently they still come in waves, except the top of the wave is now acceptably low. So yeah, not so bad now I think.

 

[BrettC]

It will probably put a bit of a dent in the amount of residential proxies but I doubt that it will change the general trend. What is interesting however: The Ars Technica article specifically mentions Asocks. Asocks advertise on their webpage specifically to developers of apps the option to integrate it into their apps as an option to earn money by making users of apps residential proxy hosts, using whitewashed wording:



They also name a couple of existing "partners" on that page:


View attachment 338839

Also, Asocks does seem to share at least parts of its IP pool with other providers of residential proxies, judging from the logs I find in my proxycheck.io dashboard. I.e. the one I shared recently. Proxycheck seems to have invested into discovering residential proxy networks recently, as more and more of these are discovered by name over the last weeks and in many cases the same IPs are used by multiple proxy services and also by VPN providers. Possibly this may affect only parts of their IP pool, yet it is for one a warning to pay attention when selecting a VPN provider (to not accidentally become a mule for bad actors unknowingly) and secondly the hit against Asocks may possibly hit other proxy providers partly as well as a very welcome collateral damage.

A couple of proxy providers I found in my logs that share IPs with Asocks according to proxycheck.io are:

• Nullproxies
• IP Royal
• Packetstream
• reProxy
• oxylabs
• evomi

As these are just random findings there may obviously be more.

I was curious about that proxycheck.io service. Quite a few of my servers are interlinked via VPN (non-standard port) for firewall and internal communication... however none of the IP addresses that I inputted are returning with the flag of VPN. Just lights up with Hosting with a risk score of 33%. My static IP for my ISP also hosts a VPN, on a non-default VPN port. It is returning as "Clean" with a risk score of 0%.

So from the looks of things, with my anecdotal findings, their service is doing port probing on the default VPN ports or well-known abused VPN ports.

Too bad this service doesn't have an API to query against: https://multirbl.valli.org/dnsbl-lookup/ - because that'd be sure useful.

 

[smallwheels]

The problem is that I have a photography forum and my members e.g. come from America, Canada, Europe, Australia, New Zealand. We also have someone from Japan and someone from Taiwan. So I can't go blocking every ASN out there. I want my forum to grow. I could block Japan and Taiwan for sure because although we have the odd member from those countries, I don't expect there to be a lot of potential new members. But Europe? America? Canada? Australia and New Zealand? Most definitely.

Yes, geoblocking works, but is pretty lethal, too. With some countries like i.e. Vietnam, Brazil and Bangladesh you don't have much choice due to the massive amount of residential proxies coming from there. With others, like i.e. Iraq, it does probably do no harm for most forums as barely legitimate users are expected to come from there. But surely, the US and Central Europa are a different story and with loads of residential proxies and other scrapers coming from the US you face a problem.

ASN blocking is worth it as you can limit it pretty much on server / data center ranges and those should not visit your forum anyway. Does not solve the issue with residential proxies, but on the other hand there are loads of ASNs that are pretty notorious for scraping, either being bad actors themselves or renting out IPs or servers/VPS etc. to scrapers either knowingly or unknowingly or tolerating them. Many of those ASNs span IPs in multiple countries and blocking them does absolutely no harm and won't affect normal users at all. You probably do have a lot of open potential here. Building up the list of those ASNs takes some time, but the logs of proxycheck.io are pretty helpful. Personally I am pretty rigorous and often block complete ASNs already after the first, second or third time an IP from them is popping up in the logs, as long as they are not residential / end user ranges.

With residential ranges you have a problem: You don't want to block them but at the same time you know, that there are a lot of residential proxies hidden within them. At the same time, while there are some with fixed IPs, with most the IPs change frequently. So plain IP based blocking is not a good solution if you expect regular users from these ranges, too. So you will need to use other criteria, i.e. a forensic approach about the identity of the client (user agent, fingerprinting, etc.) or behavioral approaches.

IP Threat Monitor offers rate limiting, which is neat, but today in my eyes not the most relevant aspect any more, as many if not most residential proxies do send only a single request before switching to the next IP. For everything else it does rely on proycheck.io and it's IP reputation API which works pretty well (and improves continuously) but can't cope with the amount of residential proxies and the speed they are using fresh IP addresses. So basically what you'd need is a real time check of any IP sending a request.

This works best with a combination of a bunch of criteria from IP reputation over forensics and fingerprinting but is an endless fight that is pretty complex and prone to failure. Using Cloudflares managed challenge instead, based on criteria as described earlier in this thread, is a relatively simple approach with, as it seems, at the moment a pretty good success rate and minimal collateral damage, if any.

As Cloudflare does not know about typical behavior of your individual visitors or the individual structure of your forum it cannot act very sophisticated, this would need either a local tool, integrated into the forum software (so an add on) or a remote tool that can connect to your forum in real time (which to my knowledge does not exist at the moment).

The new Bot Guard add on goes into that direction and it will be interesting to see how it will evolve and if and how well it will integrate with i.e. Thread Manager and how the detection rates and the false positives will turn out to be.

So basically to fight the bots successfully there are a bunch of options and tools that work best when combined cleverly in a way that suits the individual needs of a certain forum and the targets its owner has. Which in the end becomes pretty complex and there is no "one size fits all".

One abvious downside of using multiple tools in parallel is that you do not have a single point of truth but a bunch of different logfiles, so a better integration of these tools would be neat. Which of the available tools one should choose for best success rates depends from the individual situation. And as the baddies are continuously permutating and evolving whatever one does needs to do that as well.

 

[puterfixer]

The point for having a filtering service upstream from your server is that you offload the rule-based decision onto a third party which excels at verifying a LOT of traffic and has its own data sources. Essentially CloudFlare becomes your firewall, and whatever passes through is significantly cleaner than what reached them in the first place. And they do it far, far more efficiently and faster through specialized hardware, not just generic software on an all-purpose server.

Your (forum) hosting normally runs an incoming connection through Authentication and Authorization to define the role and permissions, then assemblying the resulting page from the database, and delivering it.

If you add the firewall role to the server, then an incoming connection will first trigger some filters to check source, find out in what subnet and ASN does the visitor's IP belong by querying some external sources, check if any of them are blacklisted for access, and decide if the request should be passed on to the forum, challenged with a captcha, or straight off denied. This puts extra load on your hosting and adds a non-negligible amount of latency to all traffic, including legitimate one, more than what a service like CloudFlare would add. Essentially your host's capacity to serve pages to visitors will somewhat decrease.

I'm not advertising CloudFlare, just trying to show in what way such a service could be beneficial.

Unfortunately CF can also be a mild pain to set up properly and fine tune, even in its free package. And, it will become your only path to the site. Anything happens with CF (rarely, but it's not failproof), your site is also offline. Just yesterday I was reviewing some security recommendations it was throwing me, and one of them would tie up my hosting to CF too much imho so I left it off, despite its benefits.

For a forum with a bunch of big attachments or resources to download, CF might bring another benefit: R2 storage bucket hosting. With the massive help of @digitalpoint 's magnificent App for Cloudflare, essentially you no longer need to pay for massive attachment storage at your webhost, just for what the forum itself needs; all attachments will be in CloudFlare's R2 and will be served from there to visitors, typically also faster than if they'd come from your own host. There are ways to protect them from scrapers attempting to bypass the forum's AA mechanisms.

My forum has 7.5GB of attachments in 2 buckets, one public and the other internal only, and the regular traffic does not even come close to the very generous limits of free Class A and Class B operations. Only one time I was doing an attachment clean-up and resizing of photos (not the forum's purpose to host 30+ megapixel photos), and that particular time I exceeded the limits and got an invoice of... 3 cents. Yikes

 

[smallwheels]

I'm not advertising CloudFlare, just trying to show in what way such a service could be beneficial.

There has never been a doubt about that, yet some people want to avoid it, some for good reasons and some b/c they simply don't want it. Fur sure, having the "firewall/webfilter" on the same host and even within the same application that you want to protect is a big no-go if you follow classic security architecture guidelines. But then it does not need to be CF necessarily - there are loads of options.

Cloudflare is a CDN, a firewall, storage, DNS, captcha provider, an IP reputation manager and a couple of things more. So if you use it (or even all of it) you have a lot of comfort and features on a shoestring, but again a bit of a risk at the same time as you - again - consolidated a lot of things onto a single company that you have no control of (and are in most cases not even a paying customer of). I'm not saying it would be a no-go in principle - but you should be aware of the risks (including breaking up SLL and privacy) and you should be aware of the dependencies you are in when going down that road and of the monopoly you help to create and manifest.

 

[Wildcat Media]

Essentially CloudFlare becomes your firewall, and whatever passes through is significantly cleaner than what reached them in the first place. And they do it far, far more efficiently and faster through specialized hardware, not just generic software on an all-purpose server.

I see CF as having two overall benefits for bot traffic.

First, the traffic blocked at CF is not hitting our servers, which is ultimately ideal since that saves us CPU cycles and bandwidth. (That is one reason I don't like XF add-ons for this--it's not only CPU cycles and memory, it's the added layers of the web server process, and interpreting PHP to run the forum, both of which use even more CPU cycles.)

That said, I wouldn't be opposed to firing up a separate server and running my own filtering mechanism to send traffic through, before it gets to the forum's web server. That server could take the abuse, and leave our forum servers alone. Thing is, my days of server management, coding, etc. are in the past. So it's highly unlikely I could ever do this myself, not without a couple of years to find time to do it, test, tweak, etc.

The second benefit to CF is that even if scrapers are hitting us, they will be grabbing some of that content from our CF cache, so that further reduces hits on our server. I don't know the amount of cache hits the bots are retrieving, but I assume it is probably a smaller amount as many of these bots are hitting older content that hasn't been touched in 10 or 20 years, stuff that normally wouldn't be cached. (And I've noticed our cache hit percentage is down with all the increased bot traffic.)

 

[Wildcat Media]

This sounds a bit nutty, but honestly I'd rather give everyone a managed challenge, and then let everyone with the xf_user= cookie bypass everything and never get a managed challenge. Thing is, I tried that and logged-in users were still getting the managed challenge, and I had to disable it. And for another forum, it disabled the display of images when I tried it (although I probably need a rule to include CF's R2 buckets, I'm thinking?). I set the timeout for 7 days also. I have a feeling that CF's cookie detection isn't all it's cracked up to be.

I mean, is this not correct?

Whitelist (first rule):

(http.cookie contains "xf_user=")

Last rule (after others to allow legit bots):

(not http.cookie contains "xf_user=")

 

[Suzanne O]

Ask them a simple question.
Which country are you from?
With the answers being from those countries you've listed.
All the allowed countries on your forum.

 

[zeeb0t]

If you are still having bot ("guest") troubles - my XF Bot Guard add-on has just entered STABLE and is freely available for all XenForo 2.1+, 2.2+, 2.3+ forum owners. I welcome you to try it out.

View it here:

XF Bot Guard

May 28, 2026

Challenge suspicious bots before they scrape your forum.

• zeeb0t
• abuse bot bot blocking bot detection captcha fingerprinting scrapers scraping security spam
• Add-ons [2.x]

 

[Levina]

I just looked at how much views my forum had this past month. Close to one million views when my plan is up to 400,000! Oh dear...

Pretty insane for a small forum.

 

[smallwheels]

Pretty insane for a small forum.

Sadly, 85+% of those will probably have been bots.

 

[Levina]

Sadly, 85+% of those will probably have been bots.

Yes, I know! That's what's so crazy about it. It's abnormal for a small forum like mine to have so many bots visiting. There's nothing for them there!

Oh, and I caved and just switched to Cloudflare.

 

[smallwheels]

There's nothing for them there!

Well, you have loads of pictures and the AI bros finally want to learn how many fingers humans have and things like that. They need good pictures as training materials.

 

[Anthony Parsons]

Oh, and I caved and just switched to Cloudflare.

You didn't cave, you just saw the light, is all. Free full protection, DDOS, SSL, CDN and WAF. Cloudflare made other CDN's redundant.

 

[Levina]

You didn't cave, you just saw the light, is all. Free full protection, DDOS, SSL, CDN and WAF. Cloudflare made other CDN's redundant.

Thanks Anthony.

But I need to learn how to finetune Cloudflare. I enabled the Managed Challenge for parts of the world and my members are complaining about the constant "checking if you're human" warnings. Plus I came on the site this morning to find just one robot, the Python Requests Library. Whatever that is.

 

[smallwheels]

Python Requests Library. Whatever that is.

A scraper, lumpily coded on the basis of a generic Python library (Python is a programming language). Pretty common with primitive Chinese bots but also easy to block that user agent via .htaccess (which you don't have on XF Cloud).

 

[puterfixer]

A new day, a fresh mind, a new review of filters... At least this time I'm not working reactively, under pressure of the armies of bots, and can think slowly and thoroughly. Maybe. I hope.

Current approach being tested:

Reason	Condition	Action	Implementation
100% abusive traffic, no legitimate visitors expected	Source country codes: TOR China or Source ASNs: 2 of Alibaba cloud 2 of Huawei cloud 2 of Tencent cloud 2 of ChinaNet 1 of China Unicom others are being considered	Hard block	CloudFlare
Visits coming from known datacenters - can be bots, can be people using VPNs or jump hosts for legitimate or nefarious purposes	Source ASNs: long list of various cloud providers, continuously updated manually	Hard block, interactive challenge or managed challenge, depends how hard you want to squash them	CloudFlare
Indexing spiders for search engines	User-Agent is in a list of whitelisted agents and URL does not match some paths that I do not want indexed (and are also declared as disallowed in robots.txt, such as attachments or admin section)	Allowed	CloudFlare and/or Apache .htaccess
Scraper bots, AI agents, indexing spiders, SEO crawlers, Archival bots etc., all of them identifying themselves properly through User-Agent	URL is not /robots.txt (which would explicitly tell them that the entire site is disallowed from being indexed) and User-Agent is in a long list of known bad/undesired agents	Hard block	CloudFlare and/or Apache .htaccess
Unauthenticated users accessing areas that are meant only for registered members	Cookie does not contain "xf_user=" (for XenForo, there's another one for WordPress authentication for example; after they successfully log on and keep the cookie, the rule will not trigger anymore) and URL begins with [areas that are members-only, like private messages or viewing member profiles or /admin, or areas that are not meant at all for any visitor access like /internal_data or /src]	Managed challenge; when bots become capable of bypassing it, then escalate to Interactive challenge	CloudFlare, possibly .htaccess with more work
Spam bots	Bots registering from IPs in realtime blocklists (RBL)	Flag registration for manual approval	StopForumSpam plugin

Note 1: User-Agent has been followed as a convention, but is not mandatory or verifiable in any way. Spiders can decide if they follow or not the directives in robots.txt, or if they want to identify themselves with a valid and accurate User-Agent. There are plenty of bots that disregard all of this entirely, and they impersonate a regular browser, so it's up to you to decide if you want to trust them or better to constraint their access up to blocking them entirely.

Note 2: the order of the rules matters. If a rule is to whitelist some traffic based on specific conditions and let it skip all the other remaining rules, move it forward, but keep it behind any rule that you want to always be checked first. Also, the first rules to be applied are those in CloudFlare, as your upstream tunnel of traffic; then any local firewall on the server (if you have a managed server where you can play with that); then the Apache directives in the .htaccess for the web server itself; and finally in the forum software itself (permissions for Unregistered/Guest group).