Crazy amount of guests
- Thread starter Levina
- Start date
-
- Tags
- bot swarms guests
I currently have over 100,000 guests at my website, marathonstadium.com
I've made the site require a login for guests, but it hasn't stopped them.
Sorry to ask what's probably been asked a million times, but what's currently the best way to stop or discourage them? I've been searching solutions, but reading this thread it sounds like a lot of them are already outdated.
I've made the site require a login for guests, but it hasn't stopped them.
Sorry to ask what's probably been asked a million times, but what's currently the best way to stop or discourage them? I've been searching solutions, but reading this thread it sounds like a lot of them are already outdated.
Do you use cloudflare? That's the best start to stop the bulk of them. Most of them can't solve cloudflare challenges.
Set up a custom rule that challenges the paths they are going to most, but excludes logged in members and known bots (like google) so members don't get a challenge. And maybe take it from there.
Eg
(starts_with(http.request.uri.path, "/whats-new/") and not http.cookie contains "xf_user" and not cf.client.bot and http.referer eq "")
The last bit, referer eq "" means real guests don't get a cloudflare challenge either (they have a "referer" from google or the home page, whereas bots go direct to the path. However some bots fake having a referer so it could let some through - you could try with it first - that bit could always be taken out. The only real guests that would get challenged are those clicking on whats-new.
To stop approximately 80% of the problem, you need to use cloudflare as your DNS, being the simplest of changes to make if you aren't using them already.
Then, you turn on all their bot fighting systems, then add into their WAF security (firewall) to block any countries of zero traffic interest to you (China is prob #1 as they don't really allow outside websites to China but have a lot of abusive stuff coming out of it), then you need to target the things you can right now, headers of fake browsers pretending to be real, etc.
For a while now, I've been building my own software stack, and will soon leave XF and import it into my own stack. That I have far more control over, and much more ease to modify it, so I have much better anti-bot systems within it. It is lightweight and only contains what I need it to contain, no reliance on third party paid systems, and also runs faster on half again of my current server resources, which can also run approx 10x above my current load. I also don't need to wait for updates or deal with further complicated system design catering the masses. My point, it's getting to a place where Gov probably isn't going to do much of anything to fix the issues already happening and can pretty much foresee happening as AI integration becomes worse. We aren't far away from needing AI firewalls to fight AI penetration.
Yes block China as a country. I also blocked Malaysia and Myanmar.
Don't block Singapore as a country as Google and others can come from there - you don't want to block those.
For Singapore, I have these ASN numbers for big hosting/cloud centres blocked and that seemed to stop them from Singapore.
Singapore 139341 (Tencent), 136907, 141180, 151610 (Huawei) 134963 (Alibaba)
Thailand 137876 (Tencent again)
US 45102 (Alibaba cloud - global)
That should stop a lot to start with, but not the residential proxy bots which come from every country.
The custom rule suggested above will help with those as a start.
Don't block Singapore as a country as Google and others can come from there - you don't want to block those.
For Singapore, I have these ASN numbers for big hosting/cloud centres blocked and that seemed to stop them from Singapore.
Singapore 139341 (Tencent), 136907, 141180, 151610 (Huawei) 134963 (Alibaba)
Thailand 137876 (Tencent again)
US 45102 (Alibaba cloud - global)
That should stop a lot to start with, but not the residential proxy bots which come from every country.
The custom rule suggested above will help with those as a start.
Fair as a temporary solution (and in the end the most secure way against scraping to have the site login only) but it clearly will harm your visibility in search engines very badly and also new signups.
There is no single "best way". There are various ways and which is best for you depends from your goals, your needs, your technical abilities/knowledge, the time you are willing to invest, the hosting you have (shared hosting, managed server, self managed server or XF cloud) and the individual situation of your forum regarding traffic amount, traffic sources including where your genuine audience comes from.
Which ones?
The two most common ways are
1. Cloudflare
positive:- buffers traffic, so linders your load massively
- protects from DDOS
- Built in configurable WAF can block bad bots
- speeds up loading times for (many) users via CFs worldwide CDN (not relevant for the bot topic but potentially nice)
- cleanest/best solution from a software architecture perspective as comletely decoupled from your server and basically endlessly scalable
- no cost but limited ruleset in free mode (sufficient for most setups)
- paid mode offers more options (and easier/better protection) but can be expensive
- adds complexity to the setup/architecure inc. a new single point of failure that is outside your control
- configuration of the WAF rules needs constant adjustment to stay effective
- setup and configuration are somewhat complex (and can become very complex)
- can interfere with shared hosting when using things like subdomains or such via your provider's configuration backend or your provider changes mail servers after you switched to cloudflare (b/c to make use of CF you have to switch your domain's DNS to CF first)
- breaks ssl (technically unavoidable), so built in man in the middle attack by an american company - privacy considerations may or may not occur
- does linder the load but even in very well configured setups lets a relevant number of bots still reach your forum (typically like i.e. 2.000 instead of i.e. 60.000)
- as it is the most common solution in the industry CF is the primary target of bot developers that they try to find ways around for
- dependent from the individual configuration repeated captchas can be pretty annoying for real users,
- standard configuration does let pass loads of bots
=> good to bring down server load down to a healthy level, not sufficient to protect from scraping. Filtering options are plenty, based on either IP or session
2. Add Ons for XF
2.1 IP Threat Monitor
paid local "Firewall" and Rate Limiter in form of an add onpositive:
- reliable protection
- in comparison very easy to configure
- simple initial setup, yet very detailled configuration possible if desired
- built in protection for logged in users and SEO bots
- can block VPNs, traffic from servers etc. very easily
- very userfriendly analytics
- can eliminate scaping attempts almost completely with adequate configuration (if possible with your forum audience)
- very actively developed constantly
- very fast installation and initial setup w/o having to deal with DNS and other stuff
- due to the integration in XF reliable distinction between guests and logged in users, enabling dedicated treatment for each group
- pretty good real time detection through the integration with the proxycheck IP-Reputation service
o typically needs a paid account at proxycheck.io (cheap)
o finding and setting up the individual ruleset needs some time
o works best if you have a clear genuine audience and can therefor i.e. block complete countries or network blocks
o stops bots from entering your forum and scarping your content but does not protect against DDOS
negative:
- in theory prone to load issues b/c part of the XF application instead of being a separate instance
- database growth can be remarkable, depending from the configuration
- mainly IP based, so somewhat limited to simpler filter options in comparison to i.e. Cloudflare or complex individual solutions like i.e. Anubis
=> a very good solution if you want to avoid bots alltogether and probably best if your geniuine audience is somewhat geographically limited for the most part. Due to the architectural constraints possibly not optimal as stand alone solution for very huge forums. Can play nicely along with Cloudflare and Bot Guard if desired, but also as a one stop solution. Probably the best solution if you want to avoid Cloudflare and also the complexity of a howegrown unix firewall solution.
![[XTR] IP Threat Monitor](/community/data/resource_icons/10/10134.jpg?1763724168){kind=icon}
2.2. Bot Guard
free local add on to stop bots based on behavior and propertiespositive:
- finds what IP based tools cannot find
- "intelligent" bot detection w/o need for complex configuration
- very simple initial setup, very detailed configuration possible if desired
o prone to load issues with lots of traffic, should therefor not be the first barrier that requests hit - a pre filter is highly recommended
o optional "light mode" linders the load but limits the detection somewhat
o won't find every bot, but that is not the goal anyway
negative:
- data base growth can be remarkable
- sever load goes up due to individual java-script checks per session/request
=> nice additional tool against scraping, not a protection against load and it should from experience only be used in combination with another filter in front of it to limit the additional load through the add on. Plays nicely along with Cloudflare and/or IP Threat Monitor
2.3 Bot Shield
new paid add on to protect from Botso no experience yet as new to the game.
![[XenCustomize] Bot Shield: Content Protection, Crawler Management & Activity Monitoring](/community/data/resource_icons/10/10925.jpg?1792427352){kind=icon}
2.4 Human Verification
paid add on to stop bots from entering the forum+ simple add on with barely any configuration needs (klick to enter the forum)
o protection for SEO bots built in, but more than prone to be tricked out
- can become very annoying for genuine visitors
=> quick solution with very obvious limitations. Probably a good way for an instant solution with more or less one click to set up and this way to buy the time to develop something individually fitted to your needs.
Other than that there are countless best practices and options that one can make use of - from using other CDNs than Cloudflare over server based firewalls, optionally in combination with things like Anubis or Fail2Ban, filtering in Apache or Nginx based on individual rulesets, use of actual community based black-lists, use of Apache/Nginx "plug-in-rules" like "ultimate bad bot blocker" or stuff like that - options are basically endless.
What is clearly a good idea in additon to any solution is local configuration of the XF settings: A lot of load comes from guests hitting the search or what's new as both trigger database requests. So disabling the search for guests will clearly linder load issues, this is a permission within XF. There is sadly no XF setting for disabling "What's new" for guests - you can easily hide it from the menu for guests, but if they know the URL they can hit it anyway. There is a new add on that avoids disabling search for guests and promises protection of search and whats new:
![[XB] Search Spaminator: Search Bot Protection, AI Scraper Blocking & Honeypot](/community/data/resource_icons/10/10953.jpg?1792894918){kind=icon}
No experience with this - but if those pages are the cause of your issue it may be worth a try.
For protection against scraping it can be valuable to limit what guests can view at all. This will affect registrations in a positive way but search engine ranking in a negative way. Which is unavoidable, as i.e. Google has started to use content scraped for the search engine for training of their AI. So if you want to avoid that your content gets sucked out and commercialized in an AI there should be limitations. You can use XF permissions in various ways to limit what guests can see and you can use add ons for a more granular permission setting. Personally, I do use this add on:
Pretty simple but a compromise for SEO: The first post of a thread is visible, all follow-up posts are protected from guest view. SEO will still be harmed but not as harsh as a total lock out for guests.
Last edited:
Still "no", as written countless times before in this thread. It may be an option, but clearly it is no need to use CF DNS to stop 80% of the bots.
No. It is one option. If it is a good option (or even the best option) depends from the individual situation, goals and needs.
You can use CF challenges w/o using the CF WAF if you like. It is integrated in quite a bunch of other solutions as well as one possible way.
I've read this a couple of times but after four years of running my forum I am yet to see a genuine Google bot visiting my forum from Singapore. This may happen for forums and websites based in Asian countries (due to being geographically close) but probably not for i.e. European countries. I've blocked Singapore ages ago and cannot find any harm. Apart from that IP Threat Monitor (which is what I am using) offers verfication for Google (and other SE bots) through various methods like rDNS and can, if desired, this way override a country block. I do use that and still: No Google Bot from Singapore. So this statement is probably an Urban Myth or at least limited to some geograhic regions, but not generally true.
True that residential proxies do indeed come from loads of countries (but from my experience by far not from every country) - the by far single biggest source of bad bots incl. residential proxies is the US. That's what my log files say and that's what Cloudflare radar indicates, too.
CF does not distinguish between "good bots" and "bad bots", so i.e. Google and Bing are for the most part part of US bot traffic, too. But if you look at the ASNs it becomes very clear that major parts of the US bot traffic is bad traffic. It is somewhat unclear wether residential proxies are included in the CF map and measurings - I suspect not. But clearly, blocking the US as a country while letting pass SEO bots from Google, Bing and Apple will drastically lower your bot problem. If you can afford to do so b/c you don't have genuine guest traffic from the US.
On my forum (with genuine audience coming to >95% from DACH) it currently looks like that:
What comes through from the US is to 99,9% the bots from Google, Bing and Apple - basically everything else are all bad bots and scrapers.
ASN distribution looks currently like that:
Seeing a small wave building up currently:
Source is a limited set of historically well knows bad ASNs:
Those ASNs have all been blocked already long ago in my config, so I don't care. If you have not done so you might want to take a look at your logs.
Source is a limited set of historically well knows bad ASNs:
- AS46475 (Limestone Networks, Inc.)
- AS213954 (Global Transit Systems LLC)
- AS26548 (PureVoltage Hosting Inc.)
- AS43444 (Fast Servers (Pty) Ltd)
- AS59651 (Alex Largman)
Those ASNs have all been blocked already long ago in my config, so I don't care. If you have not done so you might want to take a look at your logs.
For those interested in the informational business networks behind these attacks a couple of interesting connections: "Alex Largman" (owner of AS59651) has quite a history when it comes to "bad activities on the net" and used to be called "Aleksey Bulgakov" before he changed his name:
His ASN has two peers and upstreams. One of them is AS26548 (PureVoltage Hosting Inc.), another one of the players involved into the current wave...
the other one is AS51765 "Oy Crea Hosting Solution Limited". This ASN is interestingly enough the only peer and upstream of yet another player of this wave: AS213954 (Global Transit systems), created just a year ago, - which, being a stub asn cannot transit anything anywhere.
And - what a conincidence: Yet another player of the wave, AS43444 (Fast Servers (Pty) Ltd), also has just two peers and upstreams and those are....
Which means: Four of the five ASNs involed in the current wave are deeply and densly interconnected and also connected to an Russian expat who changed his name and has a history regarding fraudulent internet activities.
Is would probably be worth checking who the people behind AS51765 (Oy crea Nova) are and whome they peer with - there is probably a pretty good chance that there are a lot of bad apples in this list...
Only limestone networks (AS46475) seems involved into the current attack, but not in the informal business network.
Interesting to see who the players are that feed the AI industry isn't it?
My residential proxies were literally every country in the world - still are - especially after I blocked China and Russia. It's only since I managed them with Cloudflare rules that it moved to the US and mainly hosting centres in the US. More expensive for them I believe. Thailand and the Philippines were particularly bad.
That's why it seems so impossible for some - you can't block every asn in the world without restricting potential memberships and real visitors (depending on which audience your forum has), so Cloudflare challenges help, because it doesn't matter which country they are from - it depends which path they are going to or what browser they are using at times.
That's why it seems so impossible for some - you can't block every asn in the world without restricting potential memberships and real visitors (depending on which audience your forum has), so Cloudflare challenges help, because it doesn't matter which country they are from - it depends which path they are going to or what browser they are using at times.
There are currently 249 of them:
List of ISO 3166 country codes - Wikipedia
While I had requests from residential proxies in many of them (including some unexpected ones) clearly not from every country in the world.
Then they are by definition no residential proxies any more as those can by definition not be in data centers.
No, what makes you believe this? Basically with any seller of proxies data center proxies are cheaper than true residential proxies. The most expensive ones are typically mobile proxies.
On a somewhat related tangent, we used to be able to run forums that had no login. But by the time 2002 rolled around, spam and flamers had increased to where we had to find a forum with at least a login required to post. Go back to the early to mid 1990s and Usenet was actually a useful discussion platform.
This isn't the Internet we started out with. Few of us ever imagined what it would turn into (in terms of the negatives).
I'll sell off the forums before I bother spending money on hardware to serve bots. I'd entertain a hardened proxy server of my own to fight them off, but hardware has gone up so much in price now, especially RAM (which Sam Altman doesn't want us to own
), that any new hardware is going to be very expensive. My hosting costs have remained stable but I know as older hardware ages out, prices are going to start inflating.Did you mean CF and not XF? If so, I agree that's the best move, and I'm probably going to have to wade through the same thing myself. I was looking again at Anubis yesterday, though, and it's just a mountain of information for me to try to learn.
For my purposes, CF is all I can deal with right now. Perfect? No. But is it doing the job I need it to? Mostly. All our sites and forums are running with no interruptions from bot swarms. It won't last, though--that's what I fear the most. CF has other features I use that have nothing to do with filtering, so it's not a total loss. But moving away from CF's filtering or better yet, using CF to filter out the most egregious traffic, and having my own bot-wall in place to catch the rest (with maybe an XF add-on to catch any remaining stragglers) is probably what I would lean towards. I prefer security and protection in layers.
I also believe CF is going to move to a pay-only service in the not too distant future. Already, features we could be using to fight bots is being pushed and/or developed for pay-only plans, especially Business and Enterprise accounts that no independent forum owners can afford. We'll get to a point where CF has new mitigations for bot traffic available only to subscribers, and those of us on free plans will either have to live with the traffic or pay up to get protection. I don't mind paying, but their minimal offerings at the Pro level aren't worth the added cost.
Cloudflare had a mitigation in place before the issue went public. I still upgraded everything, but at least I wasn't in a full-on panic because of it. Some of the sites that went offline to get patched may even have been behind Cloudflare, yet if they didn't follow the blogs or get email updates, they would have known this going in.
I.T. security is like home security. Most of it will stop most of the attempts, but if someone wants to break in bad enough, they will keep on trying until they get in. And I've said it earlier here--thanks to AI, hackers can now bang away on their keyboards trying to break, well, anything at this point.
I don't know if this is helpful at all...
www.zenrows.com
...but this is the type of thing we are up against.
My take on it is that CF's challenges are broken. Advanced scrapers know how to solve them.
How to Bypass Cloudflare when Scraping: The 8 Best Methods · Zenrows
Learn the best hands-on techniques to bypass Cloudflare Bot Management. Understand how it works, its sensor data, and evade detection.
www.zenrows.com
...but this is the type of thing we are up against.
My take on it is that CF's challenges are broken. Advanced scrapers know how to solve them.
To be fair: Society has failed by tolerating moronous behavior. If scumbags were treated like scumbags in real life we would have less issues. In the early nineties you could login to almost any host on the internet via telnet with user guest and no password. This worked for many years until first a couple of lawyers invented spam in the usenet and found this to be a good idea. It has not gotten better since. But as always: You cannot solve a social problem with technical tools. All you may achieve this way is to somewhat hedge it to a degree if you are lucky, nothing more.
Isn't it positive that back then people had a positive attitude towards mankind? Isn't it negative, that today we are forced to basically distrust everyone as a standard and that we consider that normal and consider those who ever trusted anyone to be fools? What went wrong for this to happen?
Their monopoly status is one of the reasons why I distrust Cloudflare and don't want to be dependent from them. It is a matter of time until they will become evil players. I think they will keep the free tier as they need the people on the free tier as their sensors - they are dependent from the data customers on the free tier deliver. But clearly chances are high that the free tier will sooner or later only deliver an unsufficient amount of protection and people will be forced into paid tiers.
No, XF.
That's why I have a honeypotI don't know if this is helpful at all...
How to Bypass Cloudflare when Scraping: The 8 Best Methods · Zenrows
Learn the best hands-on techniques to bypass Cloudflare Bot Management. Understand how it works, its sensor data, and evade detection.
...but this is the type of thing we are up against.
My take on it is that CF's challenges are broken. Advanced scrapers know how to solve them.
It's working well - using Cloudflare snippets. It catches the ones that solve challenges.Yes virtually every country in the world - that is how residential proxies work - I had at least 197 countries if not more - which is why at one point I had a custom rule to challenge every country except my main membership countries. Also I said "mainly" US data centres, but there are still residential proxies from the US as well. They only moved there because I had stopped the worst of it from other countries. Maybe many of the worldwide rp's originated from the US in the first place - who knows where the botmaster is.There are currently 249 of them:
List of ISO 3166 country codes - Wikipedia
en.wikipedia.org
While I had requests from residential proxies in many of them (including some unexpected ones) clearly not from every country in the world.
Then they are by definition no residential proxies any more as those can by definition not be in data centers.
No, what makes you believe this? Basically with any seller of proxies data center proxies are cheaper than true residential proxies. The most expensive ones are typically mobile proxies.
I blocked a few of the Eastern block countries, to reduce the numbers to a few thousand, custom rules did the rest. 30% still solved challenges so now I have a honeypot. And a new custom rule for fake mobiles to clean up the dregs and stop more of them taking that route.
I only have real visitors on my site now. I'm sure they'll try something else at some point but I feel I have it under control. All using Cloudflare. Although I appreciate there are other options for blocking countries etc as well.
Last edited:
Astonishing insofar as most proxy providers (those that I am aware of) claim to offer "only" between 40 and a bit more than 120 countries and as many share the same IP pools I had not expected 197 countries and even less all 249.
You still seem to take that very personally, as a personal attack. In reality they don't care about you. You have a correlation, that you believe to see but not necessarily the coincidence you claim. It may be there but it is not a given. Clearly the waves change regarding their sources more or less frequently - I could not tell what the reason for a certain change is. Also clear is, that for quite some time proxies from the US have become even more massive than before, especially the residential ones while - at least at my forum - i.e. Argentinia, Brazil, Mexico and Vietnam are gone for quite some time regarding huge waves. I've blocked all of them anyway for ages but would not claim that this would be the reason.
