In just over a month with no protections in place, my main site has gone from 100k daily locked down, to now 700k. Slowly getting there to see just how bad this can get.
Crazy amount of guests
- Thread starter Levina
- Start date
-
- Tags
- bot swarms guests
Interesting reading from the other side of the fence (scrapers), hinting at what other battlegrounds are active these days:
One of the possible filters is the fingerprint of the TLS capabilities announced by the client, which might contradict the source (datacenter/residential) and/or user-agent. Not failproof, but can add one more defensive layer in front of the honey pot (at least temporarily before they tune it well).
One of the possible filters is the fingerprint of the TLS capabilities announced by the client, which might contradict the source (datacenter/residential) and/or user-agent. Not failproof, but can add one more defensive layer in front of the honey pot (at least temporarily before they tune it well).
Proxies are already fighting this - some charge extra so you can choose a TLS fingerprint to match the profile your scraper is publishing. Fun times!
Just dropping this in here again... when dealing with bots, my add-on is free and has worked wonders for most who try it out.
I find it ironic how some countries fight against VPNs hosted in local data centers (under the guise of websites) by checking the TLS fingerprint too. As a result, in countries with DPI filtering, perfectly legal websites don't work even within the country. AI content scrapers, government regulators… everyone is fighting against private websites
It's because many of the vpn users use the same ones as the scrapersI find it ironic how some countries fight against VPNs hosted in local data centers (under the guise of websites) by checking the TLS fingerprint too. As a result, in countries with DPI filtering, perfectly legal websites don't work even within the country. AI content scrapers, government regulators… everyone is fighting against private websites
Two days later, this is an interesting shift in behaviour. They gave up turning on/off, and just kept hitting the site.
For those into analysing treasure troves, I have attached one for you. Ran my current June 26 logs (obviously not a complete month) through a processor via maximind DB's, extracted I think about 1.4million IP's and then categorised everything roughly by actions. I'm not saying it will be perfect, but it's not bad and highlights a lot of issues from a decent dataset. As I have had all cloudflare blocking disabled for this period, this is the progressive rise happening on my main site to see what is happening without restrictions.
All in excel files, named accordingly.
All in excel files, named accordingly.
after more than a week of continuous CF "under attack" mode the number of guests on here is way lower than it used to be for ages, but probably still not all are genuine/humans:
However, this can't be the solution - UAM ist super annoying with constant captchas all the time. So I am very thankful for all the other options that have been developed and shown in this thread. It seems a bit embarrassing that XF have nothing to offer in that regard and (avoiding 3rd party add ons on their own forums) have no option for their own forum but to use UAM permanently, as it seems.
I'm sure they're competent enough to tune their server or add some resources if needed to handle it all.
Sure, but that's not the point. For one, what is true for the XF crew might not be true for XF customers and secondly: If you ask forum admins what their biggest topic and issues are the answer today would probably be:
• gain and keep new members
• protect the forum content from scrapers
Both issues have been very relevant for a long time and both have increased dramatically over the last year with AI becoming common. XF has on the other hand ignored both topics for years but instead fiddles with implementing a new editor since two years and yay's that (after years of legions of forum owners demanding it) they might possibly implement filtering of custom fields in 2.4, whenever it may see the light of day. They seem simply somewhat decoupled from reality and outpaced by miles by the progress that is happening in the world of IT. We have to be very thankful for add on devs to jump into that gap regarding the bots as what XF itself offers is far from being at the state of technology. I have not even read something where they would at least mention the problem. Simply pointing to Cloudflare as a solution does not do the trick as we see on this very forum pretty impressively.
While it's annoying users due to default settings... it's likely that it's also killing SEO for xenforo.com ( unless they whitelisted IPs of search engine crawlers, which is unlikely ).
Not good for optics.
Not good for optics.
A somewhat unusual pattern in the logs again today. Once more, the US have been a strong source of evil requests, but this time Israel made it on number 2. Israel is country blocked, so no issue from there. Proxycheck found in the US traffic quite a bunch coming from residential, most came from cox communications (which had been quite silent for months in the logs), a few also from ATT and Verizon - all not seen in a while flagged at proxycheck.
Also, an unusual pattern of cracked machines in various countries that simultaneously tried to hit the forum (the red flag means "proxy" and is very uncommon in the proxycheck dashboard in such density). These seem all to be clients, at least the providers and the DNS names suggest that.
At the same time, when this small wave hit proxycheck was once more only able to identify a fraction of them:
Which then led to Bot Guard having more work and spitting out more challenges than ever:
... and succeeding with that:
The rest of the world looks pretty grey due to country blocks in Threat Monitor for most countries that proved to be notorious for bad traffic - traffic from there never reaches Bot Guard.
So once more the the three to four step approach has proved to be working.
Also, an unusual pattern of cracked machines in various countries that simultaneously tried to hit the forum (the red flag means "proxy" and is very uncommon in the proxycheck dashboard in such density). These seem all to be clients, at least the providers and the DNS names suggest that.
At the same time, when this small wave hit proxycheck was once more only able to identify a fraction of them:
Which then led to Bot Guard having more work and spitting out more challenges than ever:
... and succeeding with that:
The rest of the world looks pretty grey due to country blocks in Threat Monitor for most countries that proved to be notorious for bad traffic - traffic from there never reaches Bot Guard.
So once more the the three to four step approach has proved to be working.
Last edited:
Yeah.. they are rotating through IP pools aren't they.
Are you using Bot Guard and another plugin that uses proxycheck?
Are you using Bot Guard and another plugin that uses proxycheck?
Is it just me with the feeling that, with the major search engines no longer bringing traffic to your site and just feeding their AI bots, we're witnessing the death of SEO era and the start of SBO (Scraper Bot Opposition) era?
In order of appearance:
1. .htaccess
- filters vulnerability scanners and bogus url-scanners via RegEx on the requested URL, partly based on / inspired by this approach
- filters named crawlers that I dislike based on submitted user-agent
- filters a couple of hostnames and a small set of IPs (leftover from earlier setup, could be done elsewhere)
2. IP Threat Manager add on
- blocks IPs based on the blacklist autocreated over time
- blocks countries (currently ~90) based on configuration (check against local maxmind db)
- blocks ASNs (currently ~550) based on configuration via API query to proxycheck.io
- blocks or allows specific VPNs based on configuration (check against local lists)
- blocks IPs and IP Ranges based on configuration (a very small handful)
- blocks IPs/Hosts based on reputation via API check against proxycheck.io
- rate limits and later blocks IPs that misbehave
3. Bot Guard Add on
- checks requesting client via JS check
- provides Turnstile Captcha to suspicious clients based on scoring
Last edited:
SEO died 20 odd years ago, the moment Google released localisation and adwords. You could no longer optimise your website for ranking outside what Google deemed your website zone, and then, even within the zone which you have no idea how far it spans, basic SEO still has no real metric difference. You can make two websites today, creates links to one, optimise every aspect of it as you believe works, then build another and do nothing, just make the website the best reflection of the business for the customer, and its not uncommon that the untouched one outperforms the optimised one.
Regardless what people think today, with the discussion in this very thread, people blocking everything AI, the difference in my sites daily posting is showing you should not be blocking them, hence I went the highly optimised server route. When I blocked and strangled the AI internet, my daily posting fell slowly from around 125 posts to 50 relying on search engines only, and when unleashed like it is right now, daily posts are about 150+ and climbing steadily.
I use AI a lot for search. If you aren't being found in AI for search because they can't access your site, then you're likely missing out on a lot of potential new traffic and members.
Just my experience playing around with all this, and I used to be one of the top SEO's 20+ years ago, basically I was one of the people you could blame when everything returned porn or similar as a listing. Retired after Google took over and stopped all the nonsense.
@smallwheels
Thanks for the info, this sounds like 75% of what i'm designing.
Imagine you're getting pretty effective protection from these layers, no?
Thanks for the info, this sounds like 75% of what i'm designing.
Imagine you're getting pretty effective protection from these layers, no?
The problem IMO is not AI search / user triggered AI bots. For instance, my add-on, actually whitelists these by default. The problem is the AI scrapers that are for model training. These need to be blocked.
Bots are not inherently bad. There just needs to be a reasonable trade.
In principe: Yes. But it depends. Through the multi-stage approach it is somewhat difficult to get proper data, the more as IP Threat Monitor currently only has very limited possibilities when it comes to analysis and the .htaccess outcome is a bit of a pain to analyze. Bot Guard does in my configuration only count what has been challenged and what was analyzed but not systematically a simple pass / checked / failed number. I could have it but then the db size would further increase and bot Guard is already somewhat db heavy in terms of size. Also, Bot Guard refers to sessions whereas Threat Monitor acts based on IPs, so not trivial to aggregate that.
What I can say is:
• Thread Monitor does block between 35% and 90% of the incoming IPs. On average days it blocks between 40 and 60%, the 30 day average is currently at 85%
• Bot Guard does not block per se but rather many of the bad clients give up along the way of the checks or the captcha.
• I have no data how many fail/give up during the JS checks but do know that the vast majority of those getting the captcha die silently:
• So it is obvious, that despite all efforts there is quite a bit of bad traffic that passes IP Threat Monitor. This is mainly traffic from resident proxies, based on private internet access ranges within Germany (and a bit from other European countries). This is where Bot Guard comes to the rescue.
• given what numbers other small forums report and what numbers I see when visiting other forums regarding guest traffic I assume that my forum has fallen off the radar of some of the bigger scrapers or is disregarded as a target, either b/c I started early with aggressive blocking or b/c they may send a kind of "scout bots" in advance of a wave or have a kind of "bot master" both of which may accidentally be blocked through my excessive proactive blocking already so that they don't even start to scrape at the scale that other forums see. Possibly the "only first post of a thread is visible to guests" may help as the scrapers that manage to pass the protection run into a lot of 403s and other limitations.
So over all I'd say my setup does work pretty well for the moment and the price to pay is worth it. It was a lot of work and a lot of time and effort needed to get there. Yet, there is room for improvement:
• if ASN blocking in Threat Manager would be done via local MaxMind DBs rather than via API-calls to proxycheck this would save loads of API calls and therefore scale better / safe on cost on a bigger forum. BTW: 90% of the server based bad traffic does come from the same ASNs and this set is not too big. It contains of the cheaper cloud Hosters that ignore issues like Digital Ocean, Clouvider, Contabo, and a hand full of others and on the other hand a relatively small set of ASNs with international ranges that seem for the most part to be bad to the bone on purpose. There are outliers here and there, but it is Pareto: you can hit 80% with 20% effort.
• probably a more sophisticated behavioral tracking could be of interest to better identify bad actors. Bot Guard has done a lot of positive in regards of identifying baddies based on their properties, but a behavioural risk analysis, fully integrated into XF, learning from the patterns of the individual forum and it's visitors and members would be another level (but be vastly complicated and possibly only feasible by XF itself).
• to improve my setup better analysis would be mandatory - at the moment I am using the sledge hammer quite a lot and for the moment it works - however: better analysis would enable more learning which would then enable a more targeted approach
• the ability to easily combine the logs/analysis of the different stages would be awesome
• by design I lack DDOS protection. Still don't want cloudflare but it would be good to move the blocking as far out from the application as possible, so possibly longer term a dedicated firewall may be on the bucket list
Last edited:
