but if they are residential proxies, as soon as you block some they will just reappear from another country. It's known as whack-a-mole.
It is not as simple as that. Clearly, it is easy to get rid of quite a lot of the annoying bots by blocking nasty countries completely if you do not have relevant genuine traffic from there and blocking nasty ASNs completely. This leaves you with a hand full of countries that you don't want to block and, within these countries, loads of ASNs and IP ranges. The proxycheck dashboard helps to identify nasty ASNs within these countries that are server ranges and can safely be blocked. Which then leaves you with the "true" residential proxies that hide within dialup ip ranges on ordinary client devices that you cannot/ want not to block flat out.
It is important to be aware that "residential proxy" describes a location in the first hand (and a routing/dispatching setup behind it), not a browser or such. While some providers of those proxies seem to offer more or less sophisticated custom clients others seem just to offer the infrastructure. And scrapers can and will use a range of different technologies and setups for scraping independent from the route they are taking IP-wise. The stupid ones are just scripts that can easily be detected and blocked through fingerprinting (by either cloud flare or bot guard) and they are not capable of solving captchas. Those who use a genuine real browser via things like selenium are not trivial to identify but those don't scale very well, take a lot of resources in comparison and need constant maintenance. So the real issue are the ones that use a more intelligent scripted approach, hide within client dialup IP ranges, escape fingerprinting by successfully pretending to be a genuine user / browser and are capable of solving captchas. Basically all sellers of residential proxies claim they would have this on offer - how many really do is unclear.
So in the end with a growing amount of bot requests you are doomed to use kind of a probabilistic approach to decide if it is a human or a bot and depending from where and how you place your bets you will either block a certain amounts of humans unwillingly or let pass a certain amount of more sophisticated bots unwillingly. This is the current state of technology and the only way out would be to lock your forum content that only logged in users are able to see it - which in most cases is not desirable, at least not in this radical fashion.
This is why I advocate for a combined approach: The more sophisticated scraping approaches are not exclusive to proxies on client networks but also used from datacenter ranges and detection via fingerprinting will fail there as often. So blocking hard by country/ASN/IP-Range what can easily be blocked w/o or with minimal collateral damage helps a lot. Using fingerprinting (with i.e. Bot Guard), captchas and other things like the services of proxycheck helps against a relevant part of the others. Limiting options, search or what content can be viewed for guests helps to hedge the damage the rest can do in terms of performance or scraping valuable content (with the trade off that it will influence your Google ranking when done harshly but effectively).
However: I have gone a very harsh approach regarding blocking and limiting guest visibility and have paid a price in terms of Google ranking. But: The amount of bots on my forum has come down to a minimum and the damage they can do is minimal as well. W/o using Cloudflare or sophisticated technology. Cloudflare and their captchas are the industry standard and this means: It is worth for bots to focus on dealing with Cloudflare and to circumvent their captchas and other protection mechanisms. Basically a rat race: Sometimes Cloudflare is in front, sometimes the bots are. Using something different than Cloudflare might therefore bring an advantage.
And there's another aspect: The bot armies have to have a central point that sets the directions. From somewhere they have to get the directions what they shall scrape. I would assume that gathering this information happens well before a bot wave starts and that it typically does not happen through residential proxies but rather from some server ranges. I don't know how they do it (i.e. by collecting sitemaps, rss-feeds or such) but I clearly see, that after some time of massive hard blocking of loads of evil ASNs the bots seem somehow to have lost interest in my forum. Either the upfront collection of what to scrape has been cut or they have decided that the very limited content they can reach at all is not interesting enough for all the trouble.
Over the last months there were only a small couple of peaks (would not even call them waves) aside of the usual noise. This is my 90 day proxycheck graph, leaving out those that have already been blocked by country blocks or other simple blocks via regex in .htaccess):
Based on those spikes of the green curve you can also clearly see, that proxycheck is not really good when it comes to identifying resident proxies from dialup client ip-addresses.
