Learnings: Identifying and getting rid of unwanted traffic

[ENF]

Why? I don't have an issue with spam oder spammer's signing up successfully on my forum. These kind of annoyances is handled 100% successfully by the spaminator series of @Ozzy47 on my forums, as I lined out earlier. So what exactly would Xon's add on be good for in the topic of this thread?

To be fair, it's our primary tool to filter out various ASN's upon registration. We found that ASN blocks are not 100% in some cases where data is (or is not) updated at the IP level. However, using the ASN configuration in Xon's Addon, we catch 100% of the ASN's we have on our absolutely-do-not-want list. We update data as we find holes, but Xon's tool(s) are the icing on the cake to prevent anyone from registering from a blocked ASN. Anyway, we elected to add his tool to the arsenal because it employed ASN-level exams and rules.

We have blocks at the OS/Network Layer, Border (ingest) & App layers (xf and other business apps where needed). It's basically set at the level of severity. The major issue we came across was people who were traveling and using temp sim or esims... these nonsensical providers route those temp cards through various dirty connections on some of the blocked ASN's. (I know, I got myself blocked by using a temp sim while in Australia and a few other countries...)

 

[smallwheels]

Xon's tool(s) are the icing on the cake

Absolutely, at many occasions.

Anyway, we elected to add his tool to the arsenal because it employed ASN-level exams and rules.

I may do that as well, but only later and as a bonus upon the multiaccount feature that I'm somewhat after.

The major issue we came across was people who were traveling and using temp sim or esims...

Traveling forum members are the main reason why I am not even more rigid in blocking areas.

 

[smallwheels]

I may do that as well, but only later and as a bonus upon the multiaccount feature tha

Well, later was early. I got curious and bought and installed the add on. Impressive. Looks like a swiss army knife.

 

[smallwheels]

Here's a list of the top ASNs that were blocked by my firewall settings - the vast majority of this blocked traffic did indeed come from that Microsoft Datacenter ASN:

As I dive deeper and deeper into the unknown the more noise has been blocked successfully more and more of those come to the surface thad managed to hide successfully until now. Being aware of your screenshot I was curious if I would find the same players - and I did for the most part: There is a little bit of Hetzner in my logs, but not much. However, Huawei Cloud is a big and very nasty one. Someone from there seems to slowly but steadily suck down my forum contents, using a huge range of different IPs and as huge set of different useragents, claiming to be normal browsers:


Whoever does this is not in a hurry and rather tries to hide himself by acting very slowly, distributed and hiding the fact that a bot is used. So next job will be to block out the Huawei cloud IPs at scale as good as possible - sounds like a lot of manual work (the more, as the whois database is inaccurate and missing to display wider parts of these IPs correctly. Arin says they would have been delegated to ripe and ripe says "we don't know a thing", so it is manual work involved as bigger parts of the IP ranges have fallen in kind of a black hole.

 

[smallwheels]

Time for the next update.

There is a little bit of Hetzner in my logs, but not much. However, Huawei Cloud is a big and very nasty one.

While Huawei keeps being a steady annoyance the ongoing blocking efforts clearly show results - less and less requests come through. However: Hetzner has changed dramtically. There has been a continuous stream of requests coming from their cloud, but to a somewhat limited amount.
On Monday and Tuesday this week this changed: I discovered a massive scraping attempt with around 35.000 requests on Monday and 105.000 requests on Tuesday coming from Hetzner Cloud, using more than 480 different IP addresses from a huge amount of ranges and hiding behind an immense amount of different generic user agents.

As a result, a huge range of IPs from Hetzner have now been blocked - the attempts are continuing, but slower and unsuccessful. To be able to handle that effectively and more efficiently I was forced to revitalize my shell scripting abilities - sed, awk and regex have been pulled back to life from the darker and hidden areas of my brain where forgotten knowledge is preserved. A very special area of fun. In the end I have now come one step closer to a semiautomatic workflow for blocking bad actors.

Out of curiosity and due to the massive amount of malice traffic coming from a provider of my own country I also contacted the abuse desk of Hetzner to see if and how they will react - atm I am still waiting for an answer.

Overall, my manual blocking approach is clearly way too time consuming to follow it forever. For the moment it is still interesting enough in terms of learning to continue with it.

 

[thumped]

Lessons.

 

[smallwheels]

Out of curiosity and due to the massive amount of malice traffic coming from a provider of my own country I also contacted the abuse desk of Hetzner to see if and how they will react - atm I am still waiting for an answer.

Update on this: I got a reply a couple of days later - it was not a DOS alike incident, so while not brilliant this is ok. Unfortunately, the content was just the opposite. Despite having been provided with all necessary information including the logfiles in txt-format they asked me to provide the logfiles - interestingly enough they did event send them back to me as a quote in their answer, as whatever tool they may use is utterly broken - the whole quoting inside their mail was a total mess. They asked for the logs to be provided not as attachment but as txt inline in the mail - a bit of a weird approach given that we are talking about 46MB of txt-logfiles of the requests of their customer which, zipped down are just 3MB. So an abuse desk that cannot deal with txt files and is unable to send proper mails and unable to use proper quoting. Interesting.

They did not read my abuse report properly as they were stating I had complained about one IP (whereas I had in fact complained about a distibuted scraping attempt from more than 480 IPs from their network) and were not willing to tell me the identity of their customer for reasons of data protection laws (clearly an artificially made up statement w/o foundation, given the evidence I had sent them) but were asking if it was ok if they would give my identity and my logfiles to their customer. Else they would not be able to do anything about the incident they said.

So basically a somewhat disturbing answer and in fact the most unprofessional answer I ever got from an abuse desk in decades. I replied to them but did not get any answer back.

As a consequence, while I do not consider Hetzner to be a rogue provider or a black-hat provider they are at least a grey hat, given their behaviour. Probably not purposefully but due to broken processes, a bad attitude and incompetent staff. Their abuse desk acts unprofessionally and irresponsibly, contacting them seems simply a waste of time. Blocking Hetzner's networks completely from accessing your forums is faster and easier and more effective on top of that - and well deserved.

The scraping attempts from their networks continue, but to a smaller, slowed down extent. I don't care as they are provided with a 403 now.