XF 2.0 Getting hit by over 80 bots on my site

Hey guys. I have had the xenforo 2 forum running great but I noticed lately I have had 80 guests on my forum nonstop for the whole weekend.
There are over 80 instances of this ip range 54.36.148.*
I have already set the range as banned in xenforo. I looked in the apache access logs and they seem to be all hitting the userprofile pages, one after the other but as the ip is blocked they get 403 errors instead.

I am just puzzled by this. Anyone got some incite what purpose this serves for anyone? Are they trying to hack into my forum? Or is it trying to harvest my data? Is there anything I could do to protect myself more?

1529330375788.webp
 
Here is the access log:
Sorry they are 303 errors, not 403. is there possibly a known vulnerability on profile post pages that they are trying to take advantage of?

View attachment 178180

You need to identify where the bots are from. I note that your log files don't show the user agent - you should look into turning that on in your web server settings so that you can see which bots are generating these requests.

You can also try installing something like my Known Bots addon which identifies some of the more common bots - this will then show up in the Current visitors list so you can clearly see which bots are doing what.

Bots aren't necessarily a problem - most of them do the right thing and you can always use a robots.txt file to block them. Most bots will honour robots.txt directives - it's only the rogue bots that you'll need to do IP blocks for. Google has a good page with lots of info: Robots.txt Specifications
 
I think it's a bad idea to allow only a few bots in - you are potentially limiting your sites visibility in various domain-specific or geo-specific search engines.

How about the following?
... these are just a few examples.

If your site only serves a small geo-specific audience, then it may not matter much - but otherwise, I wouldn't be arbitrarily blocking bots.

I'd suggest you work to identify the bots that are active on your site - and then if anything is causing problems - look to block those specific bots.

Trying to fix a problem that doesn't necessarily exist is only likely to break other things you hadn't considered - unintended consequences.

I typically have hundreds of bots on my site at any one time (275 on one of my sites right now) - but it usually has very little impact on my sites performance.
 
Thanks Sim, I did notice that the cpu hit was not significant from those 78 ip's constantly hitting me site, and even less now that they are blocked. If it gets to a point it impacts performance i will look into adding to myservers iptables to block them.
 
Thanks Sim, I did notice that the cpu hit was not significant from those 78 ip's constantly hitting me site, and even less now that they are blocked. If it gets to a point it impacts performance i will look into adding to myservers iptables to block them.

Did you identify the source of those bots? What was the user agent?
 
Well, what do you expect, Andy is spamming everyone with his countless addons trying to get as many subscribers as possible.
I am not Andy but I want to say a few words.

No offence, but you are probably the last person who should talk about "spamming" as you often cross the line and keep replying on things in almost every thread where you have no business to talk. Of course you are free to talk wherever you want and whatever you want, but so can Andy.

To his defense, he is not spamming at all. He is providing useful addons and reminding people who seek for solutions that there are addons developed by him to solve those issues. And I don't think he is doing it for the money as you imply, which would be okay too by the way, he is just getting a small revenue for his time. You can get for 25$ access to ALL of his addons, not just one. So if he would do it for money, I imagine he wouldn't choose this business model. He'd do it like others where you buy a single license for that amount of money.
And in addition to that, he provided the same amount of addons for XF1 for free.

He is providing support privately most of the time, so a huge amount of his time goes to that. I guess as a compensation and motivation, he switched from the free model to the paid one so his time is not a complete waste, as he is not obliged to provide any addons at all or support them. Again, you have access to all of his addons for that amount of money (for 1 year), which is a really fair price if you compare it with the other developers.

edit:

This is from my hosting company:

https://hetzner.co.za/help-centre/website/bot-indexing/
 
Last edited:
Care to elaborate?

Most bots are programmed to not be a strain on a site, at last it's in their own interest not to get banned. But some crappy bots can put a massive strain on a forum (I saw this with my own eyes on one of the sites I monitor).

Especially some bots which claim to be of "online reputation management" purposes.
They will target forums with so many hits, that a while ago one such particular bot was consuming 38% of our monthly traffic.
 
Top Bottom