XF 2.0 Getting hit by over 80 bots on my site

[Northern Coder]

Hey guys. I have had the xenforo 2 forum running great but I noticed lately I have had 80 guests on my forum nonstop for the whole weekend.
There are over 80 instances of this ip range 54.36.148.*
I have already set the range as banned in xenforo. I looked in the apache access logs and they seem to be all hitting the userprofile pages, one after the other but as the ip is blocked they get 403 errors instead.

I am just puzzled by this. Anyone got some incite what purpose this serves for anyone? Are they trying to hack into my forum? Or is it trying to harvest my data? Is there anything I could do to protect myself more?

 

[S Thomas]

Care to elaborate?

Most bots are programmed to not be a strain on a site, at last it's in their own interest not to get banned. But some crappy bots can put a massive strain on a forum (I saw this with my own eyes on one of the sites I monitor).

Especially some bots which claim to be of "online reputation management" purposes.
They will target forums with so many hits, that a while ago one such particular bot was consuming 38% of our monthly traffic.

Because these type of bots are typically not categorized as spiders / crawlers / robots, but as scripts / bots / unwanted traffic. That's pretty sure what Brogan was referring to. Legit crawlers won't harm your site, although they might ignore your rules.

 

[valdet]

Because these type of bots are typically not categorized as spiders / crawlers / robots, but as scripts / bots / unwanted traffic. That's pretty sure what Brogan was referring to. Legit crawlers won't harm your site, although they might ignore your rules.

They're legit crawlers and they're so aggressive in sucking up bandwidth that they will ignore the user agent disallow rules, although they claim to respect them.

Magpie Crawler is the one I am referring to
https://xenforo.com/community/threads/magpierss-robot-spider-is-pulling-7gb-a-month.123859/

 

[S Thomas]

Na, those are bad bots / unwanted traffic, there's a comprehensive list for nginx / apache here: https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker / https://github.com/mitchellkrogza/apache-ultimate-bad-bot-blocker
magpie's on that list aswell

 

[Joe Kuhn]

Blocking MauiBot today. Will leave it blocked after finding this: http://zb-block.net/zbf/showthread.php?t=101

 

[cmeinck]

I'm having a similar issue. I installed KnownBots and that's helped identify a few bots I did not want on the site. However, it appears there are bots not being identified. The biggest issue for me is the impact on Google Analytics. Whatever is hitting my site, it's inflating my traffic in GA. They are constantly hitting the What's New. They are all coming from an Ashburn, VA.