Is your forum being hammered by Bots?

[RandallC]

I dont know why but the last few days my forum has been getting hammered by bots. The ones that have been really hitting me are

SEMrush - WHY
ahref - scrapper I think
OVH.

I ended up blocking OVH subnet in cpanel because 40 was to much. My forum is very local and very niche. I am not sure why they have a craze for my forum but was wondering what else I could do besides blocking in cpanel.

My robots.txt does not seem to be slowing them down or they have not re-read it yet.

 

[JoyFreak]

I have no great knowledge on bots either.

But I have exactly the same bots as you leeching on my forums. I would like to understand why and where they've come from and how to prevent them? If they are bad bots or good bots and what are there benefits if good ?

I also get a lot of similar IPs: (ip88.ip-54-36-149.eu) are they real people or also bots?
I also get a lot of similar IPs: (ec2-18-222-251-156.us-east-2.compute.amazonaws.com) are they real people or bots?

 

[RandallC]

ovh is known spammers
ahrefs is a I do not know
SEMRush is a seo type website but the bots and IPs I checked were reported for spam.

 

[AndyB]

You might like this add-on:

https://xenforo.com/community/resources/robots.5623/

 

[RandallC]

I like the idea of the plugin but with a limited working budget at the moment I will have to find ways around it at the moment.

 

[JoyFreak]

I like the idea of the plugin but with a limited working budget at the moment I will have to find ways around it at the moment.

That $35 will give you access to all his add-ons for a year and he has like 100s lol. Some of which are beneficial.

 

[RandallC]

At the moment I am still on a limited working budget.

 

[Arty]

They are not spam bots, they are bots for search engine optimisation services. Equally useless, but less harmful. Good news is they obey robots.txt

Create file robots.txt in root directory of your website (so it can be accessed as your-domain.tld/robots.txt), put this in it:

User-agent: SemrushBot
Disallow: /

User-agent: SemrushBot-SA
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: DotBot
Disallow: /

User-agent: MJ12Bot
Disallow: /

User-agent: BLEXBot
Disallow: /

 

[RandallC]

I already had some of those but added the ones I did not.

 

[SyTry]

I also get a lot of similar IPs: (ip88.ip-54-36-149.eu) are they real people or also bots?

Same for me but they are guests

 

[RandallC]

Thats OVH

IP Address Lookup for 54.36.149.35 in , France

Location: France - 54.36.149.35 is a likley static assigned Corporate IP address allocated to OVH SAS. Learn more.

whatismyipaddress.com

They are a server provider with some relaxed rules. Many times they are proxys for spammers.

 

[SyTry]

Thats OVH

https://whatismyipaddress.com/ip/54.36.149.35

They are a server provider with some relaxed rules. Many times they are proxys for spammers.

Yes , I blocked the IP by .htaccess

 

[RandallC]

I blocked the entire subnet 54.36.0.1 - 54.36.255.255 or something like that in cpanel.

 

[BassMan]

What about amazonaws.com bots? Do you block them?

I'm reading on the web and some say it's not ok to block bots from Amazon. Hm...

 

[RandallC]

i have not had a run in with them yet.

 

[BassMan]

Well, me neither, but I can see they are constantly crawling my forums.

 

[RandallC]

Thats what I mean they have not been on my forum as far as i know. I have not seem them atleast.

 

[BassMan]

Well, that answer from @Sim is useful: https://xenforo.com/community/threads/getting-hit-by-over-80-bots-on-my-site.149005/post-1261410

 

[Ozzy47]

In my experience most amazonaws.com bots are looking to harvest content or email addresses, looking for vulnerable apps, open comment forms where they can post comment spam, or something else not to your benefit. Sadly it's not uncommon for a lot of your traffic to come from bots like this. I would block them, read on being the bots may not be coming from the company itself - but could be disguised as such, and really are of no benefit to your site.

 

[RandallC]

True but at the same time my forum is a large niche but the geographical location of the userbase is niched to 1 US State. So many of those said bots have no use to me other then bing, Google, Yahoo and facebook.