Is your forum being hammered by Bots?

Deathstarr

Well-known member
I dont know why but the last few days my forum has been getting hammered by bots. The ones that have been really hitting me are

SEMrush - WHY
ahref - scrapper I think
OVH.

I ended up blocking OVH subnet in cpanel because 40 was to much. My forum is very local and very niche. I am not sure why they have a craze for my forum but was wondering what else I could do besides blocking in cpanel.

My robots.txt does not seem to be slowing them down or they have not re-read it yet.
 

JoyFreak

Formerly Zuse
I have no great knowledge on bots either.

But I have exactly the same bots as you leeching on my forums. I would like to understand why and where they've come from and how to prevent them? If they are bad bots or good bots and what are there benefits if good ?

I also get a lot of similar IPs: (ip88.ip-54-36-149.eu) are they real people or also bots?
I also get a lot of similar IPs: (ec2-18-222-251-156.us-east-2.compute.amazonaws.com) are they real people or bots?
 

Deathstarr

Well-known member
ovh is known spammers
ahrefs is a I do not know
SEMRush is a seo type website but the bots and IPs I checked were reported for spam.
 

Deathstarr

Well-known member
I like the idea of the plugin but with a limited working budget at the moment I will have to find ways around it at the moment.
 

Arty

Well-known member
They are not spam bots, they are bots for search engine optimisation services. Equally useless, but less harmful. Good news is they obey robots.txt

Create file robots.txt in root directory of your website (so it can be accessed as your-domain.tld/robots.txt), put this in it:
Code:
User-agent: SemrushBot
Disallow: /

User-agent: SemrushBot-SA
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: DotBot
Disallow: /

User-agent: MJ12Bot
Disallow: /

User-agent: BLEXBot
Disallow: /
 

BassMan

Well-known member
What about amazonaws.com bots? Do you block them?

I'm reading on the web and some say it's not ok to block bots from Amazon. Hm...
 

ozzy47

Well-known member
In my experience most amazonaws.com bots are looking to harvest content or email addresses, looking for vulnerable apps, open comment forms where they can post comment spam, or something else not to your benefit. Sadly it's not uncommon for a lot of your traffic to come from bots like this. I would block them, read on being the bots may not be coming from the company itself - but could be disguised as such, and really are of no benefit to your site.
 

Deathstarr

Well-known member
True but at the same time my forum is a large niche but the geographical location of the userbase is niched to 1 US State. So many of those said bots have no use to me other then bing, Google, Yahoo and facebook.
 
Top