Is your forum being hammered by Bots?

[Brad Padgett]

So I would like to add my input to this thread. After having about 80 bots constantly being on my site at all time, I set out to see what I could do about it. I really did not want to use my robots.txt file since some bad bots will ignore it anyways. After clicking on my 'Current members' tab and checking out who was visiting most were coming from OVH network. Now I can't find a user agent for this so I had to figure out what to do. Since I use Cloudflare I went to Firewall then to IP access Rules selected FR for France for the country (because that is where the OVH bots comes from) and selected 'Challenge', just in case someone from France wants to visit my site it won't totally block them. Once applied in a few minutes they started dropping like flies. But I was left with another 40 bots of Semrush. So it was time to take care of them as well. I went under Cloudflare once again and Firewall. This time I created a Firewall rule and called it "Bad Bots". Then added the bad bots I wanted to get rid of like in the image posted below. Low and behold I killed them all. You can add other bad bots as well. I hope this helps someone in the future with the same problem.

View attachment 195219

Thank you for the informative post. That really helped a lot. I just added those rules in with Cloudflare. Haven't had time to see how well it works but I can assume it will work well. I mainly have an issue with AmazonAWS so I added that as a contained user agent and hopefully that will rid us of those bots as well as the others you listed such as Ahrefs which is another bot that is on a lot of peoples sites. +Like thank you

 

[Arty]

View attachment 195219

Bots listed on your screenshot are actually not bad. They are useless because all they do is gather SEO statistics that are no benefit to you, but they aren't misbehaving. They are following robots.txt. So all you need to do is block them in robots.txt instead of firewall.

This is from my robots.txt that blocks those bots and they do follow it:

User-agent: SemrushBot
Disallow: /

User-agent: SemrushBot-SA
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: DotBot
Disallow: /

User-agent: MJ12Bot
Disallow: /

User-agent: BLEXBot
Disallow: /

User-agent: DomainStatsBot
Disallow: /

User-agent: ZoomSpider
Disallow: /

User-agent: MauiBot
Disallow: /

User-agent: MegaIndex.ru
Disallow: /

User-agent: megaindex.com
Disallow: /

User-agent: ltx71 - (http://ltx71.com/)
Disallow: /

 

[nodle]

Bots listed on your screenshot are actually not bad. They are useless because all they do is gather SEO statistics that are no benefit to you, but they aren't misbehaving. They are following robots.txt. So all you need to do is block them in robots.txt instead of firewall.

It depends on what you consider "bad". Yes I know they are mostly SEO bots, but why should I have them consistently crawling my site, using my resources and bandwidth? All for what? Some unknown search engine in a foreign country? Like @AndyB said once, there are only a few "good" bots that really matter Google, Bing, etc. I at first tried to block Semrush using my robots, even following their guide here. But it only stopped about 4 of them. There are a lot of bots out there that basically just don't care what you put in your robots.txt and will not obey it. That is where Cloudflare comes in and stops it at the front door. If you use Cloudflare @Arty just try what I posted above and see what you think. I bet you find it works better and is easy to add.

 

[Kirby]

You don't need over-hyped CloudFlare to block requests by User-Agent, this can be done with with Apache (.htaccess), nginx, etc. as well.
For example we block them at our loadbalancer (HAProxy).

 

[nodle]

You don't need over-hyped CloudFlare to block requests by User-Agent, this can be done with with Apache (.htaccess), nginx, etc. as well.
For example we block them at our loadbalancer (HAProxy).

Yes I know it can be done using .htaccess and other ways, but why put strain on your server?

 

[Kirby]

Unless your audience is geographically distributed, I'd argue that even .htaccess (which does not have to be used if you have proper server access) is faster than using CloudFlare

 

[DigNap15]

What did you do on Cloudfare to avoid the bots ?

I have a Brazilian Forum (0 english content) but look at the Cloudfare's report. I know that there are a bunch of google/bing records but, well..

View attachment 188980

I am not using Cloudfare yet, but I am sick of so many Bots, mostly Facebooks
Like you I am just one country orientated (New Zealand)
Yet my Statcounter analysis shows that 30% of my traffic is from the USA - I know that is not correct
I know it 99% from New Zealand

 

[DigNap15]

I may have just had a re-think!
Like many of you I was getting annoyed with the Facebook Bots
Google and Bing I don't mind, as they keep my forum at the top of Search!
Some of my members have noticed the Facebook bots, and mentioed them as a negative or put off for them (being spied on etc)
So I had a look at a few of them.
They point to specific threads on my forum, I am thinking that they are the threads where one of my members has pressed the Share With button!
If that is correct, then surely it is a good thing, as I may get more members as a result!

 

[Deathstarr]

Normally yes if it’s shared on Facebook the. Bots will show up but only for a bit.
I highly recommend Facebook block plugins for browsers. I have one on my laptop that when I’m surfing the web I don’t get ads related to it on Facebook.

 

[Max Taxable]

I am thinking that they are the threads where one of my members has pressed the Share With button!

Correct. FB and Twitterbots have to visit your site, to fetch your content posted on FB and Twitter. And every other site does this too including this very one. It's a good thing.