Is your forum being hammered by Bots?

In my experience most amazonaws.com bots are looking to harvest content or email addresses, looking for vulnerable apps, open comment forms where they can post comment spam, or something else not to your benefit. Sadly it's not uncommon for a lot of your traffic to come from bots like this. I would block them, read on being the bots may not be coming from the company itself - but could be disguised as such, and really are of no benefit to your site.
You're blocking amazonaws bots?
 
I might need to now, but since I blocked OVH my "guest" count on the forum is down which is fine because they are not real users and I think my spam problems what few got registered will go unless they get on a different subnet.

Which if you look here you can see all the IP addresses users of OVH could use. https://bgp.he.net/AS16276#_prefixes
 
Just add CloudFlare. It makes a lot of difference. My site was getting hammered by a mass of bots. Spam, scrapers, probing security weaknesses, DDoS, brute force login and just generally useless traffic consuming resources and bandwidth. All solved by adding cloudflare. I should have done that years ago as it would have saved me so much headache. And now bandwidth and SSL is free, so its actually much cheaper than without.
 
Just add CloudFlare. It makes a lot of difference. My site was getting hammered by a mass of bots. Spam, scrapers, probing security weaknesses, DDoS, brute force login and just generally useless traffic consuming resources and bandwidth. All solved by adding cloudflare. I should have done that years ago as it would have saved me so much headache. And now bandwidth and SSL is free, so its actually much cheaper than without.

What do you mean? I am with CloudFlare but haven’t seen an impact on bot activity?
 
You need to configure it properly. Its blocking 900-1200 bots a day for me.
 
Last edited:
I have it now on my forum. Seems to be working nicely. Want a bandwidth issue for me just a spam problem.
 
I wouldn't say mine was being hammered but after reading this thread, I checked my Guests. 55 with a large number having semrush or OVH IPs. Blocked those those IP ranges in cPanel and now I'm down below 20 guests with a few OVH addresses still showing that should go away eventually. I'm going to probe the remaining IPs to see if I find anything else interesting.

Useful thread. I haven't really been paying much attention to the Guests until now.
 
Can someone give me there robots.txt so I can use it lol as I don’t know how to block them

When you find a bot you want to block, you add them to your robots.txt file like so.
Code:
User-agent: SemrushBot
Disallow: /

Note that most bad bots are going to ignore it though.
 
What does this bit mean?
Disallow: /
User-agent: SemrushBot-SA
User-agent: MJ12Bot
User-agent: BLEXBot
Disallow: /

and all those before that?
 
After I re-routed through Cloudflare and blocked ovh. My forum is no longer getting 20-50 bots roaming. Cloudflare also helps caching the forum and seems to make it faster.
 
I prefer blocking at the server level. While Cloudflare may block many bots, it also increases time to first byte for page delivery.
 
On non-cached pages, GTMetrix is showing 800ms time to first byte, about 4 times higher than Google Pagespeed recommends.

There's simply no way to avoid it, non-cached pages are going to be slower using Cloudflare, because there are actually two sets of connections the data must go though both directions.

browser request --> cloudflare --> server
server response --> cloudflare --> browser
 
Then you have something wrong with your server. Cloudflare shouldn't add that much.

Test this: http://api.iconify.design/fa-home.svg?height=512 and change height to something random to make sure its not cached. Loads in about 100ms for me. Not cached, so goes through Cloudflare to my origin server.

In Cloudflare settings make sure you are connecting to origin server with HTTP, not HTTPS. That reduces page loading speed because when CF connects to your server it doesn't need to do handshake step.
 
Top Bottom