Is SEMrushbot ok

Alvin63

Active member
I have SEMrush bot crawling my forum like mad at the moment (I signed up for a free 30 day trial with Semrush to check a few things). Is this good or bad? It seems to be searching everything (even the user title ladder). I read it could slow down bandwidth. Doesn't appear to be right now but I wondered if it could stop other bots searching or just not be a good thing generally. Any tips?
 

Max Taxable

Well-known member
Is SEMrushbot ok

Yes. Not harmful and is "noble" in that it "obeys" robots.txt

You invited it. It's gonna take a good look around at first, then only occasionally later.
 

Tracy Perry

Well-known member
If you aren't using SEMrush, simply block it as it serves no real useful purpose if you aren't using their site stats.
As stated, they do honor robots.txt, so it's not that hard to stop it.
 

Alvin63

Active member
Thank you. I have noticed photos loading slower - might be coincidence but it is supposed to affect bandwidth. No I'm not really using it - it's too complicated. I had a quick look and think it's for people a bit more expert than me. So how do I block it in robots.txt? I never was sure where to find robots.txt
 

Alvin63

Active member
It's definitely affecting page loading/photo loading - had a few comments about that. And it's still searching like crazy! Think I'll block it - if someone could kindly explain to me how to do it and where to find robots.txt.
 

Tracy Perry

Well-known member
So how do I block it in robots.txt?
My robots.txt
Code:
User-agent: PetalBot
User-agent: AspiegelBot
User-agent: AhrefsBot
User-agent: SemrushBot
User-agent: DotBot
User-agent: MauiBot
User-agent: MJ12bot
User-agent: SemRush
Disallow: /

User-agent: Amazonbot
Disallow: /threads/*/reply

User-agent: Twitterbot
Allow: /attachments


User-agent: *
Disallow: /find-new/
Disallow: /account/
Disallow: /goto/
Disallow: /posts/
Disallow: /login/
Disallow: /admin.php
Disallow: /lost-password/
Disallow: /misc/contact/
Disallow: /members/
Disallow: /misc/style
Disallow: /whats-new/
Disallow: /search/
Disallow: /misc/language
Disallow: /css.php*
Disallow: /online/
Disallow: /forums/rss-feeds.25/
Allow: /
 
Top