Is SEMrushbot ok

[Alvin63]

I have SEMrush bot crawling my forum like mad at the moment (I signed up for a free 30 day trial with Semrush to check a few things). Is this good or bad? It seems to be searching everything (even the user title ladder). I read it could slow down bandwidth. Doesn't appear to be right now but I wondered if it could stop other bots searching or just not be a good thing generally. Any tips?

 

[Max Taxable]

Is SEMrushbot ok

Yes. Not harmful and is "noble" in that it "obeys" robots.txt

You invited it. It's gonna take a good look around at first, then only occasionally later.

 

[TPerry]

If you aren't using SEMrush, simply block it as it serves no real useful purpose if you aren't using their site stats.
As stated, they do honor robots.txt, so it's not that hard to stop it.

 

[Alvin63]

Thank you. I have noticed photos loading slower - might be coincidence but it is supposed to affect bandwidth. No I'm not really using it - it's too complicated. I had a quick look and think it's for people a bit more expert than me. So how do I block it in robots.txt? I never was sure where to find robots.txt

 

[Alvin63]

It's definitely affecting page loading/photo loading - had a few comments about that. And it's still searching like crazy! Think I'll block it - if someone could kindly explain to me how to do it and where to find robots.txt.

 

[TPerry]

So how do I block it in robots.txt?

My robots.txt

User-agent: PetalBot
User-agent: AspiegelBot
User-agent: AhrefsBot
User-agent: SemrushBot
User-agent: DotBot
User-agent: MauiBot
User-agent: MJ12bot
User-agent: SemRush
Disallow: /

User-agent: Amazonbot
Disallow: /threads/*/reply

User-agent: Twitterbot
Allow: /attachments


User-agent: *
Disallow: /find-new/
Disallow: /account/
Disallow: /goto/
Disallow: /posts/
Disallow: /login/
Disallow: /admin.php
Disallow: /lost-password/
Disallow: /misc/contact/
Disallow: /members/
Disallow: /misc/style
Disallow: /whats-new/
Disallow: /search/
Disallow: /misc/language
Disallow: /css.php*
Disallow: /online/
Disallow: /forums/rss-feeds.25/
Allow: /

 

[Max Taxable]

So how do I block it in robots.txt?

Robots.txt doesn't actually block anything. "Noble" crawlers though, should obey it.