RandallC
Well-known member
https://xenforo.com/community/threads/is-your-forum-being-hammered-by-bots.156431/ old but still relevant.
How to block Ahrefs, Semrush, Serpstat, Majestic SEO by htaccess or any method far away robots.txt
- Thread starter jamalfree
- Start date
Not quite hammered by bots but that ruleset I posted for cloudflare blocked 1.88k attempted visits by ahrefs alone in 24 hours.
This at least helps webmasters to save on server resources: Bandwith, cpu, i/o ram usage etc. I don't really see the use in either other than to be a statistic for another competitor. As far as I can tell, they won't help me rank any better with google, bing or yandex... unless I pay, which even then they probably won't help much. So yeah it's something I don't want or as far as I can tell need.
This at least helps webmasters to save on server resources: Bandwith, cpu, i/o ram usage etc. I don't really see the use in either other than to be a statistic for another competitor. As far as I can tell, they won't help me rank any better with google, bing or yandex... unless I pay, which even then they probably won't help much. So yeah it's something I don't want or as far as I can tell need.
That's a bit overkill for a list. Also if you block them with cloudflare it works better as you use their resources to block the bot instead of the bots using your stuff to initiate the block. The general rule of thumb for a .htaccess is, the less in it, the better. You just gave us a pretty good sample of what not to do to a .htaccess.
General well known tips for optimizing .htaccess performance:
General well known tips for optimizing .htaccess performance:
- Minimize the number of directives: Only include essential configurations in your .htaccess file.
- Consider server configuration: If you have access to the main server configuration file ( httpd.conf ), it's generally recommended to place directives there for better performance, as it is parsed less frequently.
- Combine directives: When possible, combine multiple directives into a single line to reduce parsing overhead.
Last edited:
RandallC
Well-known member
Can you give us some research on your proclaimed less is more just due to the fact that it was built (robot.txt) to do what the webmaster needs. The example given was looks to be a little long and could be combined. However adding Cloudflare in front can also be bad due to the extra hops it takes this slowing down your side until the request reaches your server from
Cloudflare.
I’m all for help which is why I posted the older thread linking where it was discussed that certain bots do nothing for websites. But between a well written robots.txt, blocking countries that you do not target as well as cloudflare most basic bot detection and challenges that will eliminate most bots.
Do you mind elaborating on what you mean with the extra hops from cloudflare? I've tried blocking some with robots.txt (ahrefs & semrush) and their response is to send 20-40 at a time to bypass it, however blocking them via cloudflare, zaps them before they even touch my server.
RandallC
Well-known member
If you are using cloudflare, your traffic is passing through the cloudflare servers before it reaches your server. I am just trying to see the slower difference in a robots.txt or running cloudflare. I agree that a well written robots.txt is better but even a poorly one is t slow,slow but it depends on what your trying to achieve faster servers or robots not hammering your site. Or somewhere in the middle.
Ok let me put this another way.. If I am blocking, as per my attachment on page one, are you trying to tell me that the more I add to that list, the more hops I'd get? I do get that blocking it in 2+ ways would cause issues but I want you to elaborate. Because I'm not really understanding what you're trying to say, that I suggested to do would add more hops.
As for my robots.txt I keep it pretty clean.
As for my robots.txt I keep it pretty clean.
Code:
Sitemap: https://www.vgcheat.com/sitemap.php
User-agent: rogerbot
Crawl-Delay: 3
User-agent: dotbot
Crawl-Delay: 3
User-agent: SiteAuditBot
Crawl-Delay: 3
User-agent: Amazonbot
Disallow: /threads/*/reply
User-agent: *
Disallow: /whats-new/
Disallow: /account/
Disallow: /attachments/
Disallow: /goto/
Disallow: /login/
Disallow: /search/
Disallow: /admin.php
Disallow: /misc/*
Allow: /
Last edited:
