XF Bot Guard

XF Bot Guard 1.4.1

No permission to download
Hi,
First of all great add-on which has really helped with the bots and my site crashing.
One question though, I have an API key so the resources on my site had be added automatically to a website.

I had this message from the person which managed the site via the API key - Is there anyway around this?
When you look in the settings of the add on ($yourforum/admin.php?options/groups/zeeBotGuard/) you'll find, that there is a whitelist for IPs that you can use (under "Excluded IPs/CIDRs/ASNs") as well as the option for trusted reverse dns ("Additional trusted crawler reverse DNS domains") as well as the option to exclude rss feeds (Exclude RSS/Atom feeds from protected areas) as well as the option to exclude certain paths (which could be your API) under (Excluded paths and routes).

If you did not realize any of that you obviously did not look into the settings of the add on and neither in the FAQ for the resource / add on nor in the (pretty excellent) documentation of the add on. All of which was created to avoid trouble and unneccesary support requests for a free add on. Which, as a side note, leaves the question if you really should run an add on of the complexity of Bot Guard if you do not in the slightest care what it does and what it's options and boundaries are...

The Faq states:

Will it break login, registration, APIs, webhooks, or payment callbacks?​

Default exclusions are designed to avoid that.
Admin, login, register, API, webhook-style paths, CAPTCHA, static assets, robots.txt, sitemap-style paths, and similar sensitive routes are excluded by default. Custom third-party endpoints should still be reviewed and explicitly excluded when needed.

As you say

I have an API key so the resources on my site had be added automatically to a website.
but do not bother to say what kind of API you are using it is pretty impossible to answer to your issue in a useful way anyway. I'd recommend going through the options and docs listed above - it seems that there are plenty of options to solve your issue.
 
Hi,
First of all great add-on which has really helped with the bots and my site crashing.
One question though, I have an API key so the resources on my site had be added automatically to a website.

I had this message from the person which managed the site via the API key - Is there anyway around this?
You can whitelist their IP, I did this for search engines and some other stuff.
 
For the past few days we've been getting hit with waves of guest traffic that is crippling the server and resulting in 502 and 504 errors.

We'll get tens of thousands of guests that all seem to be stuck on the challenge page but never end up in the bot count.

1794348148458.webp

Are there any adjustments we can make to help with this?
 
We have been having the same issue for months. When it hits about 75,000 guests, the sql server was crashing. XF Botguard and Cloudflare combined worked as a good solution for us. We had 30k yesterday, so I bumped up the security level a bit for 4 hours, and they gave up. I think this is probably the new normal.

I have 2 Xenforo forums on the same server. One is a magnet for the bots, the other is not.
 
Thanks for reporting this.

XF Bot Guard is only trying to prevent caching where it needs to: Bot Guard’s own validation endpoints and protected public HTML responses. It is not intended to broadly disable caching for unrelated assets or unprotected responses.

The nginx warning appears to be caused by the same Expires: 0 cache-safety header being sent twice on some protected responses. Because both values are identical, this should not change the actual browser/proxy behaviour, so it is not a breaking issue. That said, the warning is avoidable, and I’ll tidy this up so the cache-shield headers are applied idempotently and each header is only sent once.
I installed add-on yesterday and it seams that this problem still making error log very big.
 
We'll get tens of thousands of guests that all seem to be stuck on the challenge page but never end up in the bot count.
A friend of mine with his XF forum has the same issue. Still attacked by numerous bots, well... in fact no real bots but a bot network mostly consisting out of older Chrome versions (now even up tot version 151).
Still this slows down things. Already even with heavy .htaccess ip range blocks, so we are wondering if we can do more.

At this moment using Cloudflare is not an option.
 
For the past few days we've been getting hit with waves of guest traffic that is crippling the server and resulting in 502 and 504 errors.

We'll get tens of thousands of guests that all seem to be stuck on the challenge page but never end up in the bot count.

View attachment 350222

Are there any adjustments we can make to help with this?
I have similar bot issues but I don't think they're keeping the connection alive by sitting there, based on my keepalive_timeout and other nginx settings. I find lightDb mode seems to be better at not writing as much to logs which helped.
 
I have similar bot issues but I don't think they're keeping the connection alive by sitting there, based on my keepalive_timeout and other nginx settings. I find lightDb mode seems to be better at not writing as much to logs which helped.
We've been using lightDB mode as well, but the bot attacks were still slowing things down when there were enough of them. A few days ago we took our site offline for a few hours to upgrade our database from MySQL to MariaDB. While we were down, the bots must have given up and moved on to some other site because the number of guests was back to normal when we came back online.
 
For the past few days we've been getting hit with waves of guest traffic that is crippling the server and resulting in 502 and 504 errors.

We'll get tens of thousands of guests that all seem to be stuck on the challenge page but never end up in the bot count.

View attachment 350222

Are there any adjustments we can make to help with this?
Are they stuck in the challenge or during the interstitial process? In case of the latter it seems to work as designed.
When it hits about 75,000 guests, the sql server was crashing. XF Botguard and Cloudflare combined worked as a good solution for us.
I installed add-on yesterday and it seams that this problem still making error log very big.
Still this slows down things. Already even with heavy .htaccess ip range blocks, so we are wondering if we can do more.

At this moment using Cloudflare is not an option.
I find lightDb mode seems to be better at not writing as much to logs which helped.
We've been using lightDB mode as well, but the bot attacks were still slowing things down when there were enough of them.
In general I would recommend not to put Bot Guard directly in front but rather put some other filter in front of it. Rather use Bot Guard as second or third line of defense, as it consumes more resources for an JS based IP check than a simple IP-based block would. Extensive logging makes the problem worse. Light mode was developed to improve the situation here, still, conceptionally a pre-filter seems the way better option. .htaccess filtering goes into that direction but becomes impractical to maintain if you want to block a lot of ASNs or a country and lack i.e. mod_ASN or alike in Apache. Depending from the situation this may give some options:

 
Are they stuck in the challenge or during the interstitial process? In case of the latter it seems to work as designed.
Most guests listed under Current Visitors appear to be stuck on the browser validation.

In general I would recommend not to put Bot Guard directly in front but rather put some other filter in front of it. Rather use Bot Guard as second or third line of defense, as it consumes more resources for an JS based IP check than a simple IP-based block would.
We are considering setting up Anubis as the 1st line in front of Bot Guard.
 
We are considering setting up Anubis as the 1st line in front of Bot Guard.
That might help a lot. We have some phpBB board which were under attack and somebody wrote a AnubisBB extension so it could be used as extension for phpBB and this had great results.
Would be nice to see something like that as addon for XF because not everyone has server access.
 
Back
Top Bottom