XF Bot Guard 1.4.1

[smallwheels]

So yes, I agree: it is not 100%. I do not think 100% is a realistic or safe promise.

My statement wasn't intended as criticism but rather as an observation that could help to improve detection.

A number of suspicious guests still reaching old or buried threads is definitely something worth looking at, though. There will always be edge cases, especially with bots that behave more like normal browsers or arrive through old indexed URLs, external links, archives, bookmarks, or previously discovered URLs.

I am also planning additional detection features that should help with exactly this type of behaviour, including deliberately placed hidden/trap URLs that normal users should never reach, but crawlers and scrapers often will. That should give Bot Guard another strong signal for identifying bots that currently appear more human-like.

This would be awesome!

 

[smallwheels]

I could introduce a mechanism for you to block certain IPs - which, when hashed, can then be used to link sessions and fingerprints and so on, effectively blocking them even if they come back on new IPs. However, I will say - that is probably not going to scale. You will spend forever basically doing that.

I don't think this would make sense, the more as we are talking about resident proxies here - the IP is of no value. What could however be interesting is the fingerprint or other properties of the "browser" plus the behavior of the client.

 

[zeeb0t]

My statement wasn't intended as criticism but rather as an observation that could help to improve detection.

Oh, don't worry - I wasn't taking it as criticism in the strictest sense. I just have to be careful with how hard I code Bot Guard to scrutinise every visitor, otherwise we end up with much worse problems - unhappy humans.

But there are still dozens of tricks coming. I have just been focused very much on hardening the product for big forums... because it seems like they are beginning to pick up Bot Guard and install it. Which is amazing!

 

[zeeb0t]

I don't think this would make sense, the more as we are talking about resident proxies here - the IP is of no value. What could however be interesting is the fingerprint or other properties of the "browser" plus the behavior of the client.

Yeah, agreed - I do not mean traditional manual IP blocking.

With residential proxies, the IP itself is often not the useful part. What I mean is more like using that visitor/IP as a starting point.

For example, an admin could flag a suspicious guest/session, and Bot Guard could then look at the related signals around that visit: browser fingerprint, session behaviour, request pattern, timing, old-thread access, repeat characteristics, and other entropy.

From there, Bot Guard would not necessarily block that IP outright. It would use the related properties to raise the score on similar future activity.

So it would be less “block this IP” and more “this visitor looked suspicious - learn from the surrounding signals and treat similar traffic with more suspicion”.

 

[Rhody]

I'm just reporting that 1.2.3 went in flawlessly, and I opted for the full conversion. It took about 8 minutes to convert. Stats showed 0 until I manually ran the hourly cron.

 

[zeeb0t]

I'm just reporting that 1.2.3 went in flawlessly, and I opted for the full conversion. It took about 8 minutes to convert. Stats showed 0 until I manually ran the hourly cron.

View attachment 339480

Wonderful! Thanks for reporting. And yes, the cron being hourly means after upgrade, it does require a manual run (or just wait until it runs on schedule). On the first run, it will more or less catch you up. Depending on how much data, it may be more than one run, as it is designed not to overwhelm the server resources just to build report snapshots. Once it is caught up, it will keep your dashboard up to date each hour in arrears (so it runs at 10 mins past the hour, and snapshots the prior full hour from 0 min to 59 min inclusive).

 

[zeeb0t]

Detailed documentation for XF Bot Guard attached (this will be going on the resource/FAQ section, but sharing it here for existing users).

 

[Nettpreneur]

Detailed documentation for XF Bot Guard attached (this will be going on the resource/FAQ section, but sharing it here for existing users).

Awesome job @zeeb0t , I'll try out your addon and give feedback in a day or two.

 

[zeeb0t]

Awesome job @zeeb0t , I'll try out your addon and give feedback in a day or two.

Awesome! I am here, if you need any assistance. Cheers!

 

[zeeb0t]

zeeb0t updated XF Bot Guard with a new update entry:

Minor Release: v1.2.4

XF Bot Guard 1.2.4

Release type: Minor, with major reliability improvements

This release introduces intelligent event-detail buffering to help protect database performance during periods of heavy traffic or bot activity. XF Bot Guard can now automatically detect database pressure and temporarily shift write-heavy event-detail storage to disk, importing data back into the database smoothly over time without affecting protection decisions, telemetry, detection signals...

Read the rest of this update entry...

 

[zeeb0t]

This update adds a new automatic event detail buffering system.

This is designed to help forums during heavy or sudden bot traffic spikes, when the database server may already be under pressure.

When XF Bot Guard detects that the database is struggling, it can temporarily move some event-detail writing work to disk instead of pushing everything into the database immediately. Those details are then written back to the database in a controlled and load-sensitive way, instead of adding to the kind of constant database pressure caused by hammering bots.

The result is:

• no loss of event detail data
• no reduction in Bot Guard’s detection capability
• less pressure on an already stressed database server

By default, this feature runs in automatic mode. You can also change it in the options to be always on or disabled.

The updated documentation is attached.

As always, if you have feedback or need help, I am here for you.

If XF Bot Guard is helping your forum, please also consider leaving a 5-star rating. Just go to the add-on Overview page and click the rating button near the top-right of the page, under the download button.

 

[zeeb0t]

zeeb0t updated XF Bot Guard with a new update entry:

Bug Fix: v1.2.5

XF Bot Guard 1.2.5

Release type: bug fix

This patch resolves a dashboard snapshot cron issue introduced with event-detail buffering in XF Bot Guard 1.2.4.

In some cases, when buffered event details were still pending, the dashboard snapshot cron could attempt to write an internal status value that was too long for the snapshot status column. This could trigger a MySQL “Data too long for column 'status'” server error.

This patch also improves dashboard snapshot...

Read the rest of this update entry...

 

[Rhody]

Great new feature! Even with close to 17000 bots on my system, it's holding its own.

 

[azzurro]

Please apply the latest 1.2.3 - but carefully consider your upgrade path. Then, for your forum in particular, I would try with the following retention settings:

Data type	Default	For you
Raw event data	336 hours	24 hours
Visitor data	336 hours	24 hours
Session data	336 hours	24 hours
Counter data	336 hours	24 hours
Dashboard snapshots	17,520 hours	17,520 hours (2 years) - or however long you want to retain dashboard data. This area is lightweight. Shouldn't grow crazy fast, and you can always revise this down as needed.
Zero-signal IP data	2,160 hours	24 hours

1.2.3 has been running great for over a day with the above setting changes. Zero-signal IP data I set to the minimum allowed 720 rather than the 24 mentioned above but all seems fine. I just installed 1.2.5 and no issues so far. Thanks again!

 

[duderuud]

Just installed this to test it out. Some pages are hardly readable:



Still settings things up, will let you know how it goed @zeeb0t

 

[smallwheels]

Some pages are hardly readable

possibly easier if you switch off dark mode?

 

[duderuud]

That helps a lot, thanks Still, could be styled better...

Another question, not sure how to solve this:


Tried disabling Redis caching but that didn't help. Nginx caching maybe?

 

[beerForo]

This is a great add-n but could you space out your releases?

 

[zeeb0t]

This is a great add-n but could you space out your releases?

Slow down? haha na, I understand. I am going to start bundling releases / changes. That will give the appearance of slowing down.

 

[zeeb0t]

That helps a lot, thanks Still, could be styled better...

Another question, not sure how to solve this:
View attachment 339552

Tried disabling Redis caching but that didn't help. Nginx caching maybe?

Its just a warning that it detected some cache headers but not the ones it is truly worried about.

It says this incase there was some kind of cache hit header I don't know about... but found artifacts that MIGHT be an incidacator of page cache, hence the warning. Its basically asking you to confirm.

But in your instance, its clearly Cloudflare - so just ignore that message. I will tighten up the logic.