XF Bot Guard 1.4.1

[zeeb0t]

Aaaand we're slowing back down again as the bot number approaches 60k. What a rollercoaster of a morning, lol.

View attachment 340569


EDIT: And we're back down under 1k again. Rollercoaster indeed.

That rollercoaster would be because the bots seem to be coming in big waves - and Bot Guard is easing back into normal operation as a new wave hits, so ends up enacting survival traits again... then back to normal... then survival... etc.

You see, survival traits enact once the server is becoming loaded - but with large waves, that can be almost instant.

New features arriving next week are probably ideal for you. As it removes much of the DB write-path as an optional running mode. Lightweight, most features still available.

 

[zeeb0t]

@Chromaniac has the MMDB issue resolved. For anyone else having the same issue - looks like Cloudflare on my end was rejecting a bunch of attempts to download the files. I added a path rule so it allows all requests there. By the time you read this, your install will probably already be resolved. If not, please run the "XF Bot Guard MMDB refresh" cron.

 

[Apt Irrelevance]

That rollercoaster would be because the bots seem to be coming in big waves - and Bot Guard is easing back into normal operation as a new wave hits, so ends up enacting survival traits again... then back to normal... then survival... etc.

You see, survival traits enact once the server is becoming loaded - but with large waves, that can be almost instant.

Makes sense. So would you recommend we leave survival mode on permanently?

 

[zeeb0t]

Makes sense. So would you recommend we leave survival mode on permanently?

I personally leave it enabled permanently.

It is not 100% perfect, as it relies on server-side signals to detect load. If the load spike is sudden, it may miss the first few minutes before it has enough data to react. That said, in my view, this is still much better than the forum becoming unavailable for a longer period of time.

I’ll look at improving the detection mechanism so it can identify those trends faster.

The tricky part is that I do not simply use “lots of bots are visiting” as the trigger. Different servers can handle very different levels of traffic, so enabling the policy purely because bot traffic is high would not make sense for every setup.

Instead, I’ll work on adding quicker and more reliable load signals so the system can react sooner when the server is genuinely starting to struggle.

 

[Apt Irrelevance]

Sounds great, thanks for the info.

 

[zeeb0t]

Sounds great, thanks for the info.

You're welcome!

 

[Kentarok]

Is it possible to disable protection for IPs from a specific country?

 

[zeeb0t]

Is it possible to disable protection for IPs from a specific country?

I can add a feature for this, it is now on my backlog.

 

[zeeb0t]

v1.3.6 is cooking up to be a pretty strong release.

The new Light DB mode is the big one. It manages to retain almost all protections while dramatically reducing database growth, which should be a real win for larger forums where storage overhead can become costly.

There are also several CPU and memory optimisations coming that will benefit all modes of operation, not just Light DB mode.

More soon.

 

[woody]

v1.3.6 is cooking up to be a pretty strong release.

The new Light DB mode is the big one. It manages to retain almost all protections while dramatically reducing database growth, which should be a real win for larger forums where storage overhead can become costly.

There are also several CPU and memory optimisations coming that will benefit all modes of operation, not just Light DB mode.

More soon.

Awaiting for this...had to disable the plugin today, even a new badass server (768gb ram + 64 cores) was kicking 503 errors due to this plugin. 12m table rows proved brutal.

 

[zeeb0t]

Awaiting for this...had to disable the plugin today, even a new badass server (768gb ram + 64 cores) was kicking 503 errors due to this plugin. 12m table rows proved brutal.

yeah, ouch! this should do a lot to improve your situation

 

[zeeb0t]

zeeb0t updated XF Bot Guard with a new update entry:

Major Release: v1.3.6

XF Bot Guard 1.3.6

Important upgrade notes for existing customers:

Light-DB runtime mode requires XenForo application cache to be configured first. If you are not enabling Light-DB mode, no extra action is required. To use Light-DB mode, configure XenForo cache first: XenForo cache documentation.

Release type: major

XF Bot Guard 1.3.6 is a major performance and scalability release, introducing the new optional Light-DB runtime mode alongside broad CPU, memory, compatibility, and admin reliability improvements.

Read the rest of this update entry...

 

[zeeb0t]

Light-DB mode is here

A few important notes for anyone who decides to use it:

1. XenForo cache must be enabled and working

You must have XenForo cache enabled. The Bot Guard health check will tell you whether your cache is configured correctly and whether the required checks pass.

If you enable Light-DB mode without a working cache, Bot Guard will fail open, which means it is effectively disabled.

2. Dashboard metrics are intentionally limited

Dashboard metrics are slim-to-none in Light-DB mode.

Most of what is reported on the dashboard relies on visitor session data that standard DB operation records. Because Light-DB mode avoids much of that database activity, there simply is not much session data available to report against.

That said, the default enabled Adjust Members Online counts for challenged visitors feature still works in Light-DB mode. You can still use the Members Online panel to get a general sense of what Bot Guard is treating as real visitors / allowed bots compared with everything else.

3. This is the first public release

This is the first public release of Light-DB mode. I have tested it in my own environment and found it ran without issue, with noticeably snappier page loads.

That said, as this is the first release - and it also includes several performance improvements for standard DB mode - please let me know if you run into any issues and I will look into them.

 

[zeeb0t]

@Ridemonkey @Faust @Anthony Parsons @Apt Irrelevance @woody - each if you either wanted to know when Light DB is released, reported an issue now fixed, or in some way showed interested in this space (or I believe your traffic volumes may benefit from the release). So just tagging you accordingly.

Sorry if I missed some folks - I am sure I have... the conversation here happens swiftly and it's hard to keep a working memory of who to tag.

 

[Apt Irrelevance]

Got it installed and Light-DB mode enabled. Excited to see how it runs with redis.

 

[zeeb0t]

Got it installed and Light-DB mode enabled. Excited to see how it runs with redis.

Nice. Something I will do in future release is leverage the cacher hit counts for some dashboard snapshot telemetry. Problem is, it's not 100% accurate due to the way cache works - it can't be used as an absolute source of truth. Anyway, for now the dashboards will be mostly useless under Light-DB, but I will see what I can do to give some additional data points from other sources.

 

[zeeb0t]

For those enabling Light-DB on existing installations - you may wish to safely truncate the following tables to immediately reclaim data storage used by normal DB mode that are not populated in Light-DB mode.

TRUNCATE TABLE xf_zee_botguard_event_reason;
TRUNCATE TABLE xf_zee_botguard_event_detail;
TRUNCATE TABLE xf_zee_botguard_event;
TRUNCATE TABLE xf_zee_botguard_counter;
TRUNCATE TABLE xf_zee_botguard_fp_ip;
TRUNCATE TABLE xf_zee_botguard_ip;
TRUNCATE TABLE xf_zee_botguard_session;
TRUNCATE TABLE xf_zee_botguard_visitor;

 

[Apt Irrelevance]

Nice. Something I will do in future release is leverage the cacher hit counts for some dashboard snapshot telemetry. Problem is, it's not 100% accurate due to the way cache works - it can't be used as an absolute source of truth. Anyway, for now the dashboards will be mostly useless under Light-DB, but I will see what I can do to give some additional data points from other sources.

If it makes things run smoother when we have 70k bots online, it's a worthy tradeoff. We've been going through it the past few days.

 

[zeeb0t]

If it makes things run smoother when we have 70k bots online, it's a worthy tradeoff. We've been going through it the past few days.

Looks like you are at 20k bots atm, page seems snappy on my end. I'll let you continue to monitor, I hope it makes a big difference. I don't see why it shouldn't... but nothing like a proper live drill to find out for sure.

 

[zeeb0t]

If it makes things run smoother when we have 70k bots online, it's a worthy tradeoff. We've been going through it the past few days.

Btw, I still recommend keeping "Enable origin survival mode" enabled even on Light-DB. Sheer volumes of bots can still overwhelm regardless.