XF Bot Guard 1.4.1

[Chromaniac]

So, just curious why you are using xenforo framework to serve the verification challenge instead of a barebones page like the surge guard addon.

Also, i do worry that serving SO MANY captchas might eventually make cloudflare made with you lol.

 

[Aaratech]

@smallwheels, @Dragonfruit, @akok

I’m sharing a pre-release build of XF Bot Guard 1.0.7 for those who specifically requested better support for legitimate crawlers/fetchers without needing to rely on Cloudflare verified-bot headers.

I am already running this build in production, but before I package it as an official release, I’d appreciate some wider testing from sites that have been affected by search crawlers or fetchers being challenged incorrectly.

The build is attached to this post.

What this build adds​

The main new feature is a DB-backed verified crawler/fetcher allowlist.

Instead of trusting user-agent strings, reverse DNS, or spoofable request headers, XF Bot Guard can now allow requests where the request IP matches an official provider-published crawler/fetcher CIDR range.

In plain English:

Request IP is inside an official crawler/fetcher IP range
=
allow before Bot Guard challenge/risk scoring

This is designed to help legitimate crawlers get through Bot Guard safely, especially on sites that are not behind Cloudflare or similar infrastructure.

Included providers/sources​

This first version only includes strong, official, machine-readable IP/CIDR sources.

Included sources are:

• Google common crawlers
• Google special-case crawlers
• Google user-triggered fetchers
• Google-controlled user-triggered fetchers
• Google user-triggered agents
• Bingbot
• Applebot
• DuckDuckBot

This does not add support for user-agent-only bots, commercial SEO tools, AI crawlers, DNS-only verification, or manually maintained IP lists.

New option​

A new option has been added:

zeeBotGuardAllowVerifiedCrawlers

Admin label:

Allow verified search crawlers and fetchers

Default:

Enabled

This option is separate from the existing known-bot/header-based option:

zeeBotGuardAllowKnownBots

The existing option remains for upstream verified-bot headers. The new option is for local DB-backed IP range verification using official provider feeds.

How to check whether it is working​

Allowed crawler/fetcher visits under the new option are logged with this event type:

verified_crawler_allow

So, in the Bot Guard event log, look for:

verified_crawler_allow

The event metadata should include details such as:

{
"provider": "google",
"source_id": "google_common_crawlers",
"source_label": "Google common crawlers",
"cidr": "66.249.64.0/19",
"source_age_seconds": 3600,
"verification": "official_cidr_json"
}

This event is different from the older/header-based known crawler event:

known_crawler_allow

For this new 1.0.7 feature, the event you want is:

verified_crawler_allow

Health check​

After installing/upgrading, please check the XF Bot Guard health check.

The health check should now report the status of:

• The verified crawler option
• The known crawler source table
• The known crawler range table
• How many official sources are enabled
• How many have refreshed successfully
• Whether any sources have never refreshed
• Whether any source data is stale
• Whether any recent source refresh failed

The important thing to confirm is that the official IP sources imported successfully and that known crawler ranges are available.

If the health check says the option is enabled but no known crawler ranges are available, then the feature will not be able to allow any crawler/fetcher visits yet.

Refresh behaviour​

The official IP/CIDR feeds are refreshed automatically by a new cron entry:

zeeBgKnownCrawlerRefresh

The refresh is designed to be safe:

• Successful refreshes replace ranges transactionally.
• Temporary provider/API failures do not wipe existing known-good ranges.
• Invalid or empty source updates are rejected.
• Last-known-good data remains usable.
• No remote HTTP or DNS checks happen during normal page requests.

What I’d like tested​

Please check:

1. The add-on installs/upgrades cleanly.
2. The health check shows the known crawler sources and ranges correctly.
3. The new option is present and enabled by default.
4. Legitimate Google/Bing/Apple/DuckDuckGo crawler or fetcher visits show as verified_crawler_allow.
5. Normal unknown bots still go through the usual Bot Guard flow.
6. There are no unexpected PHP errors, cron errors, or install/upgrade issues.

So far so good, only a tiny styling issue btw in the logs that i noticed.

Yandex bot still being blocked by captcha. Probably shouldnt be allowed in this version as well, just pointing that out, since im just testing this addon for like 30 minutes and i didnt get to it yet if i can allow it somehow manually or not. Still going trough all that haha. Its a nice addon.

 

[zeeb0t]

So, just curious why you are using xenforo framework to serve the verification challenge instead of a barebones page like the surge guard addon.

Also, i do worry that serving SO MANY captchas might eventually make cloudflare made with you lol.

They serve different purposes.

XF Bot Guard intentionally serves the verification challenge through XenForo because the challenge is tied into XenForo’s own CAPTCHA provider, session handling, CSRF handling, style/phrases, safe return URLs, and Bot Guard’s own reputation/logging system.

A barebones challenge page could be lighter, that is true, and that is closer to what XF Surge Guard is designed for when the goal is to short-circuit obvious garbage request volume before XenForo does unnecessary work.

However, Bot Guard is intentionally different. It is an application-layer verification system. The trade-off is that XenForo is booted for the challenge flow, but the benefit is a much more integrated and reliable XenForo-aware verification process.

So there is no contradiction between the two products. They sit at different layers and work well together. Surge Guard is better suited to absorbing obvious surging garbage traffic earlier, especially on nonsense URLs or request patterns where XenForo should not need to do much work. Bot Guard is better suited to suspicious but potentially legitimate public page traffic where XenForo-aware CAPTCHA verification and trust/reputation handling are useful.

Used together, they are a strong suite of product

On CAPTCHA volume: yes, if a forum is under heavy scraping pressure and does not have edge/WAF/rate-limit protection in front of it, Bot Guard may end up doing a lot of the leg work. I still recommend edge protection where available, because ideally CAPTCHA should not be the only line of defence against raw bot floods.

That said, for Cloudflare Turnstile specifically, assuming that is the CAPTCHA provider you mean, the current public Turnstile plan information lists unlimited challenges / verification requests. I am not giving legal advice, and Cloudflare’s general terms still apply, but I do not see normal defensive use by a forum as a terms issue. If a site is being scraped heavily, using the configured CAPTCHA provider to verify suspicious traffic is exactly the kind of defensive use this feature is intended for.

So far so good, only a tiny styling issue btw in the logs that i noticed.

Noted. I will address this in an upcoming release

Yandex bot still being blocked by captcha. Probably shouldnt be allowed in this version as well, just pointing that out, since im just testing this addon for like 30 minutes and i didnt get to it yet if i can allow it somehow manually or not. Still going trough all that haha.

Yes, that is intentional in this version.

At the moment, XF Bot Guard only auto-trusts crawler providers where I can rely on published machine-readable IP/CIDR feeds, with IPv4 and IPv6 support. User-agent headers alone are not trusted because they are trivial to forge.

Yandex is a different case. Yandex does publish an IP ranges page, but it is not the same kind of crawler-specific JSON feed used by the currently supported verified crawler sources. Yandex’s own guidance is to verify Yandex robots using reverse DNS and then confirm the hostname resolves back to the same IP.

That rDNS-style verification path is planned for a later version, and that should allow proper support for Yandex and other crawler types that cannot safely be verified from a simple published JSON range feed.

For now, if you want to allow Yandex manually, you can add Yandex’s published IP ranges to:

Admin CP → Setup → Options → XF Bot Guard → Excluded IPs/CIDRs

Add one IP address or CIDR range per line.

e.g., taken right now from https://yandex.com/ips

5.45.192.0/18
5.255.192.0/18
37.9.64.0/18
37.140.128.0/18
77.88.0.0/18
84.252.160.0/19
87.250.224.0/19
90.156.176.0/20
92.255.112.0/20
93.158.128.0/18
95.108.128.0/17
141.8.128.0/18
178.154.128.0/18
185.32.187.0/24
213.180.192.0/19
2a02:6b8::/29

Just be aware that this is a manual exclusion. You would be responsible for keeping those ranges up to date if Yandex changes them. Once rDNS verification is added, that should be the cleaner long-term approach - and you would then be able to remove these from that setting.

 

[zeeb0t]

Its a nice addon.

Glad you like it if you wouldn't mind, same goes for others here - I would surely appreciate receiving ratings for the add-on. Being new, the social proof would go a long way. Thank you!

 

[zeeb0t]

zeeb0t updated XF Bot Guard with a new update entry:

Release: v1.0.8

Verified Crawler/Fetcher IP Allowlist & Counter Cache Optimisation​

This update adds XF Bot Guard’s own DB-backed verified crawler/fetcher allowlist, along with a XenForo cache-backed counter optimisation to reduce repeated database reads where a XenForo cache is configured.

This release bundles the 1.0.7 crawler/fetcher allowlist work together with the 1.0.8 counter cache improvement, so 1.0.7 has been skipped as a separate public release.

Version note...​

Read the rest of this update entry...

 

[smallwheels]

Upgraded to 1.0.8 after 1.0.7 had been humming along nicely through the night w/o issues, letting Search Engine Bots pass. It seems that it does challenge most guests with a captcha, giving most them a score of either 75 or 85 (while some get way more than 100), and only letting very few guests pass unobstructedly, possibly being a bit overcritical when it comes to distinguish humans from bots (or lacking finer granular criteria for that to do it successfully upfront). But then, having switched to CF Turnstyle captcha, checking the captcha box once should not be too annoying to humans and they should be used to it anyway from other webpages. With a classical captcha it would possibly be a different story.

With IP Threat Monitor in place that does the majority of the filtering based on a pretty rigorous rule set before requests reach Bot Guard this seems a pretty nice combination: My guest count had already been pretty low in comparison as Threat Monitor filters out most of the crap successfully, but I knew that I had a bunch of bots through resident proxies within it that I could not catch with IP Threat Monitor as they are hidden within the geographical area and the ASNs my regular visitors use, too. Now Bot Guard takes on this job. Successfully, as it seems until now. As long as the bots aren't able to solve Turnstyle captchas successfully at least. As a result, my guest count has been lowered further by about 30-70%. Nice.

As a consequence I might be able to make some of the rules in Threat Monitor less strict, namely unblock some residential ASNs of Cable/DSL providers where considerable amount of residential proxy requests come from along with few legitimate ones. In my case this mainly affects the US, but also UK and a couple of other countries. So Bot Guard hopefully enables a more granular approach towards blocking, potentially giving legitimate users from bot loaded ASNs the chance to reach my forums again.

I have to dive a bit deeper into the rules and decisions of Bot Guard to understand the decisions better. What could be worth watching is the size of the log. Even at my pretty low visitor count (and even lower guest count) it is growing pretty fast, possibly growing the database considerably if your forum has a lot of traffic. Not checked yet, but probably worth having an eye on that.

 

[zeeb0t]

Upgraded to 1.0.8 after 1.0.7 had been humming along nicely through the night w/o issues, letting Search Engine Bots pass. It seems that it does challenge most guests with a captcha, giving most them a score of either 75 or 85 (while some get way more than 100), and only letting very few guests pass unobstructedly, possibly being a bit overcritical when it comes to distinguish humans from bots (or lacking finer granular criteria for that to do it successfully upfront). But then, having switched to CF Turnstyle captcha, checking the captcha box once should not be too annoying to humans and they should be used to it anyway from other webpages. With a classical captcha it would possibly be a different story.

With IP Threat Monitor in place that does the majority of the filtering based on a pretty rigorous rule set before requests reach Bot Guard this seems a pretty nice combination: My guest count had already been pretty low in comparison as Threat Monitor filters out most of the crap successfully, but I knew that I had a bunch of bots through resident proxies within it that I could not catch with IP Threat Monitor as they are hidden within the geographical area and the ASNs my regular visitors use, too. Now Bot Guard takes on this job. Successfully, as it seems until now. As long as the bots aren't able to solve Turnstyle captchas successfully at least. As a result, my guest count has been lowered further by about 30-70%. Nice.

As a consequence I might be able to make some of the rules in Threat Monitor less strict, namely unblock some residential ASNs of Cable/DSL providers where considerable amount of residential proxy requests come from along with few legitimate ones. In my case this mainly affects the US, but also UK and a couple of other countries. So Bot Guard hopefully enables a more granular approach towards blocking, potentially giving legitimate users from bot loaded ASNs the chance to reach my forums again.

I have to dive a bit deeper into the rules and decisions of Bot Guard to understand the decisions better. What could be worth watching is the size of the log. Even at my pretty low visitor count (and even lower guest count) it is growing pretty fast, possibly growing the database considerably if your forum has a lot of traffic. Not checked yet, but probably worth having an eye on that.

Your post, as well as a message from one of my forum users (I run it on my users to get human feedback for potential issues), has helped me identify a bit of a race condition to session/cookie continuity that seems to be occurring. I am just testing a fix right now, and will post back shortly. If it tests out true, it will greatly reduce the number of "human-looking" CAPTCHA targets.

 

[smallwheels]

a bit of a race condition to session/cookie continuity that seems to be occurring

The "classical" 75 shows:

"code": "missing_or_unconfirmed_fingerprint",
        "points": 25
    },
    {
        "code": "no_server_cookie_continuity",
        "points": 20
    },
    {
        "code": "no_server_js_confirmation",
        "points": 30
    },
    {
        "code": "strict_no_browser_no_grace",
        "points": 0
    }

This is hit (with identical reasons) by basically almost any guest visitor that is not a verified Search Engine Bot.

85 is hit when someone requesting i.e. the "what's new" page (which probably is not uncommon with human visitors). It adds 10 points, so we end up with 85:

{
        "code": "missing_or_unconfirmed_fingerprint",
        "points": 25
    },
    {
        "code": "no_server_cookie_continuity",
        "points": 20
    },
    {
        "code": "no_server_js_confirmation",
        "points": 30
    },
    {
        "code": "route_family_find_new",
        "points": 10
    },
    {
        "code": "strict_no_browser_no_grace",
        "points": 0
    }

while 100 shows

{
        "code": "missing_or_unconfirmed_fingerprint",
        "points": 25
    },
    {
        "code": "no_server_cookie_continuity",
        "points": 20
    },
    {
        "code": "no_server_js_confirmation",
        "points": 30
    },
    {
        "code": "entropy_fetch_metadata_non_navigation",
        "points": 25
    },
    {
        "code": "strict_no_browser_no_grace",
        "points": 0
    }

(in this case giving 25 on top of the classic 75 b/c the request targeted at a tag)

 

[Dragonfruit]

So upon testing on guest, I get prompted, I do the captcha, and while browsing, on the online activity, my guest tester is still "Submitting browser verification", even though I'm just clicking around, and it's not showing the captcha anymore. I can view the forum and threads in full.

 

[zeeb0t]

zeeb0t updated XF Bot Guard with a new update entry:

Release: v1.0.9

This is a minor maintenance release that fixes an issue where XF Bot Guard could be too quick to challenge some genuine visitors before browser verification had been fully established.

• Improved browser verification continuity so genuine visitors are less likely to be asked to verify again while browsing normally.
• Strengthened handling for clients that do not support JavaScript or cookies, so repeat automated requests are challenged more reliably.
• Refined visitor tracking...

Read the rest of this update entry...

 

[zeeb0t]

@smallwheels and @Dragonfruit the latest 1.0.9 fixes a race condition that was causing XF Bot Guard to be too quick to challenge. I advise upgrading.

Re: the "Submitting browser verification" message not updating in the members online area: this is something controlled by XenForo, probably a cache value or whatnot. It is responsible for deciding which page the user is apparently on - I only provide labels for it to use when it determines they are looking at this add-ons content.

 

[Chromaniac]

I see that for exempted logged in members as well.

 

[zeeb0t]

I see that for exempted logged in members as well.

Please send me a message with some screenshots and your configuration, I'll have a look. Plus, please apply the latest 1.0.9 version.

 

[zeeb0t]

I see that for exempted logged in members as well.

Oh, sorry - I misread the exact wording you mentioned.

“Submitting browser verification” is expected. That wording refers to the browser verification / collector step, where XF Bot Guard is collecting the browser signal and related continuity data.

It does not mean the user is being challenged with CAPTCHA at that moment.

Routes and user groups can be excluded from the challenge gate, but Bot Guard can still collect browser verification data more broadly. That is intentional, because the add-on uses that data for session continuity and reputation. For example, if the visitor logs out later, changes state, or enters an area that is protected by Bot Guard, it already has browser/session history available rather than starting from nothing at that point.

So in practical terms:

“Submitting browser verification” = browser signal / fingerprint collection.

It does not mean the visitor is being forced through the Bot Guard CAPTCHA challenge.

If an exempt user is actually seeing the CAPTCHA challenge page, that would be different. But the Current visitors wording by itself is expected behaviour.

 

[smallwheels]

1.0.9 fixes a race condition that was causing XF Bot Guard to be too quick to challenge. I advise upgrading.

Guest count went up immediately from 2 to 12 plus two facing the challenge. Before 1.0.9 it was the other way round. No idea if that is now good or bad / better or worse.

 

[zeeb0t]

Guest count went up immediately from 2 to 12 plus two facing the challenge. Before 1.0.9 it was the other way round. No idea if that is now good or bad / better or worse.

With this fix applied, it slows down the sometimes "aggressive" judgement. Previously, there was a race condition meaning visitors were quickly dispatched to the challenge gate, before their browser could be fingerprinted and entropy collected. Now, it is awaiting the proper time before sending them that way. It was premature at times.

The same logic still applies, however, in that scoring and safe vs. unsafe risk factors are all counted the same.

 

[zeeb0t]

zeeb0t updated XF Bot Guard with a new update entry:

Release: v1.1.0

New Bootstrap History Protection​

This update adds a new behavioural protection layer for clients that repeatedly receive bootstrap access but never complete normal browser verification.

This is designed to reduce cases where rotating-IP crawlers or scrapers repeatedly appear as fresh visitors and continue receiving short-lived bootstrap allowance without ever behaving like a real browser.

Behaviour-based bootstrap suppression​

XF Bot Guard now...

Read the rest of this update entry...

 

[Chromaniac]

just a fyi. I am on the latest version. and the members online tab (/online/?type=member) basically just show browser verification for everyone. it was mostly working fine 2-3 versions ago where most members would eventually show what they are doing on the forum. now, it does show where some of them actually are but on a refresh, it is back to browser verification.

 

[zeeb0t]

just a fyi. I am on the latest version. and the members online tab (/online/?type=member) basically just show browser verification for everyone. it was mostly working fine 2-3 versions ago where most members would eventually show what they are doing on the forum. now, it does show where some of them actually are but on a refresh, it is back to browser verification.

I will look in to seeing if it is possible to actually suppress this page type outright. Technically, it's not even a page. It's a data endpoint. XenForo would ideally not even consider it a 'what the user is looking at' type of thing. I will figure out how to suppress it, and release a patch for this.

 

[akok]

e.g., taken right now from

Wouldn't it be better to add whitelists via Reverse DNS? That way, search engines wouldn't have to specify too many IP addresses.
And secondly, is it possible to add comments to the list? With so many IP addresses, it's easy to lose track of why you added them.