XF Bot Guard 1.4.1

[zeeb0t]

Ah apologies, the thread's moving fast. Haven't read every message.

Nobody had asked yet... don't worry haha! But yeah, I did think of it - but also don't sit on new features I feel are kick-ass. So I just noted it for later and pushed it out

 

[Dragonfruit]

Nobody had asked yet... don't worry haha! But yeah, I did think of it - but also don't sit on new features I feel are kick-ass. So I just noted it for later and pushed it out

xD I read that as "I have stated"...

 

[zeeb0t]

xD I read that as "I have stated"...

Oh, nah. Even if I had previously stated it (I hadn't) I am not normally so blunt haha. All good mate!

 

[azzurro]

For anyone using XF Bot Guard behind Cloudflare, these are my own Cloudflare cache settings.

This is not the only valid setup, but it shows the important distinction: I cache specific static/media-style XenForo endpoints only. I do not cache normal forum pages, guest HTML responses, login/register pages, XF Bot Guard verification pages, or challenge/interstitial responses.

XF Bot Guard needs fresh guest requests. Anything that turns normal guest page responses into cached edge HTML is likely to cause problems.

Cloudflare → Caching → Configuration

• Caching Level: Standard
• Browser Cache TTL: 4 hours
• CSAM Scanning Tool: Enabled, but not relevant to this discussion
• Crawler Hints: Enabled
• Always Online™: Disabled
• Development Mode: Disabled
• Enable Query String Sort: Disabled

Cloudflare → Caching → Tiered Cache

• Tiered Cache Topology: Enabled
• Smart Tiered Cache: Enabled
• Regional Tiered Cache: Disabled

Cloudflare → Caching → Cache Reserve

• Cache Reserve: Disabled

Cloudflare → Caching → Cache Rules

These are the only explicit cache rules I use.

1. Cache proxy.php images

(http.request.method eq "GET" and http.request.uri.path eq "/proxy.php" and http.request.uri.query contains "image=")

• Cache eligibility: Eligible for cache
• Edge TTL: Use cache-control header if present, cache request with Cloudflare's default TTL for the response status if not
• Status code TTL:
  • 100–103: No store
  • 200–226: 7 days
  • 300–526: No store

2. Cache css.php

(http.request.method eq "GET" and http.request.uri.path eq "/css.php" and http.request.uri.query contains "css=")

• Cache eligibility: Eligible for cache
• Edge TTL: Use cache-control header if present, cache request with Cloudflare's default TTL for the response status if not
• Status code TTL:
  • 100–103: No store
  • 200–226: 7 days
  • 300–526: No store

3. Cache webmanifest.php

(http.request.method eq "GET" and http.request.uri.path eq "/webmanifest.php")

• Cache eligibility: Eligible for cache
• Edge TTL: Use cache-control header if present, cache request with Cloudflare's default TTL for the response status if not
• Status code TTL:
  • 100–103: No store
  • 200–226: 7 days
  • 300–526: No store

4. Cache /attachments/*

(http.request.method eq "GET" and starts_with(http.request.uri.path, "/attachments/"))

• Cache eligibility: Eligible for cache
• Edge TTL: Use cache-control header if present, cache request with Cloudflare's default TTL for the response status if not
• Status code TTL:
  • 100–103: No store
  • 200–226: 7 days
  • 300–526: No store

What matters for XF Bot Guard compatibility

The key point is that these rules are intentionally narrow. They cache things like proxied images, CSS, web manifest output, and attachments. They do not tell Cloudflare to cache full XenForo pages.

Cloudflare itself is not the problem. Caching itself is not the problem. Cloudflare’s standard caching is perfectly fine when it respects normal dynamic-page freshness.

The problem starts when another Cloudflare setting, page rule, cache rule, worker, plugin, or add-on causes guest HTML pages to be cached.

Specific note for the DigitalPoint Cloudflare add-on

If you are using the DigitalPoint Cloudflare add-on, the incompatible feature is:

Cloudflare → Edge caching → Cache pages for guests

That feature is designed to cache XenForo HTML pages for guests at Cloudflare’s edge.

This is different from simply caching CSS, proxied images, web manifests, or attachments. Guest page caching means normal XenForo HTML pages can be served from Cloudflare without the request reaching XenForo/PHP.

That is the part XF Bot Guard is not compatible with.

The guest page caching feature works by creating a Cloudflare Cache Rule for guests and by making eligible XenForo guest HTML responses cacheable.

The Cloudflare rule to look for will usually be named something like:

Cache XenForo guest pages

The expression will look similar to this, although your cookie prefix may differ:

(not http.cookie contains "xf_session=" and not http.cookie contains "xf_user=")

If you use a different XenForo cookie prefix, the cookie names in the rule may be different. The important part is that it is a broad rule matching visitors who do not have the XenForo session/user cookies.

How to disable it

In the XenForo Admin CP:

Admin CP → Cloudflare → Edge caching

Find:

Cache pages for guests

Disable it.

After disabling it, the guest page caching time should be zero/off. In technical terms, the DigitalPoint option behind this is:

cfPageCachingSeconds = 0

You should not need to manually edit that option if you disable it through the add-on’s Cloudflare edge caching screen.

Verify the Cloudflare rule was removed or disabled

After disabling the option in XenForo, also check Cloudflare directly.

Go to:

Cloudflare dashboard → Caching → Cache Rules

Look for any rule named something like:

Cache XenForo guest pages

or any broad guest-page rule using an expression similar to:

(not http.cookie contains "xf_session=" and not http.cookie contains "xf_user=")

If that rule is still present and enabled, delete it or disable it.

It is not enough to only disable the XenForo-side option if Cloudflare still has a broad rule that makes guest HTML eligible for caching.

Purge Cloudflare cache afterwards

After disabling guest page caching and confirming the Cloudflare rule is gone or disabled, purge Cloudflare cache.

At minimum, purge cached HTML/forum pages. If in doubt, purge everything.

This avoids testing against stale cached guest pages that were stored before the setting was changed.

How to verify it is working correctly

Test as a logged-out guest, preferably in a private/incognito browser window.

Normal forum pages should not be served as cached Cloudflare HTML.

For normal XenForo pages, you do not want to see Cloudflare serving them as:

cf-cache-status: HIT

You also do not want normal guest HTML pages returning cache headers like:

cache-control: max-age=0,s-maxage=3600

or any other positive s-maxage value.

Seeing cached responses for things like CSS, proxied images, web manifests, or attachments is fine if you intentionally configured those rules. The issue is normal guest HTML pages.

Rule of thumb

It is generally fine to cache static/media-style endpoints.

It is not fine to cache:

• Normal forum pages
• Guest HTML responses
• Login or register pages
• XF Bot Guard verification pages
• Challenge/interstitial responses
• Anything that depends on a fresh visitor state, cookie state, session state, or validation state

So for DigitalPoint Cloudflare add-on users, the practical recommendation is:

• You may keep using the add-on if you want its other Cloudflare management features.
• You may keep narrowly-scoped static/media cache rules if you understand what they do.
• You should disable Cache pages for guests.
• You should confirm the Cache XenForo guest pages Cloudflare Cache Rule is removed or disabled.
• You should purge Cloudflare cache after changing it.

Once guest HTML page caching is disabled, XF Bot Guard and Cloudflare can coexist normally.

I checked my cloudflare rules and didn't find anything that caches guest pages. I inspected the 'validating browser...' page and ran it through Claude, it reports:

Bug: /bot-guard/collect returns 400 on first visit for users who haven't yet accepted the cookie consent notice. XenForo rejects the AJAX POST with "Cookies are required to use this site" before Bot Guard can complete browser validation, leaving visitors permanently stuck on the "Validating Browser" page until they reload.


Steps to reproduce: Visit the site in a fresh browser session with no cookies. Bot Guard validation page appears but never completes.


Expected behavior: The collect endpoint should bypass cookie consent since it's a pre-consent security check.

 

[zeeb0t]

I checked my cloudflare rules and didn't find anything that caches guest pages. I inspected the 'validating browser...' page and ran it through Claude, it reports:

Bug: /bot-guard/collect returns 400 on first visit for users who haven't yet accepted the cookie consent notice. XenForo rejects the AJAX POST with "Cookies are required to use this site" before Bot Guard can complete browser validation, leaving visitors permanently stuck on the "Validating Browser" page until they reload.


Steps to reproduce: Visit the site in a fresh browser session with no cookies. Bot Guard validation page appears but never completes.


Expected behavior: The collect endpoint should bypass cookie consent since it's a pre-consent security check.

Which cookie consent manager add-on or setting in XenForo? I will look into this. This cookie is one of those "vital to even use the site" and disabling cookies cannot be a way to bypass security. But cookie consent usually applies to certain types of cookies, so I will make it work in that scenario too.

 

[azzurro]

Which cookie consent manager add-on or setting in XenForo? I will look into this. This cookie is one of those "vital to even use the site" and disabling cookies cannot be a way to bypass security. But cookie consent usually applies to certain types of cookies, so I will make it work in that scenario too.

I don't recall any cookie consent changes on my end, I'll take a look.

 

[azzurro]

According to Claude:

Further finding: Tested with browser dev tools on a fresh session (no existing cookies). The Bot Guard validation page does not set an xf_session cookie. When the JS fires the collect POST, no xf_session is present in the request headers. XenForo then attempts to initialize a session during the collect POST itself, setting xf_csrf in the collect response — but by that point it has already rejected the request with "Cookies are required."

There are no cookie consent settings in XenForo 2.3.10 admin — consentMode: 'simple' appears to be the default. The issue is that the validation page bypasses normal XenForo session creation, leaving the collect POST with no established session.

Environment: XenForo 2.3.10, Cloudflare proxied (Cloudflare managed challenge completes first, setting cf_clearance, before the Bot Guard validation page loads).

 

[zeeb0t]

According to Claude:

Further finding: Tested with browser dev tools on a fresh session (no existing cookies). The Bot Guard validation page does not set an xf_session cookie. When the JS fires the collect POST, no xf_session is present in the request headers. XenForo then attempts to initialize a session during the collect POST itself, setting xf_csrf in the collect response — but by that point it has already rejected the request with "Cookies are required."

There are no cookie consent settings in XenForo 2.3.10 admin — consentMode: 'simple' appears to be the default. The issue is that the validation page bypasses normal XenForo session creation, leaving the collect POST with no established session.

Environment: XenForo 2.3.10, Cloudflare proxied (Cloudflare managed challenge completes first, setting cf_clearance, before the Bot Guard validation page loads).

The POST is managed by the add-on. This part of the process occurs outwide the wider XenForo workflow. Claude has not extensively mapped it, so is not 100% correct. Anyway, I'll investigate the symptoms and provide a fix. Its all working fine on vanilla installs and my own, with proper responsing etc... so I'll hunt down via symptoms, not to worry.

 

[zeeb0t]

The POST is managed by the add-on. This part of the process occurs outwide the wider XenForo workflow. Claude has not extensively mapped it, so is not 100% correct. Anyway, I'll investigate the symptoms and provide a fix. Its all working fine on vanilla installs and my own, with proper responsing etc... so I'll hunt down via symptoms, not to worry.

Btw on vanilla install I've never seen a consent notice. Hence I believe its just wrong about it being default. I do not modify the vanilla vendor files and it is tested against each version before release.

Nevertheless, it will be soon patched from when working behind constent notices. Give me a couple hours

 

[zeeb0t]

According to Claude:

Further finding: Tested with browser dev tools on a fresh session (no existing cookies). The Bot Guard validation page does not set an xf_session cookie. When the JS fires the collect POST, no xf_session is present in the request headers. XenForo then attempts to initialize a session during the collect POST itself, setting xf_csrf in the collect response — but by that point it has already rejected the request with "Cookies are required."

There are no cookie consent settings in XenForo 2.3.10 admin — consentMode: 'simple' appears to be the default. The issue is that the validation page bypasses normal XenForo session creation, leaving the collect POST with no established session.

Environment: XenForo 2.3.10, Cloudflare proxied (Cloudflare managed challenge completes first, setting cf_clearance, before the Bot Guard validation page loads).

I have mapped this further and I do not think this is actually cookie consent in the literal sense. The “Cookies are required to use this site” message is XenForo’s CSRF/cookie protection path, and on a first-hit bootstrap validation there may not yet be an xf_csrf/xf_session cookie available to the browser.

The Bot Guard bootstrap collector is intentionally a pre-session browser proof step, so it should not depend on normal XenForo CSRF/session cookie state. The collector already uses a short-lived signed CollectorProof token for that bootstrap path, but I can see a compatibility gap where XenForo may reject the POST before Bot Guard’s collect handler gets to complete.

I will patch this so the bootstrap collect POST is handled earlier and only for the signed bootstrap collector request. Normal collector posts from full XenForo pages will continue to use the normal XenForo CSRF path.

As a temporary workaround, disable “Run browser validation bootstrap before challenging fresh visitors”. That will avoid the lightweight validating-browser bootstrap path until the patch is available.

A patch will be issued in a few hours, once testing has confirmed it is stable.

 

[azzurro]

I have mapped this further and I do not think this is actually cookie consent in the literal sense. The “Cookies are required to use this site” message is XenForo’s CSRF/cookie protection path, and on a first-hit bootstrap validation there may not yet be an xf_csrf/xf_session cookie available to the browser.

The Bot Guard bootstrap collector is intentionally a pre-session browser proof step, so it should not depend on normal XenForo CSRF/session cookie state. The collector already uses a short-lived signed CollectorProof token for that bootstrap path, but I can see a compatibility gap where XenForo may reject the POST before Bot Guard’s collect handler gets to complete.

I will patch this so the bootstrap collect POST is handled earlier and only for the signed bootstrap collector request. Normal collector posts from full XenForo pages will continue to use the normal XenForo CSRF path.

As a temporary workaround, disable “Run browser validation bootstrap before challenging fresh visitors”. That will avoid the lightweight validating-browser bootstrap path until the patch is available.

A patch will be issued in a few hours, once testing has confirmed it is stable.

Thanks!

I just disabled the browser validation bootstrap, it goes to a Xenforo 'Verification required' page with a Cloudflare 'verify you are human' page to continue.

 

[zeeb0t]

Thanks!

I just disabled the browser validation bootstrap, it goes to a Xenforo 'Verification required' page with a Cloudflare 'verify you are human' page to continue.

Is that page within XenForo? If so - this is the XF Bot Guard challenge gate (expected). If it isn't inside Cloudflare, you may have configured Cloudflare to challenge everyone - or guest users - and that should probably be disabled while using XF Bot Guard.

 

[azzurro]

Is that page within XenForo? If so - this is the XF Bot Guard challenge gate (expected). If it isn't inside Cloudflare, you may have configured Cloudflare to challenge everyone - or guest users - and that should probably be disabled while using XF Bot Guard.

Yes, it's a box within a XenForo page. Clicking through loads the page as expected.

 

[zeeb0t]

Yes, it's a box within a XenForo page. Clicking through loads the page as expected.

Perfect. Without the browser validation occuring beforehand, this is expected behaviour.

 

[Chromaniac]

so yeah. interstitial was not working for me even after disabling the edge caching. i guess it is related to the issue already discussed on this page? i have disabled the new option and would see if it gets resolved after the next update. thanks! i see that this addon is already being recommended by one of the xf developer.

 

[zeeb0t]

so yeah. interstitial was not working for me even after disabling the edge caching. i guess it is related to the issue already discussed on this page? i have disabled the new option and would see if it gets resolved after the next update. thanks! i see that this addon is already being recommended by one of the xf developer.

Quite possibly.

I am almost done testing a patch for this.

The patch does a few things:

• It moves the interstitial bootstrap collect handling earlier in the XenForo request lifecycle. The interstitial collect request is intentionally a special pre-session / pre-validation request, so it should not depend on the normal XenForo POST/session/CSRF flow being fully established first.
• It adds a small back-off-and-retry sequence for the interstitial collect POST, so a transient failure does not leave the visitor stuck on the validating screen.
• If the interstitial collect still fails after those retries, it redirects the visitor to the full XF Bot Guard challenge route. From there, they can complete the configured CAPTCHA provider if required, and the normal non-interstitial fingerprint/collect flow can run instead.

So the aim here is not to weaken the validation. It is to make this particular edge case recover cleanly instead of leaving the visitor stuck.

For what it is worth, I was not able to reproduce this manually, but after checking my own traffic logs I can confirm that /bot-guard/collect was returning 400 responses on my own forum roughly 4.6% of the time. So the issue is real, but it appears to be environmental and timing-sensitive rather than something that happens consistently on every install. It may depend on the browser, extensions, privacy settings, Cloudflare/proxy timing, server timing, or some combination of those.

Once I finish testing the patch, I will push an update.

 

[zeeb0t]

so yeah. interstitial was not working for me even after disabling the edge caching. i guess it is related to the issue already discussed on this page? i have disabled the new option and would see if it gets resolved after the next update. thanks! i see that this addon is already being recommended by one of the xf developer.

p.s., thanks for the heads up about the xf dev. Amazing stuff

 

[Chromaniac]

not sure if it is a bug. running the latest version. robots page does not show google branding for googlebots anymore lol. bingbot appears just fine. i have tried enabling and disabling the Allow verified search crawlers and fetchers option and behavior remains same. not sure if it is clashing with Known Bots | XenForo community addon.

 

[zeeb0t]

not sure if it is a bug. running the latest version. robots page does not show google branding for googlebots anymore lol. bingbot appears just fine. i have tried enabling and disabling the Allow verified search crawlers and fetchers option and behavior remains same. not sure if it is clashing with Known Bots | XenForo community addon.

Expected behaviour, or at least not a functional bug.

XF Bot Guard updates XenForo’s session activity bot state, similar to what Known Bots does, but for a slightly different purpose. It is not just trusting the user-agent label. It is trying to make the session activity reflect Bot Guard’s verified view of the visitor.

That means two things:

1. Traffic claiming to be Googlebot, Bingbot, etc. can be moved out of the “identified bot” state if it does not verify properly.
2. Traffic that XenForo did not identify as a bot can still be marked accurately if Bot Guard verifies it against the crawler/vendor’s published IPs/CIDRs.

This matters because plenty of bad traffic spoofs Googlebot-style user agents. Bot Guard is intentionally not treating “says it is Googlebot” as the same thing as “is actually Googlebot”.

So in this case, the missing Google branding is most likely just a cosmetic/session label overlap between XF Bot Guard and Known Bots. Known Bots appears to use the nicer branded label, while XF Bot Guard writes a more standardised bot name based on its own verified classification.

The “Allow verified search crawlers and fetchers” option controls how verified crawlers are allowed through Bot Guard. It does not control whether Known Bots’ branded display label wins on the robots page.

 

[Chromaniac]

alright. it is kind of getting a bit confusing. should i keep both the bot source options enabled now (Allow known verified crawler headers where determinable and Allow verified search crawlers and fetchers)? or one of them is enough lol.