XF Bot Guard 1.4.1

[zeeb0t]

There's not really much for a novice like me to be happy or unhappy with on those pages, so I just enabled it. But I noticed a UI problem:

Thanks, will look into this.

I also have a question. The reason I installed this addon is this: https://community.centminmod.com/th...ring-the-xenforo-forum-search-endpoint.33033/ - my forum access logs are showing bots hitting the /search/ endpoint, which they don't need to be doing, and it causes heavy load on the server.

Is this addon an appropriate tool/solution for that? I'm not seeing an option to specifically protect the /search/ endpoint. I only see the "Protected areas" option, and there's only the option to switch from "all public threads" to "custom path/route". That makes it seem like I'd have to give up most of the protection this addon provides if I want to specifically, and more so, protect the /search/ path.

And from what I can tell, it can't really hide the /search/ path from the bots or stop them from pinging it; it can only punish them for doing so. Most of the bots pinging it seem to be legit, and are welcome to the rest of the forum; I just don't want them hitting that one unnecessary and costly endpoint.

Yes, it can help with that, but I would separate two cases.

XF Bot Guard is appropriate when the problem is abusive or suspicious visitors repeatedly hitting expensive public routes, including XenForo search. If Protected areas is set to All public pages, then /search/ is already eligible for Bot Guard checks.

If you want a narrower scope, use Selected routes and include search plus whatever else you still want protected, such as threads and forums. The selected/custom modes are allow-lists, not an “add this one route on top of all public pages” setting.

That said, if your goal is “legitimate crawlers are welcome elsewhere, but should not request /search/ at all”, Bot Guard is not the best primary tool. It works at the XenForo/PHP layer: it can score, challenge, log and optionally export abusive IPs to Cloudflare, but it does not hide the URL or stop the first request from reaching XenForo outright.

For that policy, I would handle /search/ before XenForo:

• Add/confirm a robots.txt rule for compliant crawlers:

User-agent: *
Disallow: /search/

• Use Cloudflare WAF/rate limiting or a web-server rule if you want hard block/challenge/throttle behaviour for /search/.
• Consider disabling guest search in XenForo if public search is not needed.

So: keep Bot Guard for the broader bot-abuse problem, but use robots/Cloudflare/web-server/XenForo search permissions for the precise “don’t crawl search” rule.

 

[thomas1]

First, thank you for this wonderful add-on and for all the time and effort you have poured into it!

And my apologies if I am asking something that has already been answered!


I'm trying to allow Internet Archive / Wayback through Bot Guard 1.4.1 without weakening normal guest protection. Cloudflare identifies the Wayback fetcher as Internet Archive, AS7941. One observed request came from 207.241.226.157 with:

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/148.0.0.0 Safari/537.36

Reverse DNS for that IP is:

wwwb-spn17.us.archive.org

and it resolves back to 207.241.226.157.

I added archive.org to "Additional trusted crawler reverse DNS domains", but a fresh Wayback request was still challenged.

Bot Guard logged:

challenge_required

Risk score: 75

missing_or_unconfirmed_fingerprint +25
no_server_cookie_continuity +20
no_server_js_confirmation +30

Metadata also shows:

"full_scoring_ran": true

Known crawler trust is enabled, origin lockdown is acknowledged, and the verified crawler IP allowlist is enabled and healthy.

What is the intended way to whitelist or trust Internet Archive / Wayback?

Is archive.org under "Additional trusted crawler reverse DNS domains" enough, or does that option require another crawler signal, such as a recognised User-Agent or Signature-Agent pattern?

I would prefer not to lower the challenge threshold or weaken the normal scoring rules to accommodate Wayback.

Thanks for any pointers!

 

[smallwheels]

What is the intended way to whitelist or trust Internet Archive / Wayback?

Don't know if it is the intended way but as AS7941 seems to be exclusive to Internet Archive you could try to whitelist the ASN in the settings:

 

[thomas1]

AS7941

Thank you. I did miss the "Excluded IPs" field.

However, adding their decimal ASN didn't change a thing; Wayback Machine is still being challenged by Bot Guard.

 

[zeeb0t]

First, thank you for this wonderful add-on and for all the time and effort you have poured into it!

And my apologies if I am asking something that has already been answered!


I'm trying to allow Internet Archive / Wayback through Bot Guard 1.4.1 without weakening normal guest protection. Cloudflare identifies the Wayback fetcher as Internet Archive, AS7941. One observed request came from 207.241.226.157 with:

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/148.0.0.0 Safari/537.36

Reverse DNS for that IP is:

wwwb-spn17.us.archive.org

and it resolves back to 207.241.226.157.

I added archive.org to "Additional trusted crawler reverse DNS domains", but a fresh Wayback request was still challenged.

Bot Guard logged:

challenge_required

Risk score: 75

missing_or_unconfirmed_fingerprint +25
no_server_cookie_continuity +20
no_server_js_confirmation +30

Metadata also shows:

"full_scoring_ran": true

Known crawler trust is enabled, origin lockdown is acknowledged, and the verified crawler IP allowlist is enabled and healthy.

What is the intended way to whitelist or trust Internet Archive / Wayback?

Is archive.org under "Additional trusted crawler reverse DNS domains" enough, or does that option require another crawler signal, such as a recognised User-Agent or Signature-Agent pattern?

I would prefer not to lower the challenge threshold or weaken the normal scoring rules to accommodate Wayback.

Thanks for any pointers!

Thanks, that all makes sense.

There are two separate things here:

• archive.org in Additional trusted crawler reverse DNS domains is not enough for this request. That option is not a plain rDNS allowlist. Bot Guard only uses it when the request also has a plausible crawler/fetcher user-agent, then reverse DNS matches, then forward DNS confirms the same IP. The user-agent shown is a generic HeadlessChrome UA, so it falls through to normal guest scoring.
• AS7941 does look like the better signal for the specific IP you captured. 207.241.226.157 is in Internet Archive address space. However, ASN entries in Excluded IPs/CIDRs/ASNs are not matched live from Cloudflare’s ASN label/header. Bot Guard expands configured decimal ASNs into local CIDR ranges via scheduled refresh / Health maintenance, then matches requests against that local cache.

So if 7941 was added and Wayback was still challenged, the next thing to check is whether the AS7941 range cache refreshed successfully, had not run yet, failed, expired, or did not include that IP at the time of the test.

Could you DM me the Bot Guard support bundle plus the event data for that Wayback challenge row? Please don’t post the bundle publicly.

For an immediate test, adding the exact IP or a narrow CIDR should take effect straight away. I would not lower the challenge threshold for this.

 

[thomas1]

Thank you! DM sent.

Adding that particular Wayback IP address to the "Excluded IPs" field worked.

 

[zeeb0t]

Thank you! DM sent.

Adding that particular Wayback IP address to the "Excluded IPs" field worked.

Thank you, that helps.

The support bundle shows Bot Guard is healthy overall, and it also shows the ASN path is now populated:

ASN range cache: 1 configured ASN(s) / 2 active ranges

So the most likely explanation is timing. Adding 7941 is the right kind of network-level exception for Internet Archive/Wayback, but ASN entries are not applied instantly. Bot Guard first has to expand the ASN into local CIDR ranges via the scheduled refresh / Health maintenance path, then request-time matching uses that local cache.

That also explains why adding the exact Wayback IP worked immediately: exact IP and CIDR entries are local and immediate; ASN entries need the range cache.

So, you should be able to use 7941 in Excluded IPs/CIDRs/ASNs, then make sure Health shows active ASN ranges before retesting.

 

[MaximilianKohler]

Shouldn't bots like the Wayback Machine be whitelisted by default? I disabled bot guard after reading that it blocks archive.org by default.

 

[smallwheels]

Shouldn't bots like the Wayback Machine be whitelisted by default? I disabled bot guard after reading that it blocks archive.org by default.

Everybody is different and possibly not everybody wants his forum to be archived there. But it is not hard to configure it if you like. In the end Bot Guard is a tool that can do a lot but as almost any tool that can do a lot you have to configure it to fit your individual needs - which is possible. Also, Wayback Machine has not been very good at archiving forums in the past anyway from my experience.

 

[thomas1]

Shouldn't bots like the Wayback Machine be whitelisted by default? I disabled bot guard after reading that it blocks archive.org by default.

If you follow the instructions above, Wayback Machine won't be challenged anymore. We have a lot of backlinks from Wikipedia and excluded Wikimedia's ASN as well.

 

[zeeb0t]

Shouldn't bots like the Wayback Machine be whitelisted by default? I disabled bot guard after reading that it blocks archive.org by default.

I understand the concern, but this is a deliberate conservative default rather than an oversight.

Bot Guard does include friendly defaults for widely expected search crawlers/fetchers where there is a strong verification path and broad site-owner expectation. Wayback/Internet Archive is a little different: some admins absolutely want their forums archived, and some do not. Its fetches can also present as browser-like automated HeadlessChrome traffic rather than a clean, universally trusted crawler identity.

So yes, Bot Guard detecting that as bot-like/automated traffic and challenging it is basically the product doing what it is meant to do.

Automatically allowing Wayback would effectively opt every forum into archive fetches. That feels more like a site policy choice than a safe universal default.

For admins who do want to allow it, Bot Guard gives targeted options which have been explored in the thread.

That is really the key distinction: Bot Guard is not trying to decide that every automated fetcher is morally good or bad. It is enforcing a conservative protection policy, then giving the admin controls to decide which automated traffic their own site should trust.

Wayback is a good example. Some forums want it. Some definitely do not. So I would rather make that an explicit admin choice than silently whitelist it for everyone.

 

[MaximilianKohler]

If you follow the instructions above, Wayback Machine won't be challenged anymore

If Wayback is blocked or hindered, then I would expect other useful bots/sites to be as well, including other archive sites.

Based on the recent discussion, it doesn't seem easy to detect which bots/sites may be getting blocked, and then whitelist them. And since my primary goal was to reduce /search/ queries, it doesn't seem worth the effort for me to spend time trying to detect and whitelist that sort of thing.

 

[thomas1]

If Wayback is blocked or hindered, then I would expect other useful bots/sites to be as well, including other archive sites.

Based on the recent discussion, it doesn't seem easy to detect which bots/sites may be getting blocked, and then whitelist them. And since my primary goal was to reduce /search/ queries, it doesn't seem worth the effort for me to spend time trying to detect and whitelist that sort of thing.

I can relate to your concerns; however, Bot Guard's Raw Event Log is a nifty tool for checking which requests are being challenged. It does require a bit of tinkering, but it gives you a lot of control over who can and who can't access your site. In conjunction with CF, it has removed large chunks of unwanted traffic from our website while still allowing legitimate and welcome visitors through.

 

[Yoshiro]

Passing along a bug report from our IT team:

New install of XF Bot Guard 1.4.1
Error seen:

route is VARCHAR(100) while snapshot metric_key is VARBINARY(100). A 100-character UTF-8 route containing multibyte characters can exceed 100 bytes, causing MySQL error 1406 in RoutesCollector.

We widened the route column on our side, it was taking 102 bytes

[root]# mariadb forums_xenforo -e "
SHOW COLUMNS FROM xf_zee_botguard_event LIKE 'route';


SELECT
event_id,
OCTET_LENGTH(route) AS route_bytes,
route,
LEFT(request_path,200) AS request_path
FROM xf_zee_botguard_event
WHERE OCTET_LENGTH(route) > 100
ORDER BY route_bytes DESC
LIMIT 30;
"
+-------+--------------+------+-----+---------+-------+
| Field | Type         | Null | Key | Default | Extra |
+-------+--------------+------+-----+---------+-------+
| route | varchar(100) | NO   | MUL |         |       |
+-------+--------------+------+-----+---------+-------+
+----------+-------------+--------------------------------------------------------------------------------------------------------+--------------+
| event_id | route_bytes | route                                                                                                  | request_path |
+----------+-------------+--------------------------------------------------------------------------------------------------------+--------------+
|    13415 |         102 | threads/killing-floor-3-–-backend-maintenance-announcement-and-hotfix-1-changelog.2342981/post-23696   | /index.php   |
|    13416 |         102 | threads/killing-floor-3-–-backend-maintenance-announcement-and-hotfix-1-changelog.2342981/post-23696   | /index.php   |
+----------+-------------+--------------------------------------------------------------------------------------------------------+--------------+
[root]# mariadb forums_xenforo -e "
ALTER TABLE xf_zee_botguard_report_snapshot_metric
MODIFY metric_key VARBINARY(400) NOT NULL DEFAULT '';
"
[root]# mariadb forums_xenforo -e "
SHOW COLUMNS FROM xf_zee_botguard_report_snapshot_metric LIKE 'metric_key';
"
+------------+----------------+------+-----+---------+-------+
| Field      | Type           | Null | Key | Default | Extra |
+------------+----------------+------+-----+---------+-------+
| metric_key | varbinary(400) | NO   | PRI |         |       |
+------------+----------------+------+-----+---------+-------+

 

[zeeb0t]

Passing along a bug report from our IT team:

New install of XF Bot Guard 1.4.1
Error seen:

route is VARCHAR(100) while snapshot metric_key is VARBINARY(100). A 100-character UTF-8 route containing multibyte characters can exceed 100 bytes, causing MySQL error 1406 in RoutesCollector.

We widened the route column on our side, it was taking 102 bytes

[root]# mariadb forums_xenforo -e "
SHOW COLUMNS FROM xf_zee_botguard_event LIKE 'route';


SELECT
event_id,
OCTET_LENGTH(route) AS route_bytes,
route,
LEFT(request_path,200) AS request_path
FROM xf_zee_botguard_event
WHERE OCTET_LENGTH(route) > 100
ORDER BY route_bytes DESC
LIMIT 30;
"
+-------+--------------+------+-----+---------+-------+
| Field | Type         | Null | Key | Default | Extra |
+-------+--------------+------+-----+---------+-------+
| route | varchar(100) | NO   | MUL |         |       |
+-------+--------------+------+-----+---------+-------+
+----------+-------------+--------------------------------------------------------------------------------------------------------+--------------+
| event_id | route_bytes | route                                                                                                  | request_path |
+----------+-------------+--------------------------------------------------------------------------------------------------------+--------------+
|    13415 |         102 | threads/killing-floor-3-–-backend-maintenance-announcement-and-hotfix-1-changelog.2342981/post-23696   | /index.php   |
|    13416 |         102 | threads/killing-floor-3-–-backend-maintenance-announcement-and-hotfix-1-changelog.2342981/post-23696   | /index.php   |
+----------+-------------+--------------------------------------------------------------------------------------------------------+--------------+
[root]# mariadb forums_xenforo -e "
ALTER TABLE xf_zee_botguard_report_snapshot_metric
MODIFY metric_key VARBINARY(400) NOT NULL DEFAULT '';
"
[root]# mariadb forums_xenforo -e "
SHOW COLUMNS FROM xf_zee_botguard_report_snapshot_metric LIKE 'metric_key';
"
+------------+----------------+------+-----+---------+-------+
| Field      | Type           | Null | Key | Default | Extra |
+------------+----------------+------+-----+---------+-------+
| metric_key | varbinary(400) | NO   | PRI |         |       |
+------------+----------------+------+-----+---------+-------+

Thank you for this bug report. I have prepared a fix into the next release.

 

[lantek]

The next version includes a new type of quick setting:

A new Capacity Fit Profile at the top of the existing XF Bot Guard Quick settings page. Starts from Balanced production, measures the current install using lightweight in-product capacity/readiness signals, assign known feature costs, and preview/apply the strongest viable Bot Guard configuration within the server’s current constraints.

Basically the TL/DR quick setting that is intelligent in that it samples your current server load and suggests a custom profile that will hopefully work right out of the box for you.

Any ETA on the next version? Trying to decide if I should install the current or wait for the new.

Thanks!

 

[Waterfall]

How would one go about resolving this? I've ran all the cron entries (including of course the ones relevant to these) and no go.

The downloaded local IPLocate country and ASN databases can be read, but the generated compact geo index is missing, stale, or failed validation. Request handling continues through the validated MMDB reader fallback. A managing admin opening the Health check can queue repair work, and the MMDB refresh or geo index cache warm cron will keep retrying.

Latest dashboard snapshot is more than 2 hours behind the latest eligible completed hour.

 

[smallwheels]

@zeeb0t : Sadly, there's a conflict between Bot Guard and IP Threat Monitor v1.0.49, which implemented a forced captcha for selected countries and ASNs. It seems, that once a visitor hits the captcha criteria IP Threat Monitor redirects to the Captcha Page while Bot Guard at the same time does want to make it's JS tests of the browser and does a redirect for that as well. This ends in a dead lock, as both add ons try to redirect at the same time to different locations, and the browser reporting a redirect error. The address bar in the browser shows this in such a situation:



which leads to said assumption. Would it be possible to take into account the mechanism of IP Threat Monitor in Bot Guard to solve the conflict?

 

[Dragonfruit]

@zeeb0t

I'm having a redirect bug on a XF 2.2 board, I had Claude analyze it, and this is what they gave me, they identified the issue in this addon:

Bug: visitors are sent to index.php?index.php after passing the challenge (1.4.1)

Symptom: After solving the captcha, the visitor lands on https://example.com/index.php?index.php instead of the page they were trying to reach.

When it happens: The challenge is triggered on a request whose URL is plain /index.php with no query string. The server rewrites can produce this, and so can a direct hit on the front controller.

Root cause:

1. RequestContext::fallbackRouteFromUri() calls RequestSafety::routeFromPathAndQuery('/index.php', '', true). With an empty query, that function falls through to return trim($path, '/'). So the “route” comes back as the literal string index.php, which is the entry script, not a real route.
2. ReturnUrl::fromContext() then sees two things:
   • isBareIndexReturn() is true, because the return URL is just index.php.
   • looksLikeRoute('index.php') is true.
   So it rebuilds the return URL as 'index.php?' . $route, which gives index.php?index.php.
3. That value passes validate(). It gets stored in the session (zeeBotGuardReturnUrl) and in the signed zee_bg_return ticket. Challenge::resolveReturnUrl() then redirects to it after a successful solve.

I confirmed this by running ReturnUrl::fromContext() from 1.4.1 with request_uri = '/index.php', an empty query and route = 'index.php'. It returns index.php?index.php.

Proposed fix (both changes are in Service/ReturnUrl.php; diff attached):

1. In looksLikeRoute(), return false when the route is, or ends in, index.php. This stops the rebuild from ever producing the bad URL.
2. In validate(), collapse index.php?index.php back to the bare index.php. This cleans up bad return targets already stored in sessions or tickets, and covers a visitor who gets challenged again while on the broken URL.