It also batches the maximim it will delete in a single run, seems like you are exceeding that volume in growth. I will revise this approach and provide a patch today.
XF Bot Guard 1.4.1
- Thread starter zeeb0t
- Start date
It also batches the maximim it will delete in a single run, seems like you are exceeding that volume in growth. I will revise this approach and provide a patch today.
Ok thanks. Will come back to you in a few hours. I have a clue I am chasing down but need some deeper thinking time to wiggle it all through.
This bundle helps a lot. The key detail is that the failures are specifically collector_nonce_context_mismatch, not a generic missing/reused nonce.
That means the collector token is present and valid enough to be parsed, but the context no longer matches when /bot-guard/collect runs. In this path Bot Guard binds the bootstrap collector token to the original page key, IP hash, and user-agent hash. Since you are not using Cloudflare and /online/ shows real IPs, I’m less suspicious of Cloudflare/proxy IP handling here. The next things I’d want to check are whether the posted page_key differs from the original request URI, whether the user-agent differs between the validating page and the collect POST, or whether the request is losing the bootstrap_interstitial=1 POST flag and going through the normal controller path instead of the early bootstrap path.
The cache probe difference also makes sense: the support bundle was generated at 2026-06-14 22:46:11 UTC and did not have a cached probe result available. Your health page probe was checked later at 22:47:33 UTC and reported no shared full-page cache HIT evidence. So that does not currently point to page caching as the cause.
Could you send one more capture from a failing attempt? Feel free to DM to me as this might include additional context you don't want to share in the discussion thread here.
Open the event detail for one of the collector_nonce_context_mismatch rows, and copy the redacted event detail/support data from that specific event.
At this point I can also confirm one real issue on our side: if bootstrap collection fails and falls back to the full challenge, the fallback challenge path is not preserving the original return URL correctly. That explains why /solve can send you to the default fallback route instead of the page that triggered validation. This is not a major bug, more so symptomatic of whatever is causing the mismatch on your install for everyone - but I'll patch that anyway.
zeeb0t updated XF Bot Guard with a new update entry:
Minor Release: v1.3.0
Read the rest of this update entry...
Minor Release: v1.3.0
Read the rest of this update entry...
Excellent news! By the way, for your case, I’d suggest applying one of the preset quick settings or recommended settings, just in case any previous troubleshooting has moved things too far away from the defaults - and assuming you want to return to a baseline of the standard set of options.
Now that I have this working, I see another issue. I would recommend double-checking the ?_debug=1 view for a protected page. It looks like a bunch of queries are repeating unnecessarily. You can probably do some request-level caching to reduce this significantly.
For example, I see the following query referenced 8 times:
Unless I'm missing something, this appears to just be checking the same 1 IP several times.
There are a number of different queries that repeat like this.
For example, I see the following query referenced 8 times:
Code:
SELECT 1
FROM xf_zee_botguard_self_server_ip
WHERE ip_bin = ?
AND allow_until > ?
LIMIT 1There are a number of different queries that repeat like this.
Thanks - will check it out!
BTW, it is possible that some queries will be repeated, as the different checks are modular and run independently of each other. In some cases, they may reference the same tables again.
That said, I’ll look at implementing a per-request or per-page query cache to cut down on repeated queries where the data is unlikely to change during the same page load.
This is especially likely because Bot Guard runs across five different stages of XenForo’s load order during a single page request, so having a request-level cache would be useful.
Last edited:
There are also various queries that check the table schema. If you really need to do this, you should do some kind of persistent caching (like a datastore entry), because it's really unnecessary to query the information_schema 5 times on every request.
An inexpensive way to do the information_schema check is as follows:
First, reading the code comments, I see the reason for no caching is concern over existing 1.2.2 or lower installations while uploading the 1.2.3 files, but before pressing upgrade (clicking upgrade temporarily disables an add-on, mooting the concern).
I also see in Setup that part of the 1.2.3 upgrade is permanently making the schema "binary".
So we can do a really simple check. 1.2.3 and earlier do not have an in-database key that specifies the version number, but the version number can still be inferred. 1.2.3 changed
First, reading the code comments, I see the reason for no caching is concern over existing 1.2.2 or lower installations while uploading the 1.2.3 files, but before pressing upgrade (clicking upgrade temporarily disables an add-on, mooting the concern).
I also see in Setup that part of the 1.2.3 upgrade is permanently making the schema "binary".
So we can do a really simple check. 1.2.3 and earlier do not have an in-database key that specifies the version number, but the version number can still be inferred. 1.2.3 changed
XF::options()->zeeBotGuardVisitorRetentionDays to ->zeeBotGuardVisitorRetentionHours. Therefore we can just check if the zeeBotGuardVisitorRetentionHours key exists. If it does, we can assume "binary". If the key does not exist, do the previous behavior of information_schema lookups.
Good find, I had planned in 1.3.0 to drop direct upgrade jump support for pre-data-conversion versions. So that is likely to now occur in a very near upcoming release, which, as you point out, will drop a lot of queries. But still, there are other candidates which missed some intended DB static caches in the single run-time, and they will be patched.
@pegasus @azzurro @akok @smallwheels
The latest update should hopefully address each of your findings. Please let me know how you go
I did a few manual cron runs of XF Bot Guard cleanup and now it goes for a while and now it gets timed out with a 504.
i checked the rows from xenforo.xf_zee_botguard_event and don't see any decrease. I'll try again later to see if anything changed.
It is probably likely the cron is running and your web server gave up waiting. Are you seeing any errors in the error log?
Got it, looks like it did catch up as now manual runs complete quickly. According to Claude:
Total count is now roughly flat: 2,528,724 → 2,528,747 (+23 over the whole period). That's a big improvement — previously it was growing by ~80K. This suggests pruning is now roughly matching new inflow.
But "oldest" hasn't moved at all: still 2026-06-10 21:59:48, identical to the prior check. If pruning is matching inflow but the oldest timestamp is frozen, it looks like the cleanup is deleting from the "newer end of the backlog" (or wherever its query targets) but not actually reaching the oldest rows — so the table is staying the same size rather than shrinking toward the 24h target.
The 5 oldest rows are event_type verified_crawler_allow (4 rows) and collector_nonce_expired (1 row), all from 2026-06-10 22:01:44-45. If these event types have a different retention policy than the default 24h, that would explain why pruning isn't advancing past this point — otherwise, it looks like these types might be excluded from the cleanup query.
A 504 here doesn’t necessarily mean XF Bot Guard hit a PHP/XenForo error. Manual cron runs execute inside the admin web request, while cleanup is intentionally allowed to work in bounded batches for up to about 60 seconds. If the database has a backlog or the server/proxy timeout is shorter than that, the browser can see a 504 even though cleanup is still doing normal work and no XF error is logged.
The raw event row count also may not drop immediately. Cleanup removes matching event detail rows and normalized reason rows before deleting the parent rows from xf_zee_botguard_event.
Could you run this read-only query and send me the results? It tells us whether there are actually cleanup-eligible parent events, how old the main rows are, and whether cleanup may be spending time clearing child/detail rows first.
SQL:
SELECT
COUNT(*) AS total_events,
FROM_UNIXTIME(MIN(event_date)) AS oldest_event,
FROM_UNIXTIME(MAX(event_date)) AS newest_event
FROM xf_zee_botguard_event;
SELECT COUNT(*) AS event_reason_rows
FROM xf_zee_botguard_event_reason;
SELECT COUNT(*) AS event_detail_rows
FROM xf_zee_botguard_event_detail;Just ran those and got:
+--------------+---------------------+---------------------+
| total_events | oldest_event | newest_event |
+--------------+---------------------+---------------------+
| 2527930 | 2026-06-10 22:09:38 | 2026-06-15 06:22:26 |
+--------------+---------------------+---------------------+
+-------------------+
| event_reason_rows |
+-------------------+
| 1913065 |
+-------------------+
+-------------------+
| event_detail_rows |
+-------------------+
| 2526972 |
+-------------------+
So not sure it is the same issue. I temporarily moved to a different host low spec server two three days ago and was suddenly destroyed by 500 errors. I moved back and it continued for a day. And has calmed down.
I checked at all places but couldn't find any log explaining the 500 problem. There were also a lot of defaultAjaxError @ core-compiled.js issues.
I checked at all places but couldn't find any log explaining the 500 problem. There were also a lot of defaultAjaxError @ core-compiled.js issues.
Please run this again in an hour or so, and hopefully we see further reduction.
