TL;DR: XenForo 2.3.10 makes an AI agent understand the framework. It doesn't stop the agent shipping add-ons that look fine and are silently broken. This thread is the discipline layer on top of that: turn every bug into a deterministic guard, gate on static analysis, review across multiple models, verify in a real browser, keep memory across context resets, secure by default, and self-test the tooling itself. It isn't an "AI is fine" post, it's a repeatable process for people who already ship and want fewer post-release fires, with a human who can read the code still signing off at the end. The attached kit lets you adopt the core of it in about ten minutes. Full reasoning and examples below.
2.3.10 gave us a genuinely useful foundation - the agent skills via
Up front, because it matters: the skeptics here are not wrong. Unsupervised "vibe coding" really does produce unmaintainable architecture, silent bugs, and security holes. So let me be precise about scope rather than wave it away. Most of the silent-failure and regression concerns are catchable with process rather than left to hope. The comprehension and long-term-maintenance concerns are not - those still need a developer who understands the output. The honest question is not whether discipline makes AI-assisted work as good as a careful human team; it is whether discipline can narrow the gap enough that the work is trustworthy to ship, and how much human review that still takes. In my experience it takes a lot - and the point of the process below is to make that review tractable instead of endless.
This is also not an argument against the room - it is mostly the room's own conclusions, made repeatable. @Slavik found that an add-on vibe-coded under his strict supervision was something even the AI itself could not tell apart from his hand-coded one. @ES Dev Team described needing "a human brain to fill the gap". @Forsaken put current models at most at junior-developer level. @jul asked for exactly this: guidelines for proper AI-assisted add-on development plus testing. The practices below are a worked version of those.
The most dangerous category in XenForo is what I will call "ships green but renders dead": it passes every test, looks fine in dev, and is silently broken on a real install. A backend example first, since that is the harder case:
Concretely, a guard is small. The asset-ship one is roughly: collect every asset path referenced in the templates, check each against the build manifest, fail with the missing list if any is absent. Then - and this is the part people skip - give it a known-BAD fixture it MUST flag and a known-GOOD fixture it MUST pass, and run both in a self-test, so a future refactor that blinds the guard fails loudly. A guard that silently stops firing is worse than none. (The attached kit's
Honest limitation: bug-to-guard is reactive - it locks down a class only after it has bitten once. It does not prevent novel mistakes; it stops the known ones recurring. The judgement that prevents the novel ones is still human. I have about a dozen of these now, each added the day a specific bug shipped or nearly did. If you adopt only one thing from this whole post, make it this habit plus the static-analysis gate in section 2: together they catch the most regressions for the least overhead, and you grow the rest as bugs teach you which guards you need.
The honest caveat: agents reviewing agents share correlated blind spots - a panel reduces them, it does not eliminate them. That is exactly why the deterministic guards (which do not "reason", so cannot be talked out of a finding) and a human backstop sit underneath the panel. And a multi-model panel is a real running cost - several frontier models per review is dollars and compute, not just time - so reserve the full panel for the changes that warrant it.
And what you feed the model is itself a security decision: keep production secrets and member PII out of the prompt, and remember that a local or self-hosted model - as some here, including @ES Dev Team, already run - keeps the codebase off a third party's servers, which addresses the data-harvesting risk @vbresults raised. This cuts both ways, honestly. The adversarial multi-model audit in section 3 is how AI-assisted review surfaces security issues - and the same tooling also bakes them in: @JamesBrown described showing an add-on he and the model had built to a professional software engineer before going live, who spent three weeks stripping out the security exploits they had baked in. That is not an argument against the tool - it is the entire argument for the gate. The human-in-the-loop part is the point either way.
And the maintenance tail, which is the loudest complaint in the other thread: the same guards run on every upgrade, and upgrade-path testing - install the oldest build, upgrade through each version, diff the schema against a fresh install - is part of the gate. Two specifics, since they are the loudest failures in the other thread (add-ons that cannot be uninstalled, leave orphaned tables, or warn about changed file hashes): uninstall must reverse every schema change it ever made - verified by diffing a fresh-install schema against a post-uninstall one - and the release ZIP must be produced by the framework's own build tooling rather than hand-assembled, since hand-assembly is where the bundled hashes.json and the package contents drift out of sync. The discipline does not end at release; that is usually where the slop starts.
And if you are on the receiving end - installing a stranger's add-on - here is what you can reasonably check or ask for, whatever built it, without reading a line of code: that it uninstalls cleanly on a test board (no orphaned tables or columns left behind), that the ACP file-health check passes (XF compares the installed files against the add-on's own bundled hashes.json - hand-assembling a package instead of using XenForo's build tooling is where stale or mismatched hashes creep in), and that any AI assistance is disclosed. That is the same discipline above, seen from the buyer's side.
I suspect most of this generalizes beyond XenForo - turn bugs into guards, gate on static analysis, review adversarially, verify at runtime, persist memory, secure by default, test your own tooling - but I only have XenForo evidence, so I will not over-claim it. XF just happens to have a large silent-failure surface (templates, phrases, style properties, the build-to-ZIP boundary) that rewards the approach.
I would genuinely like to know where this still falls short - especially any case where a gate gave false confidence and something shipped broken anyway. That is the failure mode I most want to close, and the people in these threads have clearly hit edges I have not.
One limit I already know: a guard only exists once its bug has bitten, so a newcomer adopting this starts with almost none and is still exposed to the traps they have not hit yet. The attached kit (
The rest of the same kit is the complete set I actually run - every review agent, every guard script, the full skill set and project-instructions file, all sanitized the same vendor-neutral way. So start with
Threads this builds on: What's new in XenForo 2.3.10 and So much AI, so many add-ons... and so many bugs.
2.3.10 gave us a genuinely useful foundation - the agent skills via
xf-dev:install-ai-agents, the phrase tooling, and the xf-make commands. Credit where it is due. This thread is about the layer that goes on top: how to take an agent from "writes plausible code" to "ships maintainable, reviewed add-ons", and which of the concerns raised in these threads this discipline actually addresses - and which it does not.Up front, because it matters: the skeptics here are not wrong. Unsupervised "vibe coding" really does produce unmaintainable architecture, silent bugs, and security holes. So let me be precise about scope rather than wave it away. Most of the silent-failure and regression concerns are catchable with process rather than left to hope. The comprehension and long-term-maintenance concerns are not - those still need a developer who understands the output. The honest question is not whether discipline makes AI-assisted work as good as a careful human team; it is whether discipline can narrow the gap enough that the work is trustworthy to ship, and how much human review that still takes. In my experience it takes a lot - and the point of the process below is to make that review tractable instead of endless.
This is also not an argument against the room - it is mostly the room's own conclusions, made repeatable. @Slavik found that an add-on vibe-coded under his strict supervision was something even the AI itself could not tell apart from his hand-coded one. @ES Dev Team described needing "a human brain to fill the gap". @Forsaken put current models at most at junior-developer level. @jul asked for exactly this: guidelines for proper AI-assisted add-on development plus testing. The practices below are a worked version of those.
1. Turn every bug into a guard (and test the guards)
The single highest-leverage habit. When the agent ships a bug, you do not just fix it - you write a deterministic check that fails the build if that class of bug ever comes back. This matters more for AI than for hand-written code: an agent reintroduces the same mistake far more often than a human does, especially after its context compacts and it "forgets" the lesson. The guard is the memory the model loses.The most dangerous category in XenForo is what I will call "ships green but renders dead": it passes every test, looks fine in dev, and is silently broken on a real install. A backend example first, since that is the harder case:
- An XFCP class extension that tightens a core method's signature. Invisible on a single-add-on site, but a compile-time FATAL on any site running a second add-on that extends the same class - it takes down the ACP, and it never showed up in your testing because you only had your own add-on installed.
- An admin action reachable by direct URL with no permission gate - a permission-coverage check flags it; a click-through test that never types the URL does not.
- A JS/CSS asset referenced by a template but missing from the build manifest - the dev server serves it, every "the element exists" test passes, and the feature is simply absent for end users.
- A reference to a FontAwesome name, an
@xf-style property, or a phrase that does not resolve - renders as blank space / unstyled / the raw key, no error logged. (@gigipmc flagged a related trap in XF's own phrase cleaner: it drops in-use phrases from a template modification if the add-on is disabled when you run it - my takeaway: run the cleaner only with the add-on enabled, and add a check that fails if an in-use phrase would be dropped.)
Concretely, a guard is small. The asset-ship one is roughly: collect every asset path referenced in the templates, check each against the build manifest, fail with the missing list if any is absent. Then - and this is the part people skip - give it a known-BAD fixture it MUST flag and a known-GOOD fixture it MUST pass, and run both in a self-test, so a future refactor that blinds the guard fails loudly. A guard that silently stops firing is worse than none. (The attached kit's
start-here/ folder has a tiny, generalized version of exactly this, with the self-test, so you can see the whole pattern end to end. The self-test is the honest part - it proves the guard still fires
Code:
$ bash example-guard.test.sh
Asset-ship guard self-test
==========================
PASS: FLAGS a referenced asset missing from the manifest (exit 2)
PASS: PASSES when every referenced asset ships (exit 0)
==========================
Self-test: 2 passed, 0 failedHonest limitation: bug-to-guard is reactive - it locks down a class only after it has bitten once. It does not prevent novel mistakes; it stops the known ones recurring. The judgement that prevents the novel ones is still human. I have about a dozen of these now, each added the day a specific bug shipped or nearly did. If you adopt only one thing from this whole post, make it this habit plus the static-analysis gate in section 2: together they catch the most regressions for the least overhead, and you grow the rest as bugs teach you which guards you need.
2. Static analysis as a hard gate, not a suggestion
Static analysis at the strictest level, plus strict rules, as a release blocker. One hard-won caveat that is itself a lesson in not trusting a single tool: the strict analyzer will happily ADD a return type to an overridden core method to satisfy itself - which is the exact change that fatals a co-installed add-on (see the XFCP trap above). So even the static-analysis gate needs its own guard. Tools check tools; nothing is trusted blindly. This gate plus the release-blocking checks is a worked version of the GitHub-gated pipeline @vbresults proposed in the other thread - automated checks that refuse the release if they fail - and answers @jul's ask for testing that runs after the code is written.3. Adversarial multi-agent (and multi-model) review
This is the room's own recommendation, not mine. @Chris D pointed at a "broad panel" of agents (Aaron Francis'scounselors fan-out). @eva2000 makes the case for multi-MODEL consensus - never relying on one model - and reports finding "10-50% more insights" that way than with a single model. @Alpha1 described the loop almost exactly: let agents find issues and describe expected vs real behavior, then let separate agents try to DISPROVE each issue, then reproduce the survivors on a dev install. @MattW, separately, reported auditing with a three-model consensus (Opus 4.8, gpt-5.5 and Deepseek V4 Pro) and, in a docker install, finding 3 permission bypasses. That is the pattern: fan out across distinct lenses (correctness, security, XF-idiom, performance), then adversarially verify - a second pass tries to refute each finding, and only survivors get actioned.The honest caveat: agents reviewing agents share correlated blind spots - a panel reduces them, it does not eliminate them. That is exactly why the deterministic guards (which do not "reason", so cannot be talked out of a finding) and a human backstop sit underneath the panel. And a multi-model panel is a real running cost - several frontier models per review is dollars and compute, not just time - so reserve the full panel for the changes that warrant it.
4. Runtime and visual verification, not "the element exists"
Code-confidence is not behaviour-confidence. Drive a real browser (Playwright, as @tajhay described using via Claude Code) and assert things actually render and work: the icon paints a real glyph with a non-zero box, the instant filter actually hides non-matching rows, there is no horizontal overflow at 360px, and the error log is empty afterwards. The subtle one worth a dedicated check: an element with the correct CSS class but placed in the wrong DOM container - it is styled correctly, passes every existence and style check, and still renders in the wrong place. "The button exists" tests pass green on a completely dead - or misplaced - button.5. Persistent memory (the compaction problem)
@tajhay named the failure mode precisely: after a compaction the agent "becomes demented and forgets everything it learned about how XenForo works". @eva2000 and @Anthony Parsons both describe the same fix - a per-project decision journal so a fresh session knows what was done and why. The structure that works: an always-loaded instruction file, plus a store of hard-won lessons, plus per-add-on notes. The knowledge survives a context reset instead of being re-learned and re-broken every session. Most "the agent forgot the rule and reintroduced the bug" problems are really memory-architecture problems.6. Security and backend rigor (the part everyone is rightly worried about)
"Outsource cheap, get hacked" is a real risk, and @terere is right that agents are weaker on backend than frontend. Bake the defenses in as checks rather than intentions: CSRF coverage on every POST/AJAX path, a permission-gate check on every data-modifying action, prepared statements everywhere, repository/service layering instead of raw SQL to create content. For outbound HTTP specifically - one concrete mechanism, not an adjective - read the response body bounded at fetch time (cap the stream as it is read), never a post-hoc length check after the whole body is already in memory; otherwise a hostile remote exhausts your memory before the "cap" runs.And what you feed the model is itself a security decision: keep production secrets and member PII out of the prompt, and remember that a local or self-hosted model - as some here, including @ES Dev Team, already run - keeps the codebase off a third party's servers, which addresses the data-harvesting risk @vbresults raised. This cuts both ways, honestly. The adversarial multi-model audit in section 3 is how AI-assisted review surfaces security issues - and the same tooling also bakes them in: @JamesBrown described showing an add-on he and the model had built to a professional software engineer before going live, who spent three weeks stripping out the security exploits they had baked in. That is not an argument against the tool - it is the entire argument for the gate. The human-in-the-loop part is the point either way.
The style and template frontier (where it is hardest)
This matters most exactly where @Chris D was candid that the official tooling cannot yet help: template work via agents is "still rough", and style work, he said, is "not something we're doing very often so ironically we don't always have the expertise necessary to steer it in the right direction." Agents are weakest here - @Alpha1's experience of the agent dumping "a massive extra.css" instead of editing style properties, and knowing the palette and the first handful of colours but not the rest, is the common failure. These are silent-visual failures (an unstyled element, a blank icon, a block in the wrong container) that throw no error - which is exactly why the@xf- style-property guard plus real browser verification (does it paint? is it in the right container?) are the safety net that is otherwise missing. Concretely: constrain the agent to editing style properties and LESS variables, and fail the build on any unresolved @xf- token or a net-new extra.css dump - which catches exactly the "massive extra.css" failure @Alpha1 described. This is the current frontier for everyone, me included.Does this actually produce shippable quality?
I will be candid: first-draft AI output is lower quality than careful hand-written code. The value of the workflow is not that the delta is zero - it is that the gates surface that delta before users do, and keep surfacing it on every change. A useful mental model is that the agent is a fast, tireless junior, and the workflow is the senior review that never gets bored of the boring checks. But the non-negotiable part that no gate replaces is a human who reads the resulting code, can explain it, and signs off. The gates make that review tractable; they do not substitute for it. And to be blunt about the limit: these gates RAISE the floor for a reviewer who can already read the code - in the hands of someone who cannot, the green checkmarks are false confidence, which is arguably worse than none. That is the "human in the loop" the more skeptical posters here endorse, and I agree with them. Honest cost: this is real overhead - multiple review passes, browser runs, upgrade-path tests, a dozen guards. It is worth it for code you will ship to and maintain for others; it is overkill for a throwaway or personal script, and I will not pretend otherwise. I make no speed claim either: the gates add time, and the payoff is fewer post-release fires, not a faster first draft. @ES Dev Team report that even their heavily supervised, human-in-the-loop setup saw AI "provided an initial boost then dragged us down", with quality always lower - I take that as the honest cost of this work, not proof it cannot be shipped: the gates keep that drag in front of you instead of in your users' hands. And shipping to others is a support duty, not just a code one: @tajhay keeps his own add-ons private partly because of the support burden, and he is right that no gate discharges it. The same human who can read and explain the code is the one who has to support it afterwards - which is one more reason this discipline is for what you will stand behind, and genuine overkill for a throwaway.And the maintenance tail, which is the loudest complaint in the other thread: the same guards run on every upgrade, and upgrade-path testing - install the oldest build, upgrade through each version, diff the schema against a fresh install - is part of the gate. Two specifics, since they are the loudest failures in the other thread (add-ons that cannot be uninstalled, leave orphaned tables, or warn about changed file hashes): uninstall must reverse every schema change it ever made - verified by diffing a fresh-install schema against a post-uninstall one - and the release ZIP must be produced by the framework's own build tooling rather than hand-assembled, since hand-assembly is where the bundled hashes.json and the package contents drift out of sync. The discipline does not end at release; that is usually where the slop starts.
On disclosure
And yes - AI-assisted add-ons should be disclosed as such. The most-repeated concrete ask in the other thread is the reasonable one. This kind of discipline is what makes that disclosure safe to give rather than scary to read; it is about earning the trust disclosure asks for, not avoiding the question. The other marketplace questions raised there - vetting, account-age gates, review integrity - are XenForo's to decide, not a developer's methodology; this thread is only about the dev-side discipline that makes any disclosure trustworthy.And if you are on the receiving end - installing a stranger's add-on - here is what you can reasonably check or ask for, whatever built it, without reading a line of code: that it uninstalls cleanly on a test board (no orphaned tables or columns left behind), that the ACP file-health check passes (XF compares the installed files against the add-on's own bundled hashes.json - hand-assembling a package instead of using XenForo's build tooling is where stale or mismatched hashes creep in), and that any AI assistance is disclosed. That is the same discipline above, seen from the buyer's side.
I suspect most of this generalizes beyond XenForo - turn bugs into guards, gate on static analysis, review adversarially, verify at runtime, persist memory, secure by default, test your own tooling - but I only have XenForo evidence, so I will not over-claim it. XF just happens to have a large silent-failure surface (templates, phrases, style properties, the build-to-ZIP boundary) that rewards the approach.
I would genuinely like to know where this still falls short - especially any case where a gate gave false confidence and something shipped broken anyway. That is the failure mode I most want to close, and the people in these threads have clearly hit edges I have not.
One limit I already know: a guard only exists once its bug has bitten, so a newcomer adopting this starts with almost none and is still exposed to the traps they have not hit yet. The attached kit (
xf-ai-dev-full-kit.zip) is built for exactly that. Unzip it and open start-here/ first - it stacks on what 2.3.10 already gave you: paste the discipline block (start-here/AGENTS-discipline-block.md) into the AGENTS.md that install-ai-agents wrote (that, plus a static-analysis gate, is the 10-minute upgrade), run the two example guards' self-tests, then grow your own set from the catalogue. The guards need only bash + grep.The rest of the same kit is the complete set I actually run - every review agent, every guard script, the full skill set and project-instructions file, all sanitized the same vendor-neutral way. So start with
start-here/; reach for the full set when these patterns click and you want the whole reference to cherry-pick. It is large and XenForo-specific by nature - a reference to adapt, not a library to drop in.Threads this builds on: What's new in XenForo 2.3.10 and So much AI, so many add-ons... and so many bugs.
Last edited:
