So much AI, so many add-ons... and so many bugs

Partially a context management issue. Not perfect but you can always improve it :)
I've been watching your memory posts. Something you outlined in your last post, was meta data is meaningful. I found this in a paperless setup I did for the wife's business. The amount of files being imported, updated, new files added, I added a database to record each files meta data, so any file comparison is achieved against that instead of reading files themselves. Running through a million plus files daily looking for changes, shifts time taken from a day to a minute or two, thanks to meta-data.
 
I audited several normal addons (not vibe coded) with Claude Code and sent the reports to the developers.

This resulted in this security release:

The other audit reports may lead to similar releases for other addons, if the developers find the reports useful.
I've began auditing some of the addons that I have installed. The first one today has already identified 3 permission bypasses which using Deepseek V4 Pro has provided working examples (that I've tested in a docker install running one of my development XF boards). Audited using Opus 4.8, gpt-5.5 and Deepseek V4 Pro for a full consensus.
 
The first one today has already identified 3 permission bypasses which using Deepseek V4 Pro has provided working examples (that I've tested in a docker install running one of my development XF boards). Audited using Opus 4.8, gpt-5.5 and Deepseek V4 Pro for a full consensus.
Addon developers are going to be busy for sure!
Yup... that system of @eva2000 is sweet. The stuff its finding for me, amazed.
I think I went a bit overboard 🤣

1781872381915.webp
 
I've began auditing some of the addons that I have installed. The first one today has already identified 3 permission bypasses which using Deepseek V4 Pro has provided working examples (that I've tested in a docker install running one of my development XF boards). Audited using Opus 4.8, gpt-5.5 and Deepseek V4 Pro for a full consensus.
The way I go about it, (in a nutshell) is first let the LLM load the code context and provide relevant XF guides, then let agents read the code and find issues and describe expected behavior, real behavior, fix, and some other specifics, then let separate agents try to disprove the issues, then let it try to reproduce the issues on a dev install and local install. (depending on the type of issue) I scrutinize issues myself from the start, to avoid it going down too many unneeded rabbit holes from wrong assumptions. The weakness in my approach is that I'm only using CC. The multi-ai approach will likely find more and disprove more right from the start.
 
Last edited:
The weakness in my approach is that I'm only using CC. The multi-ai approach will likely find more and disprove more right from the start.
From my experience with my multi-AI consult skills, I generally find between 10-50% more insights than with a single model, even Claude Fable 5 alone can't match my multi-AI consult skills when I tested and compared them. Great for planning or finding as many issues.

I did some more formal benchmarks recently comparing standalone Claude Opus 4.8 and 2 versions of my multi-AI consult skills against my AI video creator skill and video-builder GUI app that integrates 16+ Video generation LLM models into Claude Code/Desktop https://ai.georgeliu.com/p/ai-video-generation-in-claude-code

Out of 18 total issues (sum of all 3 contestant A, B, C) this is how many issues each found where each contestant may have found overlapping issues.

1781887007406.webp


Had 4 AI models judge the results and 3/4 said C won

1781887141129.webp


1781887169424.webp


1781887844805.webp


Seems /consult-codex-deepseek-zai is enough for most
 
Last edited:
Yer... from my findings, AI lacks memory capacity that humans have. That seems to be its biggest flaw in coding projects as I see it. Code is code is code, if PHP add something, then everyone knows, including AI, there is nothing to be used that isn't available per existing software. The issue as I see it, memory capacity. AI really struggles in this area. If you can't remember what you've done, then you're prone to many errors - I guess hence its good at smaller projects.

There's lots of things AI lacks that a human brain is good at.
'memory' in LLMs is very expensive, because it accumulates too much non-useful details. Very smart people are still trying to solve that.
AI typically lacks taste, good judgement, context, conscientiousness, and conciseness, until you consciously intervene.

Conversely, AI is amazing at the speed and quality which it can execute on very well-defined tasks, research things, and automate novel kinds of repetitive, well understood work.

If you want good results from AI, we need a human brain to fill the gap.
If you are telling me your software is vibe coded, i'm making a guess that, other than planning, the human touch is missing on the execution.

If you did an exceptionally good job prompting, we still have some problems:
  • AI can gradually destroy a codebase or produce enormous hefty code with progressive edits over time, so from a user POV, i don't know how long this software will last.
  • Some key design information gets lost by not having the programmer more involved in the process.
  • Is a programmer reading the code? most likely not.
  • is a programmer writing or performing tests? most likely not.

For these reasons i automatically assume vibe coded software is of low quality in the majority of cases, based on past experiences, and reading LLM-generated code.

At my shop, we usually use 'human in the loop' processes where the programmer is more engaged in the process.
Upon reading the code, it has the human touch, meets code quality and style guidelines, and neither the software director nor client have any problem with the output, long term or short term.

Big corpos are finally starting to come around to human in the loop processes, after the hangover from 'tokenmaxxing'

1781897543859.webp
 
I scrutinize issues myself from the start, to avoid it going down too many unneeded rabbit holes
That is that memory issue problem from my experience.
Claude Fable 5
Can't wait for that to return. Looking forward to playing with that.
If you want good results from AI, we need a human brain to fill the gap.
I just basically said that in a DM. Feels like Déjà vu.
we usually use 'human in the loop' processes where the programmer is more engaged in the process
I honestly didn't think there was any other way to use AI than a human in the loop at all times. My experience is everything above, AI will go off in directions without keeping it directed and focused. You have to constantly keep it on the path its supposed to be on. A human has to be memory.

So yes, if there are people who just put something in to get a plugin, do nothing, then try and use that plugin, totally agree that it is likely to be horrible.

But if you have a human in the process at all times, guiding it, being its memory to keep it focused and true, then the output is damn amazing. Better than what a typical human coder could ever do, faster, security focused, it misses nothing when checks and balances are in place.

I'm going to introduce deepseek v4 pro into my consult-codex skill, to have 3 way audit analysis.
 
One thing I have been trialling to help the memory issues, is when I create a project my first conversation is to keep a record of all the important aspects of the project, so every new conversation has a single reference source. When I am finished for the day, I go back to the conversation and tell it which conversations I have had today, go through them and add them to the md for reference. See how that goes.
 
Had 4 AI models judge the results and 3/4 said C won
I tried three models, went back to two. With three models it tries to be, 2v1, which isn't necessarily correct. When its two, they discuss the issue come to a mutually agreeable correction which both believe is fundamentally correct and secure for the setup. Two models agreeing with each other over one, two can agree to be wrong and produce a result that isn't optimal, versus a discussion on the issue, not agreeance on the problem, but if the problem exists, what are the solutions and what is the optimal solution.
 
I tried three models, went back to two. With three models it tries to be, 2v1, which isn't necessarily correct. When its two, they discuss the issue come to a mutually agreeable correction which both believe is fundamentally correct and secure for the setup. Two models agreeing with each other over one, two can agree to be wrong and produce a result that isn't optimal, versus a discussion on the issue, not agreeance on the problem, but if the problem exists, what are the solutions and what is the optimal solution.
Why i bumped to a 5 model option with 2nd feedback of synthesized summary to models for follow up heh
 
I found an interesting way to use your system @eva2000. I use it on the build, so both systems are agreeing on the build itself, claude builds, codex watches, discussion happens, agreement, continuing building. Makes for interesting coding from the get go, oversight.

Code:
All green. Here is the synthesised comparison and confirmation.

---
Codex (GPT-5.5) Response

Both AIs agreed on all four changes. Codex's notable additions: it introduced issuePhase state ('issuing' | 'pushing' | 'done' | null) and a duplicateIssued state to carry the full issued object through to the OK button. It also suggested splitting the push helper into a named pushIssuedInvoice function. The duplicate-override modal's OK button in Codex's design becomes async, awaiting the push before navigating away.

Code-Searcher (Claude) Response

Code-searcher proposed busyLabel state (a string) rather than a phase enum, and pushResult: {ok, msg}. It also matched the exact line numbers precisely across all four files.

Comparison Table

┌─────────────────────────┬───────────────────────────────────┬──────────────────────────────┐
│         Aspect          │          Codex (GPT-5.5)          │    Code-Searcher (Claude)    │
├─────────────────────────┼───────────────────────────────────┼──────────────────────────────┤
│ File paths              │ Specific, correct                 │ Specific, correct            │
├─────────────────────────┼───────────────────────────────────┼──────────────────────────────┤
│ Line numbers            │ Provided                          │ Provided, accurate           │
├─────────────────────────┼───────────────────────────────────┼──────────────────────────────┤
│ State design            │ issuePhase + duplicateIssued enum │ busyLabel + pushResult       │
├─────────────────────────┼───────────────────────────────────┼──────────────────────────────┤
│ Duplicate-override push │ Await push inside OK handler      │ Same                         │
├─────────────────────────┼───────────────────────────────────┼──────────────────────────────┤
│ Backend fields          │ Identical                         │ Identical                    │
├─────────────────────────┼───────────────────────────────────┼──────────────────────────────┤
│ Badge styling           │ Single inline style               │ Two explicit <span> variants │
├─────────────────────────┼───────────────────────────────────┼──────────────────────────────┤
│ IssuedView fix          │ !alreadyPushed &&                 │ !alreadyPushed &&            │
└─────────────────────────┴───────────────────────────────────┴──────────────────────────────┘

High agreement. Divergence was only in state-variable naming, not in logic or correctness.

---
Implementation chosen

A synthesis: busyLabel (simpler than a phase enum) + pushResult + pendingPushNum (to carry the number display to the duplicate modal's OK button without storing the full issued object). The doPush helper is extracted as both AIs recommended.

---
Confirmation checklist

- myob_pushed and myob_pushed_at in list API — added to InvoiceListItem schema and _invoice_to_list_item. Backend tests: 100 passed.
- handleIssue attempts push after successful issue — doPush(numDisplay) called on both the normal path and (via the OK button) the duplicate-override path.
- Push result shown inline before navigating — pushResult banner renders in the issue card; setTimeout(1500/2500) delays onIssued(). Cancel is disabled during push.
- MYOB badges on issued invoices in list — green "MYOB ✓" / amber "MYOB ↑" conditional on inv.status === 'issued' and inv.myob_pushed.
- "Re-push to MYOB" button gone when already pushed — condition changed from !isVoid to !isVoid && !alreadyPushed; label and style simplified to just "Push to MYOB" (purple).
 
I found an interesting way to use your system @eva2000. I use it on the build, so both systems are agreeing on the build itself, claude builds, codex watches, discussion happens, agreement, continuing building. Makes for interesting coding from the get go, oversight.
Yup my multi-AI consult skills will pay off tremendously for planning stage or post-implementation verification phases https://ai.georgeliu.com/p/deepseek-v4-in-claude-code-kilo-code - combining multiple AI models unique perspective and training to surface details that a single AI model might not pick up on first pass. FYI, I recently updated the code-searcher agent so might want to get the latest one.
 
Last edited:
I want to qoute post of @Osman from his site. I believe there are very good observations.

Hello everyone,

Recently, with the increasing coding capabilities of artificial intelligence (AI) tools, we see many users trying to develop XenForo add-ons by writing prompts to ChatGPT, Claude, or similar tools. These add-ons, installed with the logic of "Does it work? Yes, it works," might look flawless from the outside, but under the hood, they can turn into a ticking time bomb for your forum.

From the perspective of a XenForo Developer, I wanted to discuss the technical details of the massive gap between an add-on generated purely by AI and one written by a developer who has mastered the architecture. Especially with the transition to XenForo 2.3.x, this gap has become much more dangerous.

1. XenForo 2.3 and the Lack of Current Architecture (The Outdated Dataset Problem)​

As you know, with XenForo 2.3, jQuery was completely abandoned in favor of a Vanilla JS (Pure JavaScript) structure. Additionally, significant standard changes were made to the CSS compilation system, icon structure (FontAwesome), and PHP 8.x compatibilities.

  • What AI Does: Since its dataset usually stops at XF 2.1 or 2.2, it will give you plenty of jQuery code, outdated xf:js extensions, or old icon classes (<i class="fa-, etc.). The add-on will either not work on XF 2.3 or will throw JavaScript errors in the browser console.
  • What a Real Developer Does: They build a performant and compatible structure utilizing XF 2.3's new XF.Element architecture, XF.ajax vanilla requests, and the new Asset management.

2. The Entity System and Database Disasters (The N+1 Query Problem)​

The heart of XenForo is the "Entity" and "Finder" system.

  • What AI Does: It generally cannot establish complex relations properly. It might have you write raw SQL queries directly using $db->query(), or it might cause the N+1 query problem by making you use em()->find() inside a loop. Every time you refresh the page, 500 unnecessary SQL queries run in the background, and your server's load spikes.
  • What a Real Developer Does: They set up database relations perfectly. Using eager loading techniques like ->with('Profile', 'Privacy') within the Finder structure, they fetch all the required data with a single, optimized SQL query.

3. Security Vulnerabilities and Input Filtering​

This is the most critical point. XenForo has very strict rules for filtering external data (Input Filtering).

  • What AI Does: It tries to filter incoming data with standard $_POST or general PHP security functions (htmlspecialchars, etc.). It might forget XSRF (CSRF) token checks. By skipping user permission checks (\XF::visitor()->hasPermission(...)), it can leave backdoors (permission vulnerabilities) that allow a member to perform actions requiring admin privileges.
  • What a Real Developer Does: They always fetch data using the $this->filter('field_name', 'str') methodology. Before executing the action in the Controller, they strictly use lifecycle validators such as assertRegistrationRequired() and assertAdminPermission(). They build the security wall according to XenForo standards.

4. MVC Lifecycle and Business Layers (Repository/Service)​

Where you perform an action in XenForo is at least as important as how you do it.

  • What AI Does: It dumps all the add-on's data, database saving processes, and email sending into the Controller (Spaghetti Code). This makes the code unreadable and unusable for future reuse.
  • What a Real Developer Does: They delegate bulk database queries to Repositories, complex business logic like thread creation to Service classes (XF\Service), and processes that will run slowly in the background (e.g., bulk mailing, image processing) to the Job/Cron system.

5. Language (Phrase) and Template (XML) Errors​

  • What AI Does: It embeds texts like "Login" or "Submit" directly into the Template files as "Hardcoded" strings. Translating the add-on to another language or modifying theme properties becomes impossible. It might mix up Admin and Public templates.
  • What a Real Developer Does: They input all texts into the phrases.xml file located in the XML _data directory and call them within the template using {{ phrase('addon_name_example') }}. The add-on is fully multilingual-supported and respects XF Style Properties.
In Summary: Artificial intelligence is a fantastic assistant when writing code. It can solve an algorithm you're stuck on, inspire you, or help you write regex. However, someone without a full grasp of the architecture installing an AI-generated XenForo add-on is like handing the keys of a Formula 1 car to someone without a driver's license. The car might go very fast, but it will crash into the wall at the first corner.

For the sake of your system's security, data integrity, and server performance, approach add-ons from uncertain sources with caution. A real developer's XF experience is not just about "making the code work"; it's about ensuring the code works "safely, without exhausting the system, and in line with the XenForo philosophy." It also means long-term support and updates.

Final Words and an Important Warning:​

Just because an add-on "works" doesn't mean it is secure and sustainable. Whether it was written by AI or created by someone inexperienced, any add-on with an uncertain origin is a risk to your forum.

Never forget these three golden rules when choosing an add-on:
  1. Support and Sustainability: In the event of a possible XenForo update (e.g., upgrading from XF 2.3 to 2.4), will you have a point of contact if this add-on throws an error? AI can provide an instant solution but cannot offer long-term technical support.
  2. Code Quality and Review: Not every released add-on should be used directly in a production environment. The add-on's references, the developer's past work, and how strictly they adhere to coding standards must definitely be considered.
  3. Continuity of Updates: PHP versions, database infrastructures, and server architectures are constantly updated. Add-ons that are not backed up and are created with an "I wrote it, and it's done" mindset will quickly cause your forum to break.
For the security of your system, data integrity, and most importantly, the peace of mind of your members, choose professional developers who guarantee technical support, closely follow XenForo updates, and respect the architecture.

Thank you for reading. We can discuss your questions below this thread.

Source: https://xentr.net/threads/xenforoda...istirme-ve-guvenlik-riskleri.5620/#post-17618
 
I'm going to say, my experience is the complete opposite to above. In fact, Claude code found all the issues mentioned above, when I went through addons I have installed and asked it to look into 2.3 coding standards. I actually asked it to compare 2.2 standards and 2.3 standards. It removed N+1 issues to a single optimised db query, it correctly architected to XF standards after much research of 2.3, it found security vulnerabilities that coders had not known about, it had full logic of the xenforo cron and system tools, in fact is asked me to run certain queries to find all available specifics for xenforo and it knew about the template system, phrase system, and could correctly navigate them.

I'm not sure where people are getting this stuff from, OR, how they're using it. Maybe that is the issue? The user doesn't know how to use the tools correctly? I know I didn't. It was a huge learning curve, it is still a learning curve. I am always learning something and discovering new and improved ways to have these tools work for me.

Maybe people just install claude code or codex and use that singularly? That would be mistake #1 IMHO. Huge mistake. Maybe they are using free AI tools of lower quality for coding. AI Coding is not created equally from my research.

If people want to use AI to code XF addons, not that I've done it, but this would be my first priority. I would install localhost and install XF addons, media, resource and ES. These are coded by XF to XF standards. I would have the coding agent then review the XF system and those addons, how they're coded, the standards being adopted. I would then send it to understand XF standards written here, find all the latest coding requirements, how to lay it out, etc. Find all the information and tools first, understand the structure, understand what is available for it to use. After it has all that information, then, and only then, would I start talking to it about coding something for XF. That is just me though.
 
@Anthony Parsons
Thank you for sharing your experience.

I think we actually agree on more points than we disagree. The purpose of this thread was never to say that AI is useless or incapable. On the contrary, I believe AI is an extremely powerful tool when used correctly, and I use it in my own workflow as well.

The key difference is that the scenario you described is quite different from the one I was referring to.

You took the time to provide Claude with XenForo's current standards, had it research the differences between XF 2.2 and XF 2.3, analyze official XenForo products, and build an understanding of the architecture before asking it to generate code. In that situation, AI becomes a powerful assistant working alongside an informed developer.

The scenario I was describing is more common among inexperienced users: they generate an add-on with a few prompts, confirm that it appears to work, and then deploy it directly on a production forum. That is where the risk begins.

Issues such as N+1 queries, permission checks, entity relationships, service layers, and lifecycle architecture are often not immediately visible. The add-on may appear to function correctly while still containing performance, security, or maintainability problems that only become apparent later.

My main point is not that AI cannot write good XenForo code. Rather, it is that AI-generated code should still be reviewed through the lens of XenForo architecture and development best practices. A working add-on is not automatically a secure, maintainable, or standards-compliant add-on.

I completely agree that modern AI models, when given sufficient context and documentation, can identify problems that even experienced developers sometimes miss. However, I still believe that the final responsibility for code quality, architecture, and long-term maintenance ultimately belongs to the developer.

Thank you for contributing your perspective to the discussion.
 
I completely agree that modern AI models, when given sufficient context and documentation, can identify problems that even experienced developers sometimes miss. However, I still believe that the final responsibility for code quality, architecture, and long-term maintenance ultimately belongs to the developer.

This is my problem. When deciding to use the software, when something looks fully vibe coded or presented, it takes a big investment to differentiate between the works of a novice or a lazy expert before even using the software. You find the structural flaws later in the process as a user. The odds are VERY low that i find a quality product in this pile, and it's very good at wasting my time, so i avoid this pile by default.

Also, I don't believe using another AI model to review your code is adequate. This is using the same idiot savant brain that makes the dangerous, yet hard to find mistakes, to audit for those kinds of mistakes. I consider that to be unreviewed and untested code. Unreviewed and untested code should not be run in production, anywhere.
 
Back
Top Bottom