So much AI, so many add-ons... and so many bugs

PhineasD

PhineasD

Well-known member
Licensed customer
Over the last few months, the number of users uploading add-ons to the resources section has spiked.

I’ve tested a few of them and found nothing but errors and glitches. In fact, it’s easy to tell they're relying on AI to develop the add-on because they usually roll out multiple updates just to fix a single bug, and when they finally do, a brand new bug pops up out of nowhere, the addon can't be uninstalled or rebuilt, and features that were previously implemented just disappear... Anyway, they're just shooting in the dark, and it’s obvious they aren't profeesional developers.

Honestly, this makes me feel pretty insecure, and I look suspiciously at users who don’t have a history on the forum showing they are actual developers. Not to mention other cases where add-ons suddenly popped up like mushrooms overnight, only to be deleted the following weeks without even leaving the option to download them as "unmaintained."

I’d love to hear your thoughts on this.
 
Last edited:
I've seen what you're talking about and we're certainly actively avoiding these newer 'developers' that are pushing out some crazy amount of addons that appear to be entirely made by AI/LLM tools. There's no way to filter them out and I'm fairly certain a few are dishonest about the percentage of AI designed/built addons vs human designed with an experienced developer looking over the work. I think it's perfectly acceptable to use various tools for testing and helping build some things, but not like this slop that has been appearing lately.

A couple of these addons add volumes of features that just appear out of thin air and probably never tested. Other veteran developers build some, test it and then build some more to ensure stability and security.

The best advice is to very vigilant about what you're downloading and installing. Bigger sites probably have access to resources to do their own code or quality audits in testing before something ever touches a live site.

Of course, this is really a problem just about everywhere, including software development.
 
I get what you're saying. Sometimes you only have to read the add-on description and instructions to be put off. That said there are some good ideas being turned into addons, which is a good thing, but testing on a live site would be a bit silly.

I've never written a single line of HTML in my life. Yet, with Claude Code and a lot of patience, I managed to write a PWA travel journal for my members. It is comprehensive, very aesthetic, so good, it looks and behaves like a native apple app. It's brilliant and the feedback has been excellent. My members love it.

I was really proud of it but before going live with it, I showed it to my friend who's a professional software engineer who works for banks and other institutions.

The refactoring we had to do to remove all of the security exploits that me and Claude had baked in was ridiculous.

He had the code three weeks before giving it back, it looked just the same, but was now safe enough to install!
 
PhineasD said:
I’d love to hear your thoughts on this.
Click to expand...
I’m happy that all my addons are by known and trusted developers so luckily I won’t need any more and risk these dodgy ai ones.

I do believe it would be good if a register of “approved” developers existed, but otherwise it has to be buyer beware. To an extent we can look at that reviews and discussion feedback.

The concerning thing would be if these did actually compete and threaten continued development by the established and trusted devs.
 
Last edited:
Mr Lucky said:
The concerning thing would be if these did actually compete and threaten continued development by the established and trusted devs.
Click to expand...
I'd assume at least some do, as some seem to simply replicate ideas, concepts and functionality that has been developed in paid add ons that established devs created and developed further over years. At least on first sight (have not tried them out).
 
Personally i haven't bothered using AI on my forum as yet.
But i'm hoping to use it for registrations.
Get it productive instead of getting scraper bots pinching stuff
 
From a site owner’s point of view, the difficult part is that you usually can’t fully review every add-on before installing it.

I don’t mind if someone uses AI as part of their workflow. That is probably going to become more common anyway. But when an add-on is used on a real forum, there still needs to be someone behind it who understands what the code actually does, especially if it touches user data, permissions, registration, emails or database changes.

So for me the issue is not simply “AI or no AI”. It is trust, testing and long-term responsibility.

I also wouldn’t want to make it harder for good new developers to get started. Everyone has to start somewhere. But clearer resource descriptions could help a lot: tested XF/PHP versions, known limitations, uninstall notes and realistic support expectations.

That would make it easier to judge the risk without being unfair to new developers.
 
PhineasD said:
I’d love to hear your thoughts on this.
Click to expand...
Same experience...

Recently tried an add-on that looked so good, so many features and the build-in GUI was really amazing. Nothing I have seen before in ACP.
But when really using the add-on, lots of features didn't do anything in the frontend.

The author tried to fix them (probably using AI), but every upgrade would result in more duplicate items and conflicting SQL tables. Also, add-on files were updated without touching the hashes resulting in file warning in ACP. This could also indicate vibecoding, because it creates a package on its own without using XF's build-in package tools.

Not good and not going to use any add-ons from new developers for now.
 
UltimateWarrior said:
I also wouldn’t want to make it harder for good new developers to get started. Everyone has to start somewhere. But clearer resource descriptions could help a lot: tested XF/PHP versions, known limitations, uninstall notes and realistic support expectations.
Click to expand...
Any of these addons that are largely written with automated tools should declare that in their release notes. Including version compatibility and common items like that is already standard, there's just no mechanism for people to declare if something is AI built or not.

There's no challenge here about developers improving their workflows with various tools that are available now. The concern is inexperienced people suddenly pushing out addons where they don't understand what is being done. Newer developers without some sort of tangible experience could face challenges... fancy release notes and graphics means nothing if the addon doesn't work or is downright dangerous.

I had particular concerns with one addon that was seemingly trying to replicate what services like Cloudflare are already doing and IMHO, loading that kind of function on top of XenForo, just doesn't seem like a good idea. There are other concerns out there, but that's just one example.

Mr. Jinx said:
Also, add-on files were updated without touching the hashes resulting in file warning in ACP. This could also indicate vibecoding, because it creates a package on its own without using XF's build-in package tools.
Click to expand...
The hashes are a giveaway... The AI tools can replicate the packaging with the correct instructions, but even with the instructions baked in, it still misses the hashes update fairly often.

We have been building our own internal tools with various agents and we've had some successes. However, most of it requires a lot of experienced developer intervention and appropriate testing.
 
While I am not a professional developer I have several custom add-ons that I created for my site. I have never released them because they target a specific need.
However, I am working with AI at this time to take an add-on that I use and turn it into a generic add-on that anyone can use.
Being an end-user as well as someone who creates there own site specific add-ons I can understand the problem. Since I have a very popular site I am takeing all precautions developing the generic version of this add-on. It has taken weeks of testing with AI help and I consider the add-on still in the Alpha stage.
The build / release in hours or days tells me that the work is not thoroughly tested.
While I use Claude to help with the add-on transition it does not write all the code. Not only do I test on my development site with no other add-ons I test on a copy of my live site for a real world test.
One thing I cantsntly do is upload the developer resource guidelines to the AI to be sure it follows XF standards. I also use other AIs to check the work to basically get a second opinion.
In summary, don't use AI to write all the code. Learn basic coding first and then use AI to help you work.
 
Just like other forms of AI slop, app slop is here to stay. Whilst I love the infusion of new addons some of which are super cool I am deeply cautious of the hazards vibe coded crap brings. Install with caution and maybe wait for a dev to get decent rep before throwing it on your live forum.
 
Everyone is nowadays are using AI. Though I still prefer to code the core of the add-on myself. AI no matter which one you use, they yet don't follow the structure and utlize core functions of XF properly. I do see some nice add-on released. ideas were definitely good but the way they were working can be improved 1000% percent if they would have utilised the default XF working flow.

AI told me I'm a genius when explained how to use core functionality of XF to make add-on in far better way. 🤣🤣

Anyway I think @Chris D and Team needs to come up with a plan to maintain quality of add-ons being released by the developers.
 
Last edited:
ai is still stupid though. How many Ps in the word Thomas?

IMG_8156.webp



IMG_8157.webp
 
Last edited:
And its just getting started... It has not taken off yet.

The flipside is that some LLM models like Mythos will be good at finding exploits / vulnerabilities. We will probably see a significant number of communities getting hacked when Models like Mythos gets going.
 
I brought this up with Chris a while back, and attempted to follow up at least once.

There is nothing inherently wrong with using AI to code add-ons, or to help develop the add-ons, the issue comes when the person who is vibe coding has no clue how to properly develop and review the work, and especially as the add-on becomes more complex and scope changes.

Having a disclosure policy won't solve issues, but it will at least allow users here to be more educated about the add-ons they are using, rather than just blindly trusting whatever slop they mistakenly install. And some of the add-ons are definitely slop 🙃.
 
A solution I've been thinking about in three parts: mandatory tests and mock data for PHP functions in add-ons. Add-ons are pushed to github where automated tests and security checks are run. Accounts must be 2 years or older to submit add-ons and it is retroactive.

The tests will increase the quality of add-ons and provide a guardrail against slop submissions when paired with automated testing. If they fail, the code isn't uploaded to the directory. It's also odd for an account made yesterday with no posts to submit a resource today.
 
vbresults said:
A solution I've been thinking about in three parts: mandatory tests and mock data for PHP functions in add-ons. Add-ons are pushed to github where automated tests and security checks are run. Accounts must be 2 years or older to submit add-ons and it is retroactive.

The tests will increase the quality of add-ons and provide a guardrail against slop submissions when paired with automated testing. If they fail, the code isn't uploaded to the directory. It's also odd for an account made yesterday with no posts to submit a resource today.
Click to expand...

Agreed, just like AI Music slop on Spotify, video slop on YouTube etc platforms will need to take some action to test more proactively.
 
Back
Top Bottom