Resource quality review, standards, future development

[Chris D]

This is probably the best thread to continue the discussion stemming from the quoted post:

Probably for another thread, but I think this is just illustrative of broader problems within the xenforo development community.

Realistically, the problems you're describing are not exclusive to XenForo, nor are they inordinate within this community when you look at the bigger picture. Of course there are going to be some projects that fail, that's the unfortunate nature of this kind of development. There are also projects and developers that are and will remain to be extremely successful.

I'm not sure whether the folks at XF don't care about fostering better add-ons, or if they are so focused on getting 2.0 out the door that they aren't noticing what is happening out here.

But from where I sit, many of the add-ons (even the 'commercial' ones that users are paying real money for) are incomplete, buggy, or unsupported. Or all of the above.

You make this sound a little bit like our fault. That's maybe not intended, but at the same time, if it happens, what can we realistically do about it? It's unfortunate when we see projects failing to meet expectations, but there's not a lot we can do about it. As operators of this community, we certainly won't stand by and watch people exploit our customers, and we would certainly step in if the situation warranted, but it rarely is.

The fact that some of the largest and best add-on developers have packed it in & sold their modules off to Audentio should be cause for concern for everyone that relies on this platform. Particularly because the core of XF is so tightly scoped, that a significant number of add-ons are necessary just to get the feature set users expect in 2015.

This is somewhat of a misnomer. XF has had a reputation for too long that it is lacking in features. There was potentially once a contrast against this, a project that is now 5 years old, compared to other software which is much older. But in recent years, that gap has been bridged and generally the lines between "necessary" and "niche" are becoming a little bit more blurred. With the best will in the world, it will always be the case that there will be some gap in functionality, but I have to say for the most part, as alluded to earlier, the community fills this gap really well and they produce reliable, well supported products.

I am anxiously awaiting more information about 2.0, but it would be reassuring if the core devs could verbalize a plan to improve the quality and quantity of add-ons.

We have verbalized a plan. XF2 is almost exclusively about improving the quality and quantity of development - that's not exclusive to add-ons. XF2 will allow even us to develop more features, more quickly in the future.

 

[Slavik]

Also in reply to that post....

I am anxiously awaiting more information about 2.0, but it would be reassuring if the core devs could verbalize a plan to improve the quality and quantity of add-ons.

Exactly how would you think we could go about doing that? We can't force, or have the resources to check the quality or quantity of addons released by the community.

What I can suggest, and it is what I do is support the addon authors who release quality work, and provide prompt and quality fixes without requiring payment to fix bugs.

I often send people to @Snog and @Brogan, (I used to say @Chris D also but hes generally unavailable nowdays) for that reason, they have a limited number of addons, yet the work and support they provide is always top notch.

Put simply, vote with your money, support the addon authors who do the quality work and encourage them to carry on doing so. And yes, that may mean paying more than $10 for an addon, but it would be better for all to have a larger number of quality authors with a few addons each at a higher price point, than 1 or 2 addon authors with hundreds of addons at next-to-nothing which gives them an unsustainable workload.

 

[Alpha1]

Exactly how would you think we could go about doing that?

Like this:

There is an automatic system in place but its sole purpose is to perform an overall validation of each submitted package. This includes checks for file existence, unnecessary files (Mac OS X's ._* or .thumbsdb on Windows) as well as PHP and XML syntax checks. To sum it up, it only checks if the package looks like it could be installed successfully and does not clutter the software with garbage files.

Once a file has passed this check it is queued up for review which is entirely a manual process performed by our staff members.

How long does that take, let's say for a medium sized mod? What is checked, every line of code, or just common stuff?
How much does this cost the person releasing the mod?

I find it difficult to spit out any number for this, because the amount of code does not really affect the time required to perform a full check, rather than its complexity. While this might sound odd for most people, it actually makes sense: If your code contains a lot of generic business logic such as calculations, we can quickly skim that part.

On the other hand we carefully examine all those parts where data is written or updated in the database (thanks to real prepared statements SQL injections are a non-issue) to see if it does something unexpected, e.g. updating a database table while missing conditions and suddenly overwriting everything. Another important section is the display of user-provided data in templates, XSS is still something people are not aware of in 2015. Thankfully all variables that are used for output in templates are already sanitized by our template engine unless the developer explicitly opt-outs single variables ("Hello {$user}" will be automatically sanitized, "Hello {@$user}" will prevent this).

So while we do not check every single line, our experience allows us to quickly identify critical components and focus on these. Our goal is to ensure that plugins adapt the general UI/UX and contain no critical bugs that could harm the software. And last but not least, any sort of callhome function are a big no-go and result in an immediate rejection.

These checks are non-optional for all packages uploaded to our Plugin-Store, this includes both free and paid mods, and because of this free of charge. Being able to provide a trusted platform with consistent and secure plugins is far more worth than the extra time we have to invest for this.

 

[n00bsaibot]

I don't see the incentive for them to do that unless they get a piece of add-on sales, like the Apple store. Otherwise how would they off-set the added cost of labor? It wouldn't be fair to bake it into the price of base XF for people who don't use a ton of mods.

To point out the flip side, I like the lean product they sell. In fact, the add-ons I use are mostly from @AndyB to further trim features.

 

[Chris D]

To put it in perspective, Alfa1's quotes come from a developer of another forum software, and I may be wrong but I believe they do operate an app store like system and take maybe around 30% of each sale.

I want to sort of move away from this part of the discussion, though. It's not really something that XF has on its radar at this time, and I think you'll agree that going back to adwolf1's post, it doesn't actually solve the problem. We're talking there about add-ons that have bugs or developers who no longer update or support their add-ons. An approval process into the Resource Manager is not going to solve that problem. If we were to do the same checks as mentioned in Alfa1's post, then I would say with some degree of confidence that most add-ons would pass because most developers follow XF standards and therefore the code shouldn't be susceptible to SQL injection or XSS almost automatically.

My point was before that there's not a great deal we can do about developers who fail to meet expectations. This can happen for a variety of reasons; the failure of a business, issues in their personal lives, a change of circumstance such as school or work commitments. These are all things we can't impact.

All we can do is design a platform that customers want to use and developers want to develop for. We already have that, but XF2 is about making that even better.

 

[Alpha1]

One thing that would really help is to make visible which developers are completing custom development to satisfaction.
There's a lot of custom development going on and unfortunately past projects stay completely invisible. Even if they have gone awfully wrong. It would be so helpful to see who are top ranking in customer satisfaction.

 

[Martok]

One thing that would really help is to make visible which developers are completing custom development to satisfaction.
There's a lot of custom development going on and unfortunately past projects stay completely invisible. Even if they have gone awfully wrong. It would be so helpful to see who are top ranking in customer satisfaction.

Isn't that what this forum is for?

https://xenforo.com/community/forums/third-party-services-offers.42/

Members who use developers can post in their threads to say what they think (and some certainly do). This is where I would be looking before hiring a developer for some custom work. I think it's better than some rating system - after all we see enough complaints from members about the character limit on ratings in Resources.

 

[HWS]

What I can suggest, and it is what I do is support the addon authors who release quality work, and provide prompt and quality fixes without requiring payment to fix bugs.

I often send people to @Snog and @Brogan, (I used to say @Chris D also but hes generally unavailable nowdays) for that reason, they have a limited number of addons, yet the work and support they provide is always top notch.

Put simply, vote with your money, support the addon authors who do the quality work and encourage them to carry on doing so. And yes, that may mean paying more than $10 for an addon, but it would be better for all to have a larger number of quality authors with a few addons each at a higher price point, than 1 or 2 addon authors with hundreds of addons at next-to-nothing which gives them an unsustainable workload.

My point was before that there's not a great deal we can do about developers who fail to meet expectations. This can happen for a variety of reasons; the failure of a business, issues in their personal lives, a change of circumstance such as school or work commitments. These are all things we can't impact.

Being in this community for 4 years I've seen a lot of add-ons. I do PHP coding for 15+ years myself and can only say that many (if not most!) add-ons here are pieces of software with a quality I would never install at any of my servers. There are some even prolific add-on authors who simply cannot code and also do not seem to be able to learn it because their newest add-ons do not get better. Some just copy code from other sources and put them together in a very complicated way without really knowing what they do. Most do not care at all about performance or security. It is very scary what can be seen and what will be installed at some live forum sites (and the forum owners do not even know it)!

It is only the quality of XenForo itself which helps and prevents most bad add-ons from being quickly detected as problematic and mitigate the largest security holes (if the developer "accidentally" uses the default XenForo classes, I've seen add-ons taking e.g. the unfiltered input for it's tasks). Most such badly coded add-ons at least run at very small forums and most forums are very small. So even such developers can get a crowd of fans. And if they fix the 35th bug in their bad software rather quickly they also provide "good support". Even if they introduce the 36th bug with the fix.

There are only a hand full of coders who know what they do. And their add-ons tend to be more expensive than others.

And now there are also "coders" who simply bought (or got) the code from other coders, resulting in a portfolio of add-on with very different quality.

Most forum owners here do not know anything about coding and how to evaluate code quality. They simply read the description of add-ons and decide to download and install them. There are no hints about the quality of the code.

Since the 250 character limit the "review" feature is no longer able to fulfill it's task as a guideline for code quality or even customer satisfaction. People don't want to write long reviews. They would like to just rate, but cannot. Also if one has a problem with anything (even just the price or the registration requirement of the developers web site) they will write long negative reviews even for high quality software.

I think there should be a guide for XenForo owners to be able to see the quality rating of an add-on. You should at least allow your members to rate add-ons (and only optionally review it). You provide a "Most Add-Ons" tab in your ressources area but not a "High Quality" tab. Developers since then aim to be the one with the most add-ons. But is that really a quality criteria?

Going to XenForo 2, it would really make sense if you not only change the internal code of XenForo but also how you deal with add-ons and their presentation. It will make a lot of a difference on the long term if you care about the quality of add-ons people can use with your software and support quality add-on developers. See the success Apple had with it.

 

[Martok]

Most forum owners here do not know anything about coding and how to evaluate code quality.

I think there should be a guide for XenForo owners to be able to see the quality rating of an add-on.

As you said, most forum owners don't code so can't evaluate code quality, so how would they rate the 'quality' of an add-on? To many, that would probably just mean that it does what they want it to do. That isn't quality. As you have stated, there are add-on coders here whose coding skills could be called into question (there are some authors I would definitely not use) but to those who do use them, as the add-ons seem to work for them, they would probably say that these are 'quality' when any decent coder knows they are not.

 

[HWS]

as the add-ons seem to work for them, they would probably say that these are 'quality' when any decent coder knows they are not.

You are right. If a small forum gets hacked, no one will care. There won't be a headline "XenForo forum got hacked". There will be just an unsatisfied XenForo customer.

 

[Chris D]

I'm quite surprised you've been holding on to this information for 4 years. Do you really think we would knowingly allow add-ons to be published here if they contain confirmed vulnerabilities? Furthermore, do you raise these quality concerns with the author yourself? Or even publicly in the resource thread?

Is the long term answer to review and approve add-ons? Maybe. Is the short term answer to rely on knowledgeable users within the community to help weed out the bad ones, mentor the inexperienced ones or make other people aware? That would certainly help.

We're going off topic here, now. We should keep this thread about what's next for XF. Feel free to start a new thread if you wish to discuss it further.

 

[HWS]

We're going off topic here, now. We should keep this thread about what's next for XF. Feel free to start a new thread if you wish to discuss it further.

No, I don't want to discuss this "further". May I remember it was you who introduced this topic into this thread:

This is probably the best thread to continue the discussion stemming from the quoted post:

But thank you for your last answer. It helps!

Let me just shortly answer those questions, you directed to me.

I'm quite surprised you've been holding on to this information for 4 years.

I don't think it helps getting sarcastic...

Do you really think we would knowingly allow add-ons to be published here if they contain confirmed vulnerabilities?

It is a fact that there are some add-ons with more or less serious vulnerabilities.

Furthermore, do you raise these quality concerns with the author yourself?

Yes, sometimes. But always just once, if the author does not like such concerns.

Or even publicly in the resource thread?

Yes, but also just once if the author does not like his add-on constructively criticized.

I try to help (this is why I post here). And you know that.

 

[Lawrence]

Well so much for that excitement. When I seen that @Chris D made a post in this thread, I thought, just for a brief moment, that maybe a new hint, or insight, or whatever was going to be contained in his post.

Reading what he was replying to, it just turned out that most of those who create add-ons for this community, free or otherwise, had been judged and then sentenced with a massive slap in the face by the community. Nice.

 

[Chris D]

The original idea behind moving some posts to the What's Next for XenForo thread was that the original question had a component about XF2, but it seems clear that we're potentially trying to discuss something here that is not necessarily exclusive to XF2 and is potentially valid feedback about the current situation. I think replies in that thread were also potentially inviting the expectation that we had something new to talk about, regarding XF2, but that's not really the case at the moment. With that in mind, the posts have now been moved to somewhere more appropriate.

I don't think it helps getting sarcastic...

Please don't get me wrong. This wasn't sarcastic. I am genuinely surprised. As staff, we don't have any specific recollection of reports of particularly poor quality or unsafe add-ons.

It is a fact that there are some add-ons with more or less serious vulnerabilities.

Yes, sometimes. But always just once, if the author does not like such concerns.

Yes, but also just once if the author does not like his add-on constructively criticized.

In the first instance, contacting the author yourself for a resolution is the right thing to do. If the author fails or refuses to resolve that issue, it should be reported to us. For the most serious confirmed vulnerabilities we will likely remove those add-ons from the RM until those issues are fixed. Ideally, when reported, the report should give us some indication of what the vulnerability is and how it can be exploited.

It is a fact that there are some add-ons with more or less serious vulnerabilities.

I try to help (this is why I post here). And you know that.

And this is why I'm surprised. We can only act upon what we know about. We really aren't just going to ignore reports of our customers being wide open to vulnerabilities. So if you do have time, and the author hasn't rectified the issue themselves, please let us know via the Report system.

Reading what he was replying to, it just turned out that most of those who create add-ons for this community, free or otherwise, had been judged and then sentenced with a massive slap in the face by the community. Nice.

It's certainly a fact that, as with anything, a few developers have become unresponsive through a variety of different reasons. It is unfortunate though for those that fall victim to projects which fail. Thankfully they are certainly in the minority, and there are many more who are committed to creating and supporting quality work, and even more who appreciate that quality work and commitment.

 

[adwolf1]

The original idea behind moving some posts to the What's Next for XenForo thread was that the original question had a component about XF2, but it seems clear that we're potentially trying to discuss something here that is not necessarily exclusive to XF2 and is potentially valid feedback about the current situation. I think replies in that thread were also potentially inviting the expectation that we had something new to talk about, regarding XF2, but that's not really the case at the moment. With that in mind, the posts have now been moved to somewhere more appropriate.


Please don't get me wrong. This wasn't sarcastic. I am genuinely surprised. As staff, we don't have any specific recollection of reports of particularly poor quality or unsafe add-ons.

In the first instance, contacting the author yourself for a resolution is the right thing to do. If the author fails or refuses to resolve that issue, it should be reported to us. For the most serious confirmed vulnerabilities we will likely remove those add-ons from the RM until those issues are fixed. Ideally, when reported, the report should give us some indication of what the vulnerability is and how it can be exploited.

And this is why I'm surprised. We can only act upon what we know about. We really aren't just going to ignore reports of our customers being wide open to vulnerabilities. So if you do have time, and the author hasn't rectified the issue themselves, please let us know via the Report system.

It's certainly a fact that, as with anything, a few developers have become unresponsive through a variety of different reasons. It is unfortunate though for those that fall victim to projects which fail. Thankfully they are certainly in the minority, and there are many more who are committed to creating and supporting quality work, and even more who appreciate that quality work and commitment.

I didn't think my little post about the classifieds would draw this sort of a discussion! But I'm glad to see it has got people talking at least.

The add-on community as you run it now is much more akin to Google's Play store than the iOS store -- as far as I can tell there isn't much in the way of QA or monitoring, and action is only taken when problems get out of hand.

Just like in Google's store, this strategy works very well right up until it doesn't. The quality of the Android ecosystem is worse on many quantifiable metrics than the iOS store (security, privacy, code quality, etc...), and users of their phones suffer for it.

While i've worked with many good devs in the XF community (very sad to see Jon go), the reality is there are still poorly coded, poorly supported, or outright nefarious add-ons that have been served up from this community. And telling good from bad is challenging, even for sophisticated users.

A prime example would be *******'s add-ons. They were here for years, and the features those add-ons provided were important to a lot of sites. I -- as a user -- had no reason to suspect anything hinky or below-board was going on with them. So imagine my horror when their add-ons were all suddenly withdrawn from the site, and their name was treated like Voldermort. Then an avalanche of info came out about what the add-ons were doing, their source, the callbacks, etc..... and site owners like me realized just how up-a-creek we were, as some of their addons had no direct replacement (and still don't.)

I think the nature of organizations and people that are installing XF is also changing. It isn't 2010 any more -- a hobbyist who doesn't want to spend $ is much more likely to just start a facebook group for free and call it done. Or use any number of other cheap alternatives.

Site owners who choose XF are looking for the best, highest-quality experience we can give to our users. And increasingly, we are willing to pay for it. (again, if we weren't we'd choose any number of free or cheap php garbage fires, or use facebook straight away.) So in light of a customer base that is more professional, perhaps there are some steps you can take to ensure that the add-ons we're exposed to meet some standard. I think an open discussion of this would be healthy...

 

[Mr Lucky]

And yes, that may mean paying more than $10 for an addon

I would also encourage people to actually donate when an author offers quality stuff and fast response to feedback. I'm think specifically of @AndyB who just keeps coming up with stuff that just makes life easier and also works well. Some is niche, but a lot is stuff that should be in the core (and has been subsumed into the core software over a release or two)

 

[Joeychgo]

One thing I would suggest -- although its offtopic -- is having the addon controls in the acp to be separated from the standard XF controls. Doesn't have to be a separate page, just a separate section would be nice.

 

[Daniel Hood]

The problem is, what's the standard? How strict will they be enforcing standards? Also, I don't think people realize how many developers, and in how many add-ons, don't follow all standards. Take for example (no offense, but he was already mentioned in this thread) Andy. As many add-ons as he has, as many people as there are that love them, he ignores certain standards consistently. In every one of his add-ons that I've ever looked at, he queries from the controller rather than using models. Sure, nothing critical there, but in other cases there are critical things and even ignoring those -- where's the lines for what standards are going to be enforced?

Making XenForo police the entire market place will take a lot of resources. It'll also cause baseless accusations of favoritism, depending on how strict they are. Are we going to allow add-ons that don't use models? Are we going to allow add-ons that don't prefix phrases? Are we going to allow add-ons that don't defer processes that should be deferred? Are we going to allow add-ons that don't prefix template modifications and templates? The questions and standards are endless, some are just knit picking but if it's policed, they should all be enforced (arguable).

Apple does this. They are an extremely large market though and can get away with people complaining about being blacklisted or whatever other negative reviews they catch because it's a small percentage. XenForo is a much smaller market and may not be able to handle negative criticism that is nearly inevitable.

 

[Alpha1]

Or even publicly in the resource thread?

While I do not disagree with what you are saying, I'd like to pose the question: is a 30 page resource page really that public? IMHO it isn't. Serious bug reports get buried in a thread quickly so that even the author may miss it. A thread is not suitable to discuss multiple bug reports.

Consider how useful it would be if all resources would list the number of open bugs and the severity of those bugs.

No one would need to report severe issues to you because it would truly be public and easy to see for anyone.
If this were visible, then you could easily filter out resources that have long-standing severe issues or security issues.

 

[Alpha1]

Isn't that what this forum is for?

Sort of. But its shoehorning a forum for a feedback & reviews system. You can not see past developer projects, the ratings and evaluations by clients for those projects. There are no set standards to rate developers, so reviews are often vague and disregard important factors.
You can not see the ratings and evaluations by the developer for those projects.

In general its not normal to post complaints in a discussion forum, because it will lead to... discussion. and possibly drama. Its the wrong medium / format to get the desired result of objective reviews.