1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

More Developer Collaboration

Discussion in 'XenForo Development Discussions' started by Adrian Schneider, Apr 15, 2014.

  1. Adrian Schneider

    Adrian Schneider Active Member

    tl;dr - adrian ranting again.
    XenForo - we need better tooling and standardization
    Addon/Theme Devs - we need to work together and step up, if XF won't​

    It would be nice to see more of a community effort around collaboration to try and tackle the larger add-ons. Galleries, portals, stores, etc. Every decent CMS takes hundreds of hours or thousands of hours from a community to get it right, rather than relying on a single guy (or gal) to do it.

    While I have nothing against the current paid vendors who are selling, or perhaps privately building these types of projects, it's definitely something we all want yet compete for. It's disappointing to see crowdfunding campaigns go sideways, because let's face it, most add-on developers are not doing this full time. We either have other jobs, or have to do a ton of consulting on the side in order to make a go at it. And that doesn't usually last long.

    It's a lot of weight to carry, and even if you are getting something like 5 grand, that really isn't enough to cover the costs to do something to the level that we actually want. Even if it was 20 grand, would you really want to put all your eggs in that basket? It's a lot of risk for both parties. And it'll likely go sideways, and we're back to square one.

    It would sure make things easier if we worked together.

    I've been doing XF development for the past few months full time. I've probably done a few thousand hours of vBulletin add-on development as well, and luckily most of the knowledge transfers. However, since I also work outside of these bizarre communities, like many other developers here, I've also seen collaboration done right. And what I'm seeing now is not even remotely close to that.

    Here are a few things that I'd like to see happen, and I really think this will help the add-on community flourish more.

    Development Environment & Add-on/Theme Skeletons

    We need a better development environment that let's us build things on top of a filesystem. WebDAV works, but is horribly slow. Copy/pasting is a joke, and there's really no excuse for that nearly 10 years after the plugin system for vBulletin was built.

    We need a basic skeleton that we can clone, where we can dive in and start editing things, and seeing our changes in real time. And we need that skeleton to be isolated and versionable, so it can be shared with others.

    The end result should allow me to go into a new directory

    xf clone $repo $targetLocation
    And start editing the project. And when I make changes, I should see them. And when I'm done

    xf build $addonName $outputDirectory
    Getting there isn't hard. In my spare time, I've probably gotten 60% of the way there, but my approach requires some crazy linux mounts to work. It's not ideal, but it works for me.

    Online Collaboration

    Everyone is working in a silo. We all have our different brands or accounts we release things under, rather than working together. I understand there is a little competition here as far as brands go, but I think we can also recognize contributions rather than solo efforts.

    I'd rather see us work on add-ons that are unique and make forums better rather than porting things from vBulletin, or IPB, and perpetuating the current norm. I want to see more unique stuff... it's just too bad that we're all building the same things instead.

    I'd have no problem getting a basic gallery|portal|whatever system going if I have some spare time. Let's get it functional with some basic tests behind it. After that, everyone is welcome to contribute, or someone else can take over as project lead after a while. That's what is happening everywhere else.

    Effects on XenForo

    Tons of admins I talk to are more tied to the ecosystem of the software than the software itself. They won't migrate until X is ported, or until there are more add-ons around. And for fading software, they can't migrate because they have too much custom software that isn't ported or they can't find people to continue the mess they do have. They end up having to shell out big bucks to finally port, but at that point it's out of desperation. It's too late.

    The other effect is the quality will rise. Instead of one person having to do all of the work, we can share the load. Organized bug tracking, help with testing, small patches, large feature improvements. Managing a software release is a lot of hard work, and I'm sure we've all seen dozens of good projects be abandoned.

    The surrounding ecosystem is very powerful. Unfortunately, it's not very organized compared to other product communities. (Not competing ones, but still). At the very least I wanted to bring some attention to this, as it's something I've been struggling with every day for a long time. Hopefully we see better tooling come officially, but since I doubt that, it's really up to us if we want to see it.

    If you are a dev reading this, please chime in with your thoughts. I feel it's such a huge waste that I have to build tons of tooling and processes around something that should (or could) already exist. And that's just the groundwork.
    Walter, Aayush, Alfa1 and 10 others like this.
  2. thedude

    thedude Well-Known Member

    Very good points...add-on skeletons would be great. Less time setting up the environment and more time actually writing code.
  3. CyberAP

    CyberAP Well-Known Member

    What exactly do you offer? You said you've been working a couple of months on XF development but I see no resources posted here. Collaboration among XF devs is present right now, style coders fix addon bugs and vice versa, @Waindigo for example posted tons of free resources, there are lots of educational resources. To set up your environment I'd recommend using PHPStorm or WebStorm, if you need some advanced add-on pre-configuration check out this resource: http://xenforo.com/community/resources/cmf-core.2777/
    For VCS: http://xenforo.com/community/resources/cmf-development.2778/
    Rigel Kentaurus, Andrej and Jon W like this.
  4. Adrian Schneider

    Adrian Schneider Active Member

    Of the tutorials here, the volume and quality are questionable. Some are dated, contain have bad advice, or deviate from the little direction XF has given. The problem is clearly bigger than that: there needs to be standards coming from higher and that isn't present yet. Hence this thread.

    I'm still in R&D mode myself... I've been plenty active on several admin forums as well as here offering help. I'm evaluating all of the tooling and issues I'm seeing as I build my projects. As I invest my time. I don't want to post things that I have to solely maintain when I have no time to; I'd rather contribute, but that avenue is not present at this time. I like to come in fresh and do things my own way to compare... otherwise you end up with tunnel vision thinking that what we have now is okay. It's not.

    You can try to turn this around on me as being the problem, but of the developers I've been talking to here, I'm not alone. But maybe that's because we've seen other ways of doing things and don't accept the current state.

    I went with vim/zsh/tmux, aufs mounts, a custom filesystem watching tool, and even tried the webdav stuff. I'm still finding myself in the admin panel all the time though. Surely that can't be acceptable.

    Of the link you sent, how many projects are using that layout on public git repositories? How many include test suites? How many have BDD suites? How many actually have collaborators? How many are testing on Travis or another CI tool? Please link me to them and correct me, but I'm going to take a stab and say a handful, and of those, they are barely touching my criteria.

    Now, we can get defensive, or we can try and improve things. That's why I'm here.
    Brent W likes this.
  5. borbole

    borbole Well-Known Member


    Just out of curiosity, can you please post the link to some of those tutorials here whose volume and quality are questionable and contain/have bad advice?
  6. CyberAP

    CyberAP Well-Known Member

    I'm not sure what standards are you referring to. If we talk about styles we have default style which contains everything you need from elements' naming to template and property structure. When you're ready to go deeper try this article: http://xenforo.com/community/resour...beginners-best-practices-tips-and-tricks.975/
    When it comes to add-ons just learn XenForo core and official add-ons (RM and Enhanced search), that's a basic start to understanding development for XenForo.
    Workspace is a personal preference, but I think WebStorm will be the perfect choice for IDE.
    You'll have to spend a lot of time learning all these things because it's not vB. It's not that simple and requires advanced knowledge. That's the cost of all the features we have in XenForo.

    I gave you these links assuming you'll try these resources and help to improve them and not ask «how many»-like questions. If I'm right that was the original purpose of your thread, to collaborate. So what exactly do you want to contribute to community?
  7. Adrian Schneider

    Adrian Schneider Active Member

    I don't want to pick on anybody specific here. By volume I mean the number of resources. Each resource is fairly shallow, although there were a few there that were helpful. The big missing piece is from zero to an add-on. What does that process look like.

    (and don't answer that - I know the process - it's the newcomers that I'm scared for)

    I appreciate that you are trying to help. But you're misunderstanding who needs help here.

    I have no issue ripping through XenForo code, and making large modifications. I just built my site from scratch, have built two fairly complex products, and I'm working on a large vBulletin to XenForo port with dozens of custom modifications. It's easier than modifying vBulletin because it's much more logical. I used ZF for years before this, and once I looked around, the actual programming part became very easy.

    It's the process that is broken. Where do we start? How do we test our code? How do we apply more modern processes to it? These are the problems that I'm working on as my contribution. You may not understand the need for these things, but for many people, it's very dated and hard to work this way. It's cowboy coding, and I thought we got past that a few years ago. Me asking these questions is important - it is a huge indication to platform maturity as well as stability.

    I want to know if I'm the only one who cares about making things better, and if that's the case I will continue my work privately. It means I'm investing a lot more of my time into commercial add-ons, rather than free ones that have a proper collaboration process. And I'm sure I'm not the only one.

    Anyway, perhaps I am falling on deafs ears here. I was hoping the development community was a little more mature.
    Brent W likes this.
  8. CyberAP

    CyberAP Well-Known Member

    If you're asking whether you should help the community or not the short answer is yes.
  9. Rigel Kentaurus

    Rigel Kentaurus Well-Known Member

    It would be nice to see a community project, but I think most of us are just lone wolfing :)

    I understand where you are coming from, since I have seen very similar issues.

    After trying WebDav for a little, I gave up. Just not worth it, it is way too slow, and sometimes it plainly does not work. I literally went back to having the template in TextMate and pasting it in the textarea box and hitting save. That was a lot faster.

    This is why Robbo created templates from the FileSystem

    So then people would not need to bother with WebDav or the AdminCP


    About Tooling.

    We are making some big progress. In the beginning there was no Resource Manager, but the XenForo devs embraced the add-on community and created an official distribution channel for add-ons. They also broke the vBulletin.org model and enable Paid addons (it is ridiculous to frown upon the professional developers just because they want to charge for their work).

    But then some developers want to create separate sites, but still validate XenForo Customers, and then XenForo Devs gave us the XenForo validation API - https://xenforo.com/api/, so so far so good. Selling is there, Support is there.

    It would be amazing to have a command level tool, to create a skeleton. Such does not exist, and the holes have been filled by individual developers. Just this week, @Waindigo created something to auto-add files when you register an Event Listener - http://xenforo.com/community/resources/code-event-listeners-by-waindigo.3088/

    And @Chris D created an awesome add-on for packaging .. add-ons http://xenforo.com/community/resources/add-on-builder.1091/ probably motivated by the fact that AddOn Install and Upgrade has an easier life with a properly formed addon (why this is not in Core yet??)

    Before @ragtek went berserk he started doing GitHub and putting the addons in there. GitHub is amazing because someone can add a XenForo add-on, have it there, and someone else can clone it, update it or patch it, then send me a pull request so it can be incorporated in the core. I have been considering developing Better Blogs Free that way, so other people feel more open to do contributions.


    All in all, the XF developers have been helpful, but XenForo is (afaik), a 3 man company. And all that "boring" stuff, needs money. A technical writer that does documentation will have to be paid, a person that does tutorials or community management would also need to be paid. And if the developers "just do it", then they do that instead of developing the main product. Which wouldn't be that bad considering that they are spurring the ecosystem that will drive in itself more XenForo sales.

    About ourselves collaborating, with the examples that I cited, I would say that collaboration "is there", there are development tutorials and add-ons that help the developers. But for it to jump to the next milestone .. it might take a while. The business for XenForo is not consulting (afaik), so it is not like Zend Framework in which they have in their best interests to have awesome documentation for people to adopt the ecosystem. My belief is that the community itself will NEVER produce killer documentations or tutorials or a coursera track for XenForo because that is a lot of work, often boring, and there is low reward in that. That is why companies hire people for those roles, otherwise no one else would do it.

    Tooling wise, I believe it needs to come from the devs. Otherwise, if I proposed a tool (like someone did with the XenForo API for doing add-ons) it will fall into deaf ears because everyone thinks he is smarter, has a better idea, or really has no reason to conform to someone else's workflow. If it comes from the XenForo team, it becomes the official way of doing things and everybody aligns.

    So until that happens, I would say, learn the system, develop your own tools, and enjoy :)
    HWS, Brent W, borbole and 3 others like this.
  10. Adrian Schneider

    Adrian Schneider Active Member

    Hey Rigel,

    I appreciate the detailed response. I am going to compile some comparisons between the various "development environment" options, including my own. Each seems to have an overlap in features, and of course different short-comings.

    Basically, given the existing tools that are out there, we are all solving similar problems. We just need to take it to the next level combine them. Otherwise, here's what it looks like:

    - use tool A to modify templates
    - use tool B to modify other things
    - use tool C to import
    - use tool D to export
    - use tool E to to sync stuff
    (you get the point)

    Now, this is relatively easy to get a working process if you are the only person working on it. For me, I work on two machines, which, as a side effect, lets me feel the pain of having to sync two copies.

    And to someone new coming in, trying to figure that out is a complete nightmare. Otherwise, you get someone new coming in, looking, and getting immediately frustrated. When I found the developer tools you linked to, I noted:

    Followed by noting the lack of any real activity for the past year or two. Combine the two, and the hunch is dead project.

    I understand the XF team's priorities here, which is why I think some of the developers should step up and fill that gap. It's not any more work than scratchin' your own itch -- it's less. It just requires putting the egos aside.
  11. Rigel Kentaurus

    Rigel Kentaurus Well-Known Member

    XenForo is really immature in the tooling and modern development sense. For example, I cannot even start a conversation about continuos integration because it wouldn't make sense. This is a software package that has Zero unit tests after all ;)

    It would be interesting to adopt something like ZFTool or even Composer to create that skeleton and manage dependencies. I believe that will come in time.

    I also have the scenario of multiple machines. I personally use SVN (though I will move to GIT eventually). I basically commit every change that I do, that way I keep version history, revisions and can look back if needed. And in the second computer I just do an svn update when I start. I have in the version control everything (code, templates, xenforo, the nginx.conf and the php.ini)

    I am sure that the XenForo Devs put their code in some repository under their control. It would be nice if they'd share their workflow, if only to inspire other Devs to do the same thing. These days creating a github account takes seconds.

    One step further, I even deploy to my server that way. Today I ssh to the server and do svn up .. that's it. I will probably migrate that to a git push so I don't even need to ssh to the box. I don't remember the last time I used ftp. But mine is not the most common use case.

    Several months ago I proposed that there would be a deployment process for add-ons, an install to server option, but the Devs were not interested in said feature. Something similar was implemented by Chris and Waindigo with his "update add-on from the Resource Manager", which is the closest thing to updating something to the server without all the crazy process of putting many things in different places.

    .... That said, I am grateful

    I come from UBB, the process was ... "Modify this file, then this, then this, then add this code above this, then run this queries manually, then add this file here, then translate everything interweaved with the code" ... :)
    euantor and Chris D like this.
  12. xfrocks

    xfrocks Well-Known Member

    I think all devs who developed more than 5 add-ons feel the need for tooling but the problem is each person prefers it differently. I created one myself (thread about it received 2 likes and 0 replies, github has no fork) with command to create new add-on (similar to your "xf build", mine is "xf-new-addon"), controller + template skeleton for functional AdminCP management pages, installer code, health check etc. My point is there are tools out there (people mentioned ragtek and Robbo tools too), you pick one and use it if you want to. Or you develop one and share it to the community ;)
    Markos, HWS and LPH like this.
  13. euantor

    euantor Well-Known Member

    Kind of off topic, but in response to @Rigel Kentaurus, I would love to see Composer support. Of course, it's easy enough to use as is in an add-on but it would be nice to see it officially supported in some capacity. I wouldn't feel as is right now releasing an add-on utilizing it due to the fact most users likely don't know what it is/how to use it.
  14. Adrian Schneider

    Adrian Schneider Active Member

    Just because the platform we are working has no tests, it doesn't mean our code has to. I just treat it like legacy code: isolate your stuff from the platform and build it the best you can. The only shady parts are the integration, which are the static callbacks. It's easy enough to isolate them and test everything else. There's absolutely no reason we can't have CI for our projects.

    My ideal deploy workflow is just a single call from my machine (ex: bin/deploy $target) Ideally, that happens from a CI server or from a webhook after pushing. That really brings up an interesting distinction though: add-on development vs site development. Both have very different workflows. On the add-on development side, there's also the choice of having everything in-memory, or having it sync to the XenForo database.

    I also want to be able to remotely install add-ons over SSH. I have a working prototype right now, but requires scp'ing a file, running it, and then deleting it. Not ideal, but it works. (I don't want my production environment to carry any extra tooling; I want it all done remotely)

    This is the problem though. Instead of documenting our processes, collaborating, and building tools that solve our needs, we are all doing it independently. And as I noted above, no single one is without flaws.

    I'm not putting down anybody's work - every single tool I've looked at has helped me. But I still think we can do better.

    Composer could work. You can set up private package registries, and set a a package type for XenForo, and it would run the install code / import process for you. This is a great developer workflow, but obviously wouldn't work for production.

    Thanks for the discussion guys. I feel like we're getting further.

    My next steps are going to be documenting the pros/cons of the noted tools, as well as writing up what my current process is. I'd be interested to hear how everyone else is doing things as well.
  15. euantor

    euantor Well-Known Member

    My problem exactly. I haven't done a lot of public development work with XenForo, but I do plan to do a lot more. I'd love to have an easier way to use Composer so that dependencies are installed automatically when a user installs an add-on from the ACP. I'm not sure how feasible that is as I haven't looked into whether Composer can be called from a standard script (I'd assume it can seeing as it's just PHP).
  16. TBDragon

    TBDragon Active Member

    well this looks very nice

    and i like the " composer "
    i like it after i work with laravel 4 from 2 weeks ago

    i hope i can help here , but iam so beginner so most of the talk i cant discuss about it

    i will cont. reading ^^
  17. s.molinari

    s.molinari Well-Known Member

    This is a really great conversation. I am sucking it all up.;)

    Edit- actually, I need to ask the question. Doesn't the lack of the "dev things" mentioned here come from the fact that an application (this kind of issue isn't inherent to XF alone) is trying to be a platform as a second thought? And because the platform side of the application's development is a second thought, it also gets that kind of attention and thus the very warranted rant here will always be an issue?

    Last edited: Apr 17, 2014
    HWS and Adrian Schneider like this.
  18. Adrian Schneider

    Adrian Schneider Active Member

    Yes, that's definitely one of the realities, at least on an official capacity.

    But for the rest of us who use that platform, there is still room to document our knowledge and work together to refine the processes. It also adds a bit of push back to XF to know where we need help. I don't need to sell forum ecosystem to you as I'm sure you understand. Similarly, either you have every single feature that everybody wants, or you allow people to fill the gap for you.

    Clearly some official support is there, and it's being advertised as having one of the best dev. environments around. The class proxy and general plugin system is a huge step forward compared to vBulletin, but it still has some of the same challenges. These challenges don't necessarily exist in major open source blogs or other platforms. They aren't inherently forum-specific, just implementation specific.

    At the very least, if v2 has more of this stuff baked in, we've done our jobs. In the mean time, we can work on getting faster at building and releasing software. In that respect, it's no different than any other development work. Isolate dependencies (XenForo) and build quality software as you normally would.
    s.molinari likes this.
  19. Rigel Kentaurus

    Rigel Kentaurus Well-Known Member

    I am not really a mind reader, and the XenForo owners are not the talkative type about their vision and their plans for the future, so I can only infer from what I see ....

    The framework part, if it was a second thought, was executed in a pretty neat way. It wasn't built from the very beginning (were that the case, template hooks would have been there from the first release, and template substitutions would have been there too). But that said, they managed to come with a set of abstraction patterns that enables the software to be extended pretty good. Between being able to extend any class, and modify any template, a lot of things can be achieved. Add-On classes are naturally separated from the main code, so that is a win. And it is completely non-invasive.

    Yet, I don't feel like it was in their mind "let's create an awesome platform", but rather "let's create a killer forum software", and of course the competition enabled add-ons, so it was just natural to have them in XF. Creating a platform is a way different business. Today, we don't have "modules". By convention, developers create a folder inside library that naturally namespaces the add-on, but that is hardly a module, and JS and CSS are still separated. There is no quick way to add or remove an add-on without hunting down through ftp all the uploaded files.

    On top of that, the focus does not seem to be helping developers create add-ons (should it be? that sounds like a serious business decision). We have gotten many useful features along the way, though. But in the form of tutorials, documentation, guidance and tools, each developer is on his/her own.


    Besides the tooling, there is the topic of the MarketPlace.

    Even after a lot of insistence, they have also ignored and avoided the topic of creating a market. vBulletin is also horrible in this but IPB has their MarketPlace. A good store is also part of an add-on ecosystem, and it has several benefits ...

    For users, it enables them to one-click buy an add-on on a semi-trusted place, without having to go into an external site, and without having multiple accounts and processes (some people want a paypal transfer, some you need to PM them, some use eJunkie or pulley or others)

    For developers, it simplifies the selling channel. Using myself as an example, I had to extend the RM to accept PayPal payments, and pretty much build my own cart. I know @Daniel Hood did the same for XenMods, and who knows who else has needed to do that. This is distracting. Instead of developing my add-ons, I am now developing the means to sell them. All of that ideally should be provided by the MarketPlace. I would definitely use @digitalpoint 's if it were more popular

    And for themselves, it is another source of revenue if they take a cut of the sales (which they need to, since they are hosting the files, paying for the bandwidth, and indirectly doing promotion.

    Every time this topic comes up someone deviates it saying that XF cannot endorse 3rd party add-ons, about who's responsible for them, support, and many other arguably solvable problems :)


    I don't know if we will ever get better tooling for XF. I am happy with what we have right now, it enables me to modify the software in any way I want already. But I am also past the step of figuring out things and needing tools because I ended up doing my own, as I believe most serial developers did.
    Claudio, Daniel Hood and HWS like this.
  20. Adrian Schneider

    Adrian Schneider Active Member

    I'm actually quite happy with the policy here for 3rd party add-ons. vBulletin out-right banned them which I never understood. A marketplace would be a welcome improvement. I just finished integrating Stripe into a custom shopping cart / license system I built. While it wasn't fun, I prefer 100% (or 97%) rather than 70. It's nice to get into the door, but after a while you lose a lot of money.

    @Rigel Kentaurus - one point I want to make here is a developer's tools should never be considered final. That is dangerous. We should all be striving to make things better and better, and version those tools as we go. My dotfiles and custom tools continually improve, as does my platform-specific tools. In more mature ecosystems, many of the tools are already built. Not because people don't need help, but because faster is better, and people collaborate to share the love.

Share This Page