XF 1.4 Template Rebuilding Process issues / Questions

[XxUnkn0wnxX]

i don't know much how the process works but..

i have had enough its taking about 40 min just to install or upgrade an add one same with Languages or themes.

and its starting to hit the point where creating a new node/forum takes about 1-5 min to create as it needs to rebuild permissions and a few templates.

how does the process work? & why do i need to leave a browser window open all the time while it is rebuilding? does use my Browser & Computer resources to rebuild them?

why not simply have it running in background? or when adding new styles, Languages, add ones why not have a Que for this and process the lot in one go?

i very few themes but many add ones well over 100, my forums is very active, many members and a lot is going on at the time i am usually rebuilding.

i noticed when i close the board the time frame on rebuilding is slightly reduced.. but still doesn't solve the issue..

i am constantly adding more features to the site thus increasing the time to rebuild.

is there any way i can allocate more server resources to speed up the process? i have a lot of ram and good CPU which is barely being used.

what kind of tuning can i do to increase the efficiency of the rebuilding process? such as Mysql tuning - i assume the process fully relies on Mysql queries or something (any thing i can do here to super charge the rebuild process?)
ATM I am on Percona Server DB Software

Removing addones or shutting down the board is not a solution i can take.
since i & my community use 99% of the addones i have installed.

 

[Paul B]

Rebuilding shouldn't take anywhere near that long - it takes around 30 seconds on my localhost and live server.

You may need to look into some server tuning.

 

[Mike]

how does the process work? & why do i need to leave a browser window open all the time while it is rebuilding? does use my Browser & Computer resources to rebuild them?

It uses your browser to ensure that it is finished and to show you the process (as you shouldn't move on until it's finished). XenForo does not exist in an environment where we can guarantee access to task management systems.

 

[XxUnkn0wnxX]

Rebuilding should't take anywhere near that long - it takes around 30 seconds on my localhost and live server.

You may need to look into some server tuning.

even with aprox 100 add ones and 4-5 themes?

its always been like this for me..
on my testing board i have aprox 30-40 add ones and it takes about 3-5 min but it only has like 2-3 nodes

for my main on the over hand have around 800+ nodes including forums and categories..

and i am puzzled on why it takes so long to rebuild.. if there where a way to profile or run a scanner while rebuilding to detect what is causing it to rebuild so long would help..

as for further tuning i'm stuck as i have no idea what to tune to increase rebuild performance.
if knew how the process actually worked like how many queries are used when rebuilding or how much data is processed within x amount of time to figure out if something is limiting the rebuild process. IE if i need to increase Max execution time or allow for a bigger buffer for mysql service. or other Config that could help..

but i am unsure how to debug this or figure out why its so slow.. i know @Slavik offers his services for server configuration would he be able to help in this matter? or some one who specialises in this?

 

[Paul B]

even with aprox 100 add ones and 4-5 themes?

I don't have quite that many - only around 20 add-ons and 3 styles.
The time does increase for each add-on/style combination (as the total number of templates are add-ons x styles) but even so, the time does sound excessive.

 

[Mike]

Number of languages is also as significant as the number of styles. It's directly languages x styles x templates, with the number of add-ons having a large influence on the total number of templates.

 

[XxUnkn0wnxX]

well like i said above i'm stuck and unsure how to debug/resolve this... without removing add ones as i do need them..

around 5 min should be Max no longer..

Number of languages is also as significant as the number of styles. It's directly languages x styles x templates, with the number of add-ons having a large influence on the total number of templates.

well i would assume i have more then most other sites + multiplying that by the amount of nodes i have...

but what does the rebuild process solely run on? which service/web application mainly do the processing? from there that service/web application can be tuned to use more server resources to allow for faster more efficient processing of the templates. as barely any resources is being used atm. probably around 20-%30 CPU and 8-11 gigs of ram out of 64G.

as for overall Stats Estimation:
6-7 languges
5-7 themes + 3-8 more that are inactive as they act as the framework
800+ nodes
about 20-30k Messages, Threads
15-25k members
100-120ish addones

my overall goal is to increase the default values of the service/web app that processes the templates when they are rebuilding to allow for better efficiency - may Enlist a specialist for aid in this...

 

[Paul B]

6-7 languges
5-7 themes + 3-8 more that are inactive as they act as the framework

That explains it - 15 styles and 7 languages (plus 120 add-ons) equates to a massive amount in total.

Just taking it at a very basic level, 1 language x 1 style = 1, 7 languages x 15 styles = 105.
So you are doing over 100 times more than a default installation when rebuilding.

 

[Mike]

6-7 languges
5-7 themes + 3-8 more that are inactive as they act as the framework

Just to give a specific number, that's approximately 105 times more rebuilding work than someone with 1 style and 1 language (there's a bit more to it). With the number of add-ons likely adding a huge number of templates, that triggers even more work for each style-language combination.

 

[XxUnkn0wnxX]

Just to give a specific number, that's approximately 105 times more rebuilding work than someone with 1 style and 1 language (there's a bit more to it). With the number of add-ons likely adding a huge number of templates, that triggers even more work for each style-language combination.

well is there a way to make the work go quicker? like tell Service that when processing such large quantities of data tell it to use more Server resources to speed things along?

like a override of the default settings set when rebuilding templates increase the max limits...

but i a still unsure what to configure or should be looking at.

 

[Mike]

PHP will be processing as fast as it can. It's just going to use a single processor though, which is why you're seeing 20-30% usage. Most of the work is going to be CPU-bound.

Doing a multi-threaded rebuild is something that would need to be custom developed; it's not something that'd be part of XF.

I could only recommend that you remove styles and languages that you aren't using.

 

[XxUnkn0wnxX]

PHP will be processing as fast as it can. It's just going to use a single processor though, which is why you're seeing 20-30% usage. Most of the work is going to be CPU-bound.

Doing a multi-threaded rebuild is something that would need to be custom developed; it's not something that'd be part of XF.

I could only recommend that you remove styles and languages that you aren't using.

so there is nothing that i cannot configure in PHP or Mysql to improve performance? - i am using MPM event for multi thread & FastCGI but still no improvement for template rebuilding..

so in the end only way to make it go faster if xenforo itself was reconfigured/tuned to speed this up?

and what has the most significant impact? as in what causes more templates? Add ones, Languages Or styles?

but even if i further reduce styles & Languages it will still be slower due to the increasing amount of add ones i add every few months (i am trying to remove some of them by replacing them with better ones that has multiple features built into the only one add one.)

 

[Mike]

Using the latest PHP with opcode caching and ensuring that MySQL is configured to use InnoDB well (http://www.percona.com/blog/2013/09/20/innodb-performance-optimization-basics-updated/) are really all that you can do server-wise.

Otherwise, you just need to reduce the amount of work that needs to be done. You are potentially requiring each template to be compiled over 100 times to deal with all of the style and language permutations that can be used. Reducing the number of styles and languages will reduce this dramatically. Reducing the number of add-ons will also reduce the number of templates, which would also help. But otherwise, you're simply using a setup that requires more work. You can only optimize it so much before considering reducing the amount of work involved.

 

[XxUnkn0wnxX]

well this might be a suggestion but..

how come xenforo has no support for situations like this. EG the multi threaded rebuilding.

because depending on the type forum/business you are running the software should be able to suite & handle your needs.

and switching to another forum software isn't very the best option of me +it would take to much time.

when i first bought this software i expected for the software to suite all my needs as a growing community.
my goal is to grow into a popular well built site someday. and i wish to have support for the main or most common languages so our users can navigate our forums with each no matter which country they are from.

also we want to provide a vast variety of themes and features to help better manage the forums & make the forums more active by having good statistical add ones displayed within our forum & features that make the site more alive and attractive.

without performance decreasing.

i know each Web Software organisations try to make their software suite most of there consumer needs but what about consumers who need more out of the software that wasn't innately discussed about or thought of?

all i am asking is that will XF ever grow to a point where it suites every ones needs? as for me i would say my needs are keep piling on features without performance decrease and the board auto adjusting to suite the users needs without a max cap on the amount of data/work it can handle.

as in allow for more improvement on Performance and resource usage.

so depending on the size of forum the user is running and the amount of templates such as in my situation. having a mode where the Forum auto adjusts its core settings or having the ability to change some of the core functions of XF itself on how it process data and functions overall to suite the end users growing needs.

Atm the way i see it XF has a max cap once you hit that cap you cannot grow any further without sacrificing performance.

i recon in the end regarding performance and server resources. Admins should have more control over Xenforo Config & Allow for more powerful data handling Modules/Process Methods depending on where xenforo is hosted. let XF take full potential of a powerful server rather then being caped...

I know i have written a lot or repeated my self a few times but i hope i got my point across.

like as for the moment there is no Limit on the amount of Nodes, Themes, Addones you can have etc.. but significantly impacts the workload/performance

so if more time and effort is focused on performance and some of the ideas i listed is implemented i recon XF would be well balanced between performance and the users needs.

So in the Current Scenario:
The Workload/Performance can only be managed by having less Features installed.

For a well balanced Scenario:
It should be the Board to go as high as the user specify's
User wishes for a large community with features without performance decrease
the board should be able to meet the users needs aslong the board is hosted on a server that has enough resources to spare to suite the users needs. as the board grows more resources are used and if it hits to a point where it hits a warning/danger limit of the resources being allocated the board would start only then decreasing Performance. or the user deletes some features then.

 

[XxUnkn0wnxX]

well if this won't happen any time soon

would it be possible for some one to develop an add one / module for me for a price OFC to suite my needs?
such as the multithreaded PHP processing & other Improvements such as allow XF to use more of server resources then the default Built in Core Standards...

as in, is it something that can be swapped out easily and replaced by something better,
or would this be something to complex? - as in to many things would change within xenforo & module would need to be constantly updated as new versions of xenforo come out.

 

[XxUnkn0wnxX]

and one last quickie

just to be clear time on rebuilding is solely on templates right?

what about the amount of Nodes, Members, Posts Etc. do they contribute in any way? to the workload/rebuilding time? (my thinking is do they contribute as a template/page?)

 

[Mike]

all i am asking is that will XF ever grow to a point where it suites every ones needs? as for me i would say my needs are keep piling on features without performance decrease and the board auto adjusting to suite the users needs without a max cap on the amount of data/work it can handle.

By definition, not software will ever meet or suit everyone's needs. If you have specific or niche needs, you will need to get custom development done to fit those needs if it's important to you. Whether the cost (up front and potentially ongoing) is worthwhile is something you'd have to determine after discussing the changes with a developer.

No software can "keep piling on features" without performance overhead and penalties. Everything has a cost and everything has a tradeoff.

The design of the template system optimizes for run-time performance by doing a lot of pre-compilation work. The tradeoff is taking longer in the compilation step for faster page display.

what about the amount of Nodes, Members, Posts Etc. do they contribute in any way? to the workload/rebuilding time? (my thinking is do they contribute as a template/page?)

They don't contribute to template rebuild time.