XF 2.1 Slow installation

Gilly

Member
I have a friend that gave me permission to access this area. Is there a reason why the installation is very slow?

We have three test benches-- here are the specs for each:

Linux Testbench:

OS Tried: Ubuntu 18.04, 16.04. Debian 7, 8, 9
CPU: Intel Xeon E5-1630v3
RAM: 64GB DDR4 ECC 2133 MHz
HDD: 2TB 7200 RPM


Windows Testbench 1:

OS Tried: Windows Server 2008R2, 2012R2, 2016, 2019
CPU: 2x Intel Xeon X5690
RAM: 128GB DDR3 RAM 1600 MHz
SSD: 2x Seagate SSDs
HDD: 4x Western Digital SAS 10K RPM


Windows Testbench 2:

OS Tried: Windows Server 2008R2, 2012R2, 2016, 2019
CPU: 2x Intel Xeon Gold 6244
RAM: 192GB DDR4 2666 MHz
SSD: 6x Samsung Pro

On all of our installation tests, we have seen that it ranges from 1 hour and 45 min to nearly 5 hours. We did find some posts on here Xenforo.com that said "it should only take a couple of minutes" from one of the moderators. We have tried it on an SSD as well as on an HDD. I have tried it on my local computer I will list the specs below, which is the only time it took less than 5 minutes:

CPU: Intel i9 9900K
CPU Cooler: Corsair H115i RGB Platinum
SSD: Samsung SSD 970 EVO Plus
RAM: 32 GB DDR4 3000 MHz Tridentz RGB
 

Steve F

Well-known member
Surely there should be logs to give indication what to look at?

What step is taking so long when the installation is in progress?

You can't expect someone to have the answer to this, monitor the performance of the server while the installation is in progress.
 
Surely there should be logs to give indication what to look at?

What step is taking so long when the installation is in progress?

You can't expect someone to have the answer to this, monitor the performance of the server while the installation is in progress.

When I do extensive testing on over three different platforms, all with similar installation times, I would have expected others to have come forward and mention that it is a known phenomena on certain configurations, perhaps given some particular reason.

Instead, I've gotten a runaround from a moderator giving a couple of links that was either too lazy to read the entire post that I made, or doesn't know what he's talking about, who has continued to give vague and unhelpful responses. If he isn't aware of any solutions then fine— it should be left at that, and perhaps one of the software developers can step in and share their experiences. I'm a software developer myself— I understand that support staff are usually the first ones to respond to these issues, but sometimes oddball issues like this can be better resolved if they can step in and either ask some questions to provide some feedback or otherwise state that (for example) that "...in-fact, due to the methods used to deal with the database, some operations can take extended periods of time on certain hardware and/or configurations".

I will probably get a warning or violation for writing the way I did in my previous post, but that is precisely how I feel. The way the moderator responded for the most part seemed to carry the tone of not caring, and half-assed suggestions. Looking into his profile and past post history, it seems that perhaps that is just his/her general tone, so I apologize if I took it the wrong way.

I don't want to waste time creating posts when I haven't checked things out thoroughly myself. This is not a simple "unplug it, and plug it back in" or "reboot the server" solution. I took a considerable amount of time to think about writing the post in order to not waste everyone's time, all system specs are included for each test bench. In the end, I expected to have a civil discussion about the matter since I have been discussing this internally with my own staff for some time now. I tried my best to make it clear that we're running fairly average hardware and software configs, so that it's clear that we're not running on some sort of unreliable or oddball hardware.

Surely there should be logs to give indication what to look at?

None of the logs have given indication of any issues.

What step is taking so long when the installation is in progress?

Phrases take the longest. After installation, rebuilding things also seem to take the longest.
 
  • Like
Reactions: Xon

Liam W

in memoriam 1998-2020
Phrases take the longest. After installation, rebuilding things also seem to take the longest.

I wonder if you're hitting some sort of file IO limit. I'm not going to claim to be an expert on hardware or why it might be so non-performant, but phrase rebuilds do also cause a template recompile, and a PHP file is written to the filesystem for each style and language (including the master style/language), so it could be that something is causing the writing of the template files to take longer than necessary.
 
I wonder if you're hitting some sort of file IO limit. I'm not going to claim to be an expert on hardware or why it might be so non-performant, but phrase rebuilds do also cause a template recompile, and a PHP file is written to the filesystem for each style and language (including the master style/language), so it could be that something is causing the writing of the template files to take longer than necessary.

Interesting, perhaps that could be something causing issues. Thanks for the info.
 

TickTackk

Well-known member
So from my testings XF will take ~7 minutes to install when innodb_flush_log_at_trx_commit is set to 1 (default) and when set to 2 it takes less than 45 seconds.

Edit:
@ClareXoBearrx3R9 Are you running XF on SSD or HDD? If HDD then slow performance is very much expected.

Probably try with these configs shared by @Xon for SSDs:
Code:
innodb_flush_neighbors=0
innodb_read_io_threads=64
innodb_write_io_threads=64
innodb_io_capacity=3000

If you end up using nvme then innodb_io_capacity can be much higher.
 
Last edited:

Xon

Well-known member
During an XenForo template/phrase rebuilding; there is a huge number of single statement write transactions and is full of N+1 read query behaviour. And this process also triggers css cache invalidation horrifyingly frequently which is an issue for a live site doing an add-on update.

innodb_flush_log_at_trx_commit has the following possible values;
  • 0 Nothing is done on commit; rather the log buffer is written and flushed to the InnoDB redo log once a second. This gives better performance, but a server crash can erase the last second of transactions.
  • 1 The default, the log buffer is written to the InnoDB redo log file and a flush to disk performed after each transaction. This is required for full ACID compliance.
  • 2 The log buffer is written to the InnoDB redo log after each commit, but flushing takes place once a second. Performance is slightly better, but a OS or power outage can cause the last second's transactions to be lost.
The lack of batching in XenForo is what makes the innodb_flush_log_at_trx_commit=2 change such a massive performance improvement, as the default value prevents MySQL & the operating system from batching writes in the background.
 

Masetrix

Well-known member
So from my testings XF will take ~7 minutes to install when innodb_flush_log_at_trx_commit is set to 1 (default) and when set to 2 it takes less than 45 seconds.

Edit:
@ClareXoBearrx3R9 Are you running XF on SSD or HDD? If HDD then slow performance is very much expected.

Probably try with these configs shared by @Xon for SSDs:
Code:
innodb_flush_neighbors=0
innodb_read_io_threads=64
innodb_write_io_threads=64
innodb_io_capacity=3000

If you end up using nvme then innodb_io_capacity can be much higher.

This is exactly my experience with these settings when using InnoDB and using SSDs

I do not post any settings here because they depend very much on the circumstances and settings of the respective server and its software and, according to their own statements, professionals are at work who need to know such things.

The problem here is not due to the forum software. ;)
 
So from my testings XF will take ~7 minutes to install when innodb_flush_log_at_trx_commit is set to 1 (default) and when set to 2 it takes less than 45 seconds.

Edit:
@ClareXoBearrx3R9 Are you running XF on SSD or HDD? If HDD then slow performance is very much expected.

Probably try with these configs shared by @Xon for SSDs:
Code:
innodb_flush_neighbors=0
innodb_read_io_threads=64
innodb_write_io_threads=64
innodb_io_capacity=3000

If you end up using nvme then innodb_io_capacity can be much higher.

Thanks a lot for all the testing and reporting of the results. Definitely good to know the performance impact that has. In-fact, our concern was not necessarily about our installation times of 10-20 minutes greater than reported, but rather times that are hours longer.

And while we are currently running on SSDs, we were previously running on HDDs. I know people enjoy SSDs for their speed, but for enterprise use they are extremely expensive unless we're purchasing consumer-grade SSDs which simply don't have an acceptable durability for heavy enterprise use. But due to the extremes seen in this software, we switched to SSDs for the website.

I know HDDs are much slower than SSDs, but in-practice they work just fine for our use case and can still get fairly high write speeds of over 200MB/s which should be more than enough for webserver and basic database transactions where a forum is concerned.

And to further prove my point, XenForo is the only forum software so far that I have seen suffer from such an issue. I've used more than half a dozen forum software, and none of them have had this issue. So understandably, it ought to be reasonable for me to suspect XenForo is doing something out of the ordinary to cause this behavior.

During an XenForo template/phrase rebuilding; there is a huge number of single statement write transactions and is full of N+1 read query behaviour. And this process also triggers css cache invalidation horrifyingly frequently which is an issue for a live site doing an add-on update.

innodb_flush_log_at_trx_commit has the following possible values;
  • 0 Nothing is done on commit; rather the log buffer is written and flushed to the InnoDB redo log once a second. This gives better performance, but a server crash can erase the last second of transactions.
  • 1 The default, the log buffer is written to the InnoDB redo log file and a flush to disk performed after each transaction. This is required for full ACID compliance.
  • 2 The log buffer is written to the InnoDB redo log after each commit, but flushing takes place once a second. Performance is slightly better, but a OS or power outage can cause the last second's transactions to be lost.
The lack of batching in XenForo is what makes the innodb_flush_log_at_trx_commit=2 change such a massive performance improvement, as the default value prevents MySQL & the operating system from batching writes in the background.

This is an excellent and very intelligible explanation of what XenForo is doing, as well as the explanation about the config for one of the database settings. I very much appreciate this. While we are using a value of 1 for this for stability reasons, it definitely helps to know XenForo's possible weaknesses (for instance, the lack of batching) could be a big culprit in causing poor performance.

This is exactly my experience with these settings when using InnoDB and using SSDs

I do not post any settings here because they depend very much on the circumstances and settings of the respective server and its software and, according to their own statements, professionals are at work who need to know such things.

Totally understandable— those reasons are exactly the same as why I usually don't post configs since there's so much variance and such.
The problem here is not due to the forum software. ;)

One might argue that, however, from my experience other forum software doesn't suffer from these issues. :)

Popular forum software such as phpBB3, MyBB install all under 5 minutes on even the older HDDs, so it's reasonable for us to wonder why XenForo struggles so greatly on even a more modern storage device. Even Invision Power Services community suite, when installing it, we'd get installation times of about 30 minutes on our slowest hardware, and I'm not a huge fan of their bloated software. So we had some reasonable concern when installing XenForo, which for us only consists of the forum software without any addons.

One of our admins' latest tests on a mechanical hard drive with stock/default database settings show a much faster installation of ~30 minutes in a Linux install, while an identical setup on a Windows install took about 4.5 hours.

Quite frankly, I'm surprised that so many shared hosting companies are offering SSDs now... so perhaps that is one of the issues, where most people are not noticing long installation times due to the hardware. In fairness it has been a while since I've worked at a shared-hosting company (last time I worked there was about 2012). Are they rich? Or just using some generic SSDs?

Per our organization's policies, whenever we are to use an SSD it must be either SLC or MLC for endurance reasons. Even the slower SATA form-factor SSD with only 512GB capacity with SLC is going for about $5,000 USD where we are, which is quite a hefty price tag for such a small capacity of storage. And of course, the NVMe form-factor will only be more expensive.

Since we have high capacity storage needs, we mostly use hard drives, currently a number of hard drives with 16TB (with ~14.5TB usable) of individual capacity. Currently no RAID configurations as we didn't want to introduce any unnecessary complications just yet... but that's another matter of its own.

Anyway...
All in all, I very much appreciate all the constructive replies from fellow members of the community— that is ultimately what we were looking for, an understanding on what could possibly be causing such skewed installation times for us compared to the majority of the population.

Cheers!

---

Edit: On another note, I wish it didn't get so heated previously, but hope that the staff can understand the situation here. With the information I gave and replies I was getting from staff, those results were logically impossible, meaning that either someone is lying, or extremely misinformed. And I don't appreciate useless remarks that seemingly serve no purpose whatsoever. You might be a staff member, but you should not be above your own rules— because if you are, I want nothing to do with it.

Yes, I have.

Things like this are what I'm referring to.
If it's not sarcasm, then perhaps you should feature your post as the most helpful post in the entire community.

You are running a business which I understand, but this comment to me was extremely unprofessional. And I would hope I never get talked to like this from a staff member again.
 
Last edited:

Xon

Well-known member
This is an excellent and very intelligible explanation of what XenForo is doing, as well as the explanation about the config for one of the database settings. I very much appreciate this. While we are using a value of 1 for this for stability reasons, it definitely helps to know XenForo's possible weaknesses (for instance, the lack of batching) could be a big culprit in causing poor performance.
The value of 2 is reasonably safe for what is basically a forum, and has really large performance improvements in generate write heavy workloads.

But I absolutely recommend some basic MySQL tuning; as historically the defaults are absolutely awful and left a staggering amount of hardware performance unused. A huge amount of Percona's performance improvements are sane defaults over stock MySQL standards.

One might argue that, however, from my experience other forum software doesn't suffer from these issues. :)
XenForo mostly front-loads template handling in the install step which allows faster page handling but slower installs/updates.

However; XenForo 2.x has a heavy object-oriented design rather than data oriented design, which pushes for operations around a single entity at a time rather than batched operation as the default. This is fine in what is often single entity (or a few entity updates), but tends to not give as-good performance for bulk update operations like template recompiling unless handled with care.

The code base is also much newer compared to XF1.x, and as such lacks nearly a decade of polish that XF1.x had.

Quite frankly, I'm surprised that so many shared hosting companies are offering SSDs now... so perhaps that is one of the issues, where most people are not noticing long installation times due to the hardware. In fairness it has been a while since I've worked at a shared-hosting company (last time I worked there was about 2012). Are they rich? Or just using some generic SSDs?
Disks are consumables; at scale you plan for them to fail as part of your operating budget. Even with SSD wearing, spinning harddrive failure rates are surprisingly high in comparison.
 
The value of 2 is reasonably safe for what is basically a forum, and has really large performance improvements in generate write heavy workloads.

But I absolutely recommend some basic MySQL tuning; as historically the defaults are absolutely awful and left a staggering amount of hardware performance unused. A huge amount of Percona's performance improvements are sane defaults over stock MySQL standards.

Absolutely. And that makes perfect sense. Our production server have been tuned accordingly, only our test benches were left at default for purposes of simple as well as equal and fair testing.

XenForo mostly front-loads template handling in the install step which allows faster page handling but slower installs/updates.

However; XenForo 2.x has a heavy object-oriented design rather than data oriented design, which pushes for operations around a single entity at a time rather than batched operation as the default. This is fine in what is often single entity (or a few entity updates), but tends to not give as-good performance for bulk update operations like template recompiling unless handled with care.

The code base is also much newer compared to XF1.x, and as such lacks nearly a decade of polish that XF1.x had.

I suspect this might have a large impact on the performance of the entire process given the status quo.

Disks are consumables; at scale you plan for them to fail as part of your operating budget. Even with SSD wearing, spinning harddrive failure rates are surprisingly high in comparison.

While I do agree, historically (at least in my experience) it's much easier to notice failing hard drives than SSDs. I myself am a bit biased against SSDs in general given they suffer far more with write operations and because for decades I have had fairly consistent results with the old-fashioned mechanical spinning platters. And in all honesty, the speeds of all modern drives are more than enough for day to day operations on any good OS. On the consumer side, I often see people getting SSDs to save 5-10 seconds at boot time, which is quite laughable. My HDD systems all boot in under 1 minute, some even under 30 seconds. So personally, I don't see the point in paying such premium prices to save such little time, but then again I'm old fashioned.

On the other hand, there's also no arguing that the cost to capacity still leaves much to be desired. Even in a well-off company, one ought to think twice about a $5,000 medium that can hold less than 512GB of data when, for the same price, they could be getting over 181TB of usable storage from an array of spinning disks.

These fast and small storage solutions might be doable for those with little data but by no means IMHO should it be a requirement in running a simple website or forum. In this day and age, if any of my software were to behave so poorly, it would be unacceptable and likely point to some serious design flaw.
 
That's not the only reason why people get SSDs :p

Haha, fair enough!

Just an update to everyone's recommendations, it seems to have definitely helped.

One one of the Windows boxes running on an HDD, our admins were able to get an install from about 3 hours to 1 hour. Given the circumstances with the software and the things going on in the background, even though it's still quite a while, that is far better so we're definitely far more satisfied with these results. Windows seems to have something about it that makes installations take longer than usual when it comes to XF... but it is Windows so who knows LOL.

I suppose the takeaway for XF is— as for all software— one would most certainly want to fine-tune their DB server(s), run it on an SSD, and preferably on a Linux box... which is quite typical anyway is Linux is usually the OS of choice for web-hosting :)

Anyhow, cheers to everyone!
 

Marian-Sebastian

Active member
Obviously something it`s wrong. I`m not a coder, when I install the platform, I think I did it in 6-8 minutes, on shared hosting (namecheap).
Just a simply question... do you guys have some backgroup programs running on the machine?
 
@ClareXoBearrx3R9 why you guys don't try to install the platform in a simply desktop / laptop localhost... just to see what happen.

If you check our initial posts, we did do that, and those were the only times we could get installation to complete in a few minutes :)

Can you try firing up a generic centos install, loading it with mariadb 10.2 or 10.3, generic php 7.2+ and apache out the box and see how that performs.

Ive done installs recently on more optimized setups and the install times are seconds....

Very interesting. We just did this on a local workstation and did two separate tests with CentOS:
  • Installation on an external hard drive took about 5 and a half minutes
  • Installation on an internal hard drive took about 5 minutes
So that is much better. This is on the local workstation with the i9-9900K, running in a CentOS VM under VMWare.

All in all, we're quite happy with these results, but still wonder why previously and possibly in Windows the installation times are so skewed.... again mind you, other software like phpBB3 and MyBB install within minutes. Even if XF has more features, we'd imagine it shouldn't contribute to such a significant increase as we've seen.

We'll try to do some more tests perhaps with some Windows VMs to see if we get any better results on this workstation.
 

MySiteGuy

Well-known member
This is unlikely to be a hardware issue, almost certainly an issue with settings. HDD won't slow this down, it's not a huge amount of data and should fit into the buffer pool and not have a lot of writing via the IO threads. The hardware specs aren't terribly relevant to this, even in very minimal environments it performs installs fast.

Post your my.cnf (or server.cnf) file. That will tell us much more.

I assume with your background you've checked Innodb stats, what are they telling you?
 
Top