XF 2.1 Slow installation

Gilly

Member
I have a friend that gave me permission to access this area. Is there a reason why the installation is very slow?

We have three test benches-- here are the specs for each:

Linux Testbench:

OS Tried: Ubuntu 18.04, 16.04. Debian 7, 8, 9
CPU: Intel Xeon E5-1630v3
RAM: 64GB DDR4 ECC 2133 MHz
HDD: 2TB 7200 RPM


Windows Testbench 1:

OS Tried: Windows Server 2008R2, 2012R2, 2016, 2019
CPU: 2x Intel Xeon X5690
RAM: 128GB DDR3 RAM 1600 MHz
SSD: 2x Seagate SSDs
HDD: 4x Western Digital SAS 10K RPM


Windows Testbench 2:

OS Tried: Windows Server 2008R2, 2012R2, 2016, 2019
CPU: 2x Intel Xeon Gold 6244
RAM: 192GB DDR4 2666 MHz
SSD: 6x Samsung Pro

On all of our installation tests, we have seen that it ranges from 1 hour and 45 min to nearly 5 hours. We did find some posts on here Xenforo.com that said "it should only take a couple of minutes" from one of the moderators. We have tried it on an SSD as well as on an HDD. I have tried it on my local computer I will list the specs below, which is the only time it took less than 5 minutes:

CPU: Intel i9 9900K
CPU Cooler: Corsair H115i RGB Platinum
SSD: Samsung SSD 970 EVO Plus
RAM: 32 GB DDR4 3000 MHz Tridentz RGB
 
This is unlikely to be a hardware issue, almost certainly an issue with settings. HDD won't slow this down, it's not a huge amount of data and should fit into the buffer pool and not have a lot of writing via the IO threads. The hardware specs aren't terribly relevant to this, even in very minimal environments it performs installs fast.

I would hope it shouldn't be such a problem, but XF is certainly the only software so far that has been so terribly slow to install. All other forum software I've tried (as mentioned previously) installs very quickly. So I would assume, everything else being equal, that it would be an "issue" related to either the software or... perhaps the DB is not optimized for this particular application.

I assume with your background you've checked Innodb stats, what are they telling you?

I should have clarified— I'm not a professional DBA— I only meant to indicate that I am the guy responsible for all of the stuff (network, db, systems, etc). Professionally, I'm actually a software person.

Post your my.cnf (or server.cnf) file. That will tell us much more.

Sure, here's my INI file (this is a DB server running MySQL Server 8.0.11 on Windows Server 2008 R2):

INI:
# Other default tuning values
# MySQL Server Instance Configuration File
# ----------------------------------------------------------------------
# Generated by the MySQL Server Instance Configuration Wizard
#
#
# Installation Instructions
# ----------------------------------------------------------------------
#
# On Linux you can copy this file to /etc/my.cnf to set global options,
# mysql-data-dir/my.cnf to set server-specific options
# (@localstatedir@ for this installation) or to
# ~/.my.cnf to set user-specific options.
#
# On Windows you should keep this file in the installation directory
# of your server (e.g. C:\Program Files\MySQL\MySQL Server X.Y). To
# make sure the server reads the config file use the startup option
# "--defaults-file".
#
# To run the server from the command line, execute this in a
# command line shell, e.g.
# mysqld --defaults-file="C:\Program Files\MySQL\MySQL Server X.Y\my.ini"
#
# To install the server as a Windows service manually, execute this in a
# command line shell, e.g.
# mysqld --install MySQLXY --defaults-file="C:\Program Files\MySQL\MySQL Server X.Y\my.ini"
#
# And then execute this in a command line shell to start the server, e.g.
# net start MySQLXY
#
#
# Guidelines for editing this file
# ----------------------------------------------------------------------
#
# In this file, you can use all long options that the program supports.
# If you want to know the options a program supports, start the program
# with the "--help" option.
#
# More detailed information about the individual options can also be
# found in the manual.
#
# For advice on how to change settings please see
# http://dev.mysql.com/doc/refman/5.7/en/server-configuration-defaults.html
#
#
# CLIENT SECTION
# ----------------------------------------------------------------------
#
# The following options will be read by MySQL client applications.
# Note that only client applications shipped by MySQL are guaranteed
# to read this section. If you want your own MySQL client program to
# honor these values, you need to specify it as an option during the
# MySQL client library initialization.
#
[client]

# pipe=

# socket=MYSQL

port=3306

[mysql]
no-beep

# default-character-set=

# SERVER SECTION
# ----------------------------------------------------------------------
#
# The following options will be read by the MySQL Server. Make sure that
# you have installed the server correctly (see above) so it reads this
# file.
#
# server_type=1
[mysqld]

# The next three options are mutually exclusive to SERVER_PORT below.
# skip-networking
# enable-named-pipe
# shared-memory

# shared-memory-base-name=MYSQL

# The Pipe the MySQL Server will use
# socket=MYSQL

# The TCP/IP Port the MySQL Server will listen on
port=3306

# Path to installation directory. All paths are usually resolved relative to this.
# basedir="C:/Program Files/MySQL/MySQL Server 8.0/"

# Path to the database root
datadir=T:/Data

# The default character set that will be used when a new schema or table is
# created and no character set is defined
# character-set-server=

# The default authentication plugin to be used when connecting to the server
default_authentication_plugin=mysql_native_password

# The default storage engine that will be used when create new tables when
default-storage-engine=INNODB

# Set the SQL mode to strict
# sql-mode="STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION"
sql-mode=""

# General and Slow logging.
log-output=FILE
general-log=0
general_log_file="DB1.log"
slow-query-log=1
slow_query_log_file="DB1-slow.log"
long_query_time=10

# Binary Logging.
log-bin="DB1-bin"

# Error Logging.
log-error="DB1.err"

# Server Id.
server-id=1

# Specifies the on how table names are stored in the metadata.
# If set to 0, will throw an error on case-insensitive operative systems
# If set to 1, table names are stored in lowercase on disk and comparisons are not case sensitive.
# If set to 2, table names are stored as given but compared in lowercase.
# This option also applies to database names and table aliases.
# NOTE: Modify this value after Server initialization won't take effect.
lower_case_table_names=1

# Secure File Priv.
secure-file-priv="C:/ProgramData/MySQL/MySQL Server 8.0/Uploads"

# The maximum amount of concurrent sessions the MySQL server will
# allow. One of these connections will be reserved for a user with
# SUPER privileges to allow the administrator to login even if the
# connection limit has been reached.
max_connections=151

# The number of open tables for all threads. Increasing this value
# increases the number of file descriptors that mysqld requires.
# Therefore you have to make sure to set the amount of open files
# allowed to at least 4096 in the variable "open-files-limit" in
# section [mysqld_safe]
table_open_cache=2000

# Maximum size for internal (in-memory) temporary tables. If a table
# grows larger than this value, it is automatically converted to disk
# based table This limitation is for a single table. There can be many
# of them.
tmp_table_size=312M

# How many threads we should keep in a cache for reuse. When a client
# disconnects, the client's threads are put in the cache if there aren't
# more than thread_cache_size threads from before.  This greatly reduces
# the amount of thread creations needed if you have a lot of new
# connections. (Normally this doesn't give a notable performance
# improvement if you have a good thread implementation.)
thread_cache_size=10

#*** MyISAM Specific options
# The maximum size of the temporary file MySQL is allowed to use while
# recreating the index (during REPAIR, ALTER TABLE or LOAD DATA INFILE.
# If the file-size would be bigger than this, the index will be created
# through the key cache (which is slower).
myisam_max_sort_file_size=100G

# If the temporary file used for fast index creation would be bigger
# than using the key cache by the amount specified here, then prefer the
# key cache method.  This is mainly used to force long character keys in
# large tables to use the slower key cache method to create the index.
myisam_sort_buffer_size=611M

# Size of the Key Buffer, used to cache index blocks for MyISAM tables.
# Do not set it larger than 30% of your available memory, as some memory
# is also required by the OS to cache rows. Even if you're not using
# MyISAM tables, you should still set it to 8-64M as it will also be
# used for internal temporary disk tables.
key_buffer_size=8M

# Size of the buffer used for doing full table scans of MyISAM tables.
# Allocated per thread, if a full scan is needed.
read_buffer_size=64K

read_rnd_buffer_size=256K

#*** INNODB Specific options ***
# innodb_data_home_dir=

# Use this option if you have a MySQL server with InnoDB support enabled
# but you do not plan to use it. This will save memory and disk space
# and speed up some things.
# skip-innodb

# If set to 1, InnoDB will flush (fsync) the transaction logs to the
# disk at each commit, which offers full ACID behavior. If you are
# willing to compromise this safety, and you are running small
# transactions, you may set this to 0 or 2 to reduce disk I/O to the
# logs. Value 0 means that the log is only written to the log file and
# the log file flushed to disk approximately once per second. Value 2
# means the log is written to the log file at each commit, but the log
# file is only flushed to disk approximately once per second.
innodb_flush_log_at_trx_commit=1

# The size of the buffer InnoDB uses for buffering log data. As soon as
# it is full, InnoDB will have to flush it to disk. As it is flushed
# once per second anyway, it does not make sense to have it very large
# (even with long transactions).
innodb_log_buffer_size=1M

# InnoDB, unlike MyISAM, uses a buffer pool to cache both indexes and
# row data. The bigger you set this the less disk I/O is needed to
# access data in tables. On a dedicated database server you may set this
# parameter up to 80% of the machine physical memory size. Do not set it
# too large, though, because competition of the physical memory may
# cause paging in the operating system.  Note that on 32bit systems you
# might be limited to 2-3.5G of user level memory per process, so do not
# set it too high.
innodb_buffer_pool_size=8M

# Size of each log file in a log group. You should set the combined size
# of log files to about 25%-100% of your buffer pool size to avoid
# unneeded buffer pool flush activity on log file overwrite. However,
# note that a larger logfile size will increase the time needed for the
# recovery process.
innodb_log_file_size=48M

# Number of threads allowed inside the InnoDB kernel. The optimal value
# depends highly on the application, hardware as well as the OS
# scheduler properties. A too high value may lead to thread thrashing.
innodb_thread_concurrency=8

# The increment size (in MB) for extending the size of an auto-extend InnoDB system tablespace file when it becomes full.
innodb_autoextend_increment=64

# The number of regions that the InnoDB buffer pool is divided into.
# For systems with buffer pools in the multi-gigabyte range, dividing the buffer pool into separate instances can improve concurrency,
# by reducing contention as different threads read and write to cached pages.
innodb_buffer_pool_instances=8

# Determines the number of threads that can enter InnoDB concurrently.
innodb_concurrency_tickets=5000

# Specifies how long in milliseconds (ms) a block inserted into the old sublist must stay there after its first access before
# it can be moved to the new sublist.
innodb_old_blocks_time=1000

# It specifies the maximum number of .ibd files that MySQL can keep open at one time. The minimum value is 10.
innodb_open_files=300

# When this variable is enabled, InnoDB updates statistics during metadata statements.
innodb_stats_on_metadata=0

# When innodb_file_per_table is enabled (the default in 5.6.6 and higher), InnoDB stores the data and indexes for each newly created table
# in a separate .ibd file, rather than in the system tablespace.
innodb_file_per_table=1

# Use the following list of values: 0 for crc32, 1 for strict_crc32, 2 for innodb, 3 for strict_innodb, 4 for none, 5 for strict_none.
innodb_checksum_algorithm=0

# The number of outstanding connection requests MySQL can have.
# This option is useful when the main MySQL thread gets many connection requests in a very short time.
# It then takes some time (although very little) for the main thread to check the connection and start a new thread.
# The back_log value indicates how many requests can be stacked during this short time before MySQL momentarily
# stops answering new requests.
# You need to increase this only if you expect a large number of connections in a short period of time.
back_log=80

# If this is set to a nonzero value, all tables are closed every flush_time seconds to free up resources and
# synchronize unflushed data to disk.
# This option is best used only on systems with minimal resources.
flush_time=0

# The minimum size of the buffer that is used for plain index scans, range index scans, and joins that do not use
# indexes and thus perform full table scans.
join_buffer_size=256K

# The maximum size of one packet or any generated or intermediate string, or any parameter sent by the
# mysql_stmt_send_long_data() C API function.
max_allowed_packet=4M

# If more than this many successive connection requests from a host are interrupted without a successful connection,
# the server blocks that host from performing further connections.
max_connect_errors=100

# Changes the number of file descriptors available to mysqld.
# You should try increasing the value of this option if mysqld gives you the error "Too many open files".
open_files_limit=4161

# If you see many sort_merge_passes per second in SHOW GLOBAL STATUS output, you can consider increasing the
# sort_buffer_size value to speed up ORDER BY or GROUP BY operations that cannot be improved with query optimization
# or improved indexing.
sort_buffer_size=256K

# The number of table definitions (from .frm files) that can be stored in the definition cache.
# If you use a large number of tables, you can create a large table definition cache to speed up opening of tables.
# The table definition cache takes less space and does not use file descriptors, unlike the normal table cache.
# The minimum and default values are both 400.
table_definition_cache=1400

# Specify the maximum size of a row-based binary log event, in bytes.
# Rows are grouped into events smaller than this size if possible. The value should be a multiple of 256.
binlog_row_event_max_size=8K

# If the value of this variable is greater than 0, a replication slave synchronizes its master.info file to disk.
# (using fdatasync()) after every sync_master_info events.
sync_master_info=10000

# If the value of this variable is greater than 0, the MySQL server synchronizes its relay log to disk.
# (using fdatasync()) after every sync_relay_log writes to the relay log.
sync_relay_log=10000

# If the value of this variable is greater than 0, a replication slave synchronizes its relay-log.info file to disk.
# (using fdatasync()) after every sync_relay_log_info transactions.
sync_relay_log_info=10000

# Load mysql plugins at start."plugin_x ; plugin_y".
# plugin_load

# The TCP/IP Port the MySQL Server X Protocol will listen on.
loose_mysqlx_port=33060

character-set-server = utf8
 

Max Taxable

Well-known member
All other forum software I've tried (as mentioned previously) installs very quickly. So I would assume, everything else being equal, that it would be an "issue" related to either the software or... perhaps the DB is not optimized for this particular application.
You're the first I've seen or heard about to have issues with xF being slow to install. If it were a problem with the software it would be a problem for most, the math says. It's not slow to install. For you perhaps it was, this is an outlier case. A rare example. Uncommon.
 
You're the first I've seen or heard about to have issues with xF being slow to install. If it were a problem with the software it would be a problem for most, the math says.

Do you live under a rock? A simple half-hearted search on a public search engine yields numerous results:


And if you were to search the forum then you would see even more. If you're going to preach statistics, then actually do your research to back up your claims. Otherwise, you might as well be the outlier case having never heard anyone else having issues.

It's not slow to install. For you perhaps it was, this is an outlier case. A rare example. Uncommon.

You should heed your own advice:

For you perhaps it was, this is an outlier case. A rare example. Uncommon.

I have only seen a few people in this thread suggest they've never had this problem, you included.

Myself as well as two other people that I know have had these issues as well, which we've had at-length discussions about, prior to posting this thread. So it's more than just a single outlier case. So at that point, from my perspective, I could argue the same thing you did:

It is not slow to install. For you perhaps it was not, this is an outlier case. A rare example. Uncommon.

So, are you here to bust my balls are provide any more valuable information that could be helpful? :)

If you bothered to read all the posts in their entirety, you would have seen we've installed on multiple test benches with varying configurations all with similar results.

And like I said before to many, it's easy for people here to say it's uncommon, but this is the only software that has done it. I'm not saying we're not at fault, I'm only saying we've not had these issues with other software-- all fully-featured forum software.

So far, only a couple of the people that have defended XF for not being slow to install have provided any hardware, software, and/or configuration information to help provide some explanation for it.

This thread was created with the intention of hopefully finding potential solutions to our slow installation issues— not to waste time with infantile bickering and senseless pointing of fingers.
 
Last edited by a moderator:

Max Taxable

Well-known member
If you bothered to read all the posts in their entirety
I did do so. My apologies though, for striking a nerve. Wasn't my intention. You seem very touchy.
A simple half-hearted search on a public search engine yields numerous results:
Out of how many installs, 100K or more? It's still statistically insignificant. No doubt you can find as many or more with the other platforms you have had no problems with. Personally having been at this for 15 years, I have had no install trouble with any of them in that time. It's quite safe to drop the idea that it's the software that is causing your problems. It's not at all unreasonable to say it's not a problem with xenforo itself.
https://xenforo.com/community/profile-posts/29752/ You know your PC is slow when XF takes 10 minutes to install
Yes, I'd forgotten the system uses the browser and leaving the page can interrupt the install or upgrade.
 
My apologies though, for striking a nerve. Wasn't my intention. You seem very touchy.

No problem, and my apologies for assuming otherwise. Bad week I guess.

Out of how many installs, 100K or more? It's still statistically insignificant.

See the thing is, not everyone with this problem posts about it. From my experience, most people will not take the time to report or inquire about something if there's not a significant enough of a reason to do so. Since most people would install once, this issue-- if applicable to them-- would be more-or-less, a one-time deal. Because of that, they'd not bother with wasting their time inquiring— after all, for all they know it could be their own fault. So there are bound to be many more people with an issue than that we can find any record or proof of.

Thus, the problem is those who haven't complained will be often incorrectly categorized as "those who don't have a problem or issue", since they never said anything that we know of.

And while it may still be statistically insignificant, we ultimately didn't create the thread to argue about who's right or wrong here— we wanted to know what is or could be wrong.

Personally having been at this for 15 years, I have had no install trouble with any of them in that time. It's quite safe to drop the idea that it's the software that is causing your problems. It's not at all unreasonable to say it's not a problem with xenforo itself.

I've been at this for about the same amount of time you have. For me, only recently was it the first time I've ever tried XenForo while previously having installed many other software such as vBulletin, IPS Community Suite, IP.Board, phpbb3, MyBB, WoltLab Burning Board, and YaBB to name a few. All of those software installations I have never had such issues with. All these years, our configs do not drastically change either. They are optimized as we change hardware perhaps, and sometimes on a per-software basis but beyond that once they work properly, they are generally left alone.

Putting yourself in our shoes for a minute, if you had been doing things for as long as you say you have, and suddenly our software takes exceptionally long to install— not once— but on numerous instances, would it not be fair to inquire if this is a known phenomena with such software?

In reality, that's all we ever were wondering: if anyone had issues like we did and if so, were they able to improve their situation or not and if so, how.
 

Max Taxable

Well-known member
OK so if it's not due to xenforo. Then may I ask what spec do you have? If you said you have been at this for 15 years. You are using the same server configuration for 15 years. Is that right?
100s of installations over the years, most of them on client's cheap shared hosting, specs aren't known. What's known is I've never had any problems regardless. I suppose I'm just lucky.
 

Sim

Well-known member
The installation process is necessarily extremely heavy in write operations - the master data rebuild process is generating all of the data in the database after being loaded from the XML files. Once installation is complete, then write operations will significantly decrease and performance should be less of an issue - provided that you correctly tune your database and have plenty of RAM available for caching. Using a memory-based cache provider like Memcached will be critical for performance with a slow HDD, especially one that is IO bound on writes.

I did some experiments by performing a basic clean install of XenForo 2.1.7 using the CLI installer onto a variety of machines, all running the same Ubuntu 18.04 based LEMP installation built from the same install script.

The results varied quite significantly!

Machine 1: my dev server running on a BinaryLane VPS with 2GB memory, 25GB SSD and 1 vCPU, no other traffic

Time taken to import and rebuild: 708.93s - over 12.5 minutes in total

Machine 2: VirtualBox machine running on my local dev workstation, with virtual disk residing on a 4TB WD Green HDD, 1GB memory allocated to VM (on a machine with 16GB RAM), 1 vCPU allocated (on a machine with 4 cores / 8 processors), no other traffic

Time taken to import and rebuild: 257.25s - 4.5 minutes in total

Machine 3: A brand new BinaryLane VPS with 1GB memory, 20GB SSD and 1vCPU - no other traffic

Time taken to import and rebuild: 83.26s - around 87 seconds in total

Machine 4: The Linode VPS that runs my main XenForo website, with 8GB memory, 160GB SSD and 4 vCPUs - well tuned for running XenForo, but also receiving high traffic to my website during the test

Time taken to import and rebuild: 63.41s - under 70 seconds in total

Machine 5: A brand new Linode VPS freshly built (using the same install script as all the others), but only "Nanode" in size: 1GB memory, 25GB SSD, 1 vCPU - no other traffic

Time taken to import and rebuild: 45.24s - around 46 seconds in total

It seems that not all SSD VPSs are made equal - the BinaryLane VPS is great for the price, but their SSDs are much slower than Linode's - and it shows in the results I got.

I was quite surprised that my VirtualBox VM was so much faster than the SSD VPS, even running on a WD Green drive - but there would be almost zero contention for that drive, since it is only used for storage and my host OS runs off a separate SSD. Also, it's an Intel Core i7 machine (albeit 5+ years old), so it's still pretty beefy - I'm not sure how much impact CPU speed has on the install process.

I think it's clear that my dev VPS (machine 1 above) is running on old hardware - I was surprised it was so slow. I even ran it a second time to verify and it took even longer! It took 47 seconds just to create the 200 tables :eek: Maybe I need to do some more DB tuning on that dev server! Perhaps some clean up too - there's lots of rubbish in that database from years of dev work.
 

Gilly

Member
100s of installations over the years, most of them on client's cheap shared hosting, specs aren't known. What's known is I've never had any problems regardless. I suppose I'm just lucky.
Cheap shared hosting. Do you know where they got them? If it's a big company like namecheap, go daddy, blue host or any other site when you Google recommended shared hosting is not valid here.

So then let me ask you this:

You are able to install any other platform with colors that is just shot under 2 to 3 min?
 

Gilly

Member
The installation process is necessarily extremely heavy in write operations - the master data rebuild process is generating all of the data in the database after being loaded from the XML files. Once installation is complete, then write operations will significantly decrease and performance should be less of an issue - provided that you correctly tune your database and have plenty of RAM available for caching. Using a memory-based cache provider like Memcached will be critical for performance with a slow HDD, especially one that is IO bound on writes.

I did some experiments by performing a basic clean install of XenForo 2.1.7 using the CLI installer onto a variety of machines, all running the same Ubuntu 18.04 based LEMP installation built from the same install script.

The results varied quite significantly!

Machine 1: my dev server running on a BinaryLane VPS with 2GB memory, 25GB SSD and 1 vCPU, no other traffic

Time taken to import and rebuild: 708.93s - over 12.5 minutes in total

Machine 2: VirtualBox machine running on my local dev workstation, with virtual disk residing on a 4TB WD Green HDD, 1GB memory allocated to VM (on a machine with 16GB RAM), 1 vCPU allocated (on a machine with 4 cores / 8 processors), no other traffic

Time taken to import and rebuild: 257.25s - 4.5 minutes in total

Machine 3: A brand new BinaryLane VPS with 1GB memory, 20GB SSD and 1vCPU - no other traffic

Time taken to import and rebuild: 83.26s - around 87 seconds in total

Machine 4: The Linode VPS that runs my main XenForo website, with 8GB memory, 160GB SSD and 4 vCPUs - well tuned for running XenForo, but also receiving high traffic to my website during the test

Time taken to import and rebuild: 63.41s - under 70 seconds in total

Machine 5: A brand new Linode VPS freshly built (using the same install script as all the others), but only "Nanode" in size: 1GB memory, 25GB SSD, 1 vCPU - no other traffic

Time taken to import and rebuild: 45.24s - around 46 seconds in total

It seems that not all SSD VPSs are made equal - the BinaryLane VPS is great for the price, but their SSDs are much slower than Linode's - and it shows in the results I got.

I was quite surprised that my VirtualBox VM was so much faster than the SSD VPS, even running on a WD Green drive - but there would be almost zero contention for that drive, since it is only used for storage and my host OS runs off a separate SSD. Also, it's an Intel Core i7 machine (albeit 5+ years old), so it's still pretty beefy - I'm not sure how much impact CPU speed has on the install process.

I think it's clear that my dev VPS (machine 1 above) is running on old hardware - I was surprised it was so slow. I even ran it a second time to verify and it took even longer! It took 47 seconds just to create the 200 tables :eek: Maybe I need to do some more DB tuning on that dev server! Perhaps some clean up too - there's lots of rubbish in that database from years of dev work.
Also, this need to be known we are not running SSD. We are running mechanical HDD. So we need to know how to fix this for mechanical HDD. Of course SSD is going to perform better and faster. No doubt about that. So can you test on an HDD for us please.
 

Sim

Well-known member
Also, this need to be known we are not running SSD. We are running mechanical HDD. So we need to know how to fix this for mechanical HDD. Of course SSD is going to perform better and faster. No doubt about that. So can you test on an HDD for us please.

I did.

If you actually read my post - my "machine 2" was via a VirtualBox running on my local workstation where the virtual disk was on a consumer-level WD Green HDD.

Either way, the main point of my post was actually that even between SSD installations, install time can vary very significantly.

12.5 minutes to install on one SSD based machine compared to 46 seconds on a lesser-spec'd SSD machine indicates that it's not just the SSD at play here, but no doubt the server config as well.

What else is running on the machine you are trying to install XenForo on? What else is in the database? I'd be very interested to see the output of https://github.com/major/MySQLTuner-perl for your database on your machine.
 

MySiteGuy

Well-known member
I would hope it shouldn't be such a problem, but XF is certainly the only software so far that has been so terribly slow to install. All other forum software I've tried (as mentioned previously) installs very quickly. So I would assume, everything else being equal, that it would be an "issue" related to either the software or... perhaps the DB is not optimized for this particular application.



I should have clarified— I'm not a professional DBA— I only meant to indicate that I am the guy responsible for all of the stuff (network, db, systems, etc). Professionally, I'm actually a software person.



Sure, here's my INI file (this is a DB server running MySQL Server 8.0.11 on Windows Server 2008 R2):

INI:
# Other default tuning values
# MySQL Server Instance Configuration File
# ----------------------------------------------------------------------
# Generated by the MySQL Server Instance Configuration Wizard
#
#
# Installation Instructions
# ----------------------------------------------------------------------
#
# On Linux you can copy this file to /etc/my.cnf to set global options,
# mysql-data-dir/my.cnf to set server-specific options
# (@localstatedir@ for this installation) or to
# ~/.my.cnf to set user-specific options.
#
# On Windows you should keep this file in the installation directory
# of your server (e.g. C:\Program Files\MySQL\MySQL Server X.Y). To
# make sure the server reads the config file use the startup option
# "--defaults-file".
#
# To run the server from the command line, execute this in a
# command line shell, e.g.
# mysqld --defaults-file="C:\Program Files\MySQL\MySQL Server X.Y\my.ini"
#
# To install the server as a Windows service manually, execute this in a
# command line shell, e.g.
# mysqld --install MySQLXY --defaults-file="C:\Program Files\MySQL\MySQL Server X.Y\my.ini"
#
# And then execute this in a command line shell to start the server, e.g.
# net start MySQLXY
#
#
# Guidelines for editing this file
# ----------------------------------------------------------------------
#
# In this file, you can use all long options that the program supports.
# If you want to know the options a program supports, start the program
# with the "--help" option.
#
# More detailed information about the individual options can also be
# found in the manual.
#
# For advice on how to change settings please see
# http://dev.mysql.com/doc/refman/5.7/en/server-configuration-defaults.html
#
#
# CLIENT SECTION
# ----------------------------------------------------------------------
#
# The following options will be read by MySQL client applications.
# Note that only client applications shipped by MySQL are guaranteed
# to read this section. If you want your own MySQL client program to
# honor these values, you need to specify it as an option during the
# MySQL client library initialization.
#
[client]

# pipe=

# socket=MYSQL

port=3306

[mysql]
no-beep

# default-character-set=

# SERVER SECTION
# ----------------------------------------------------------------------
#
# The following options will be read by the MySQL Server. Make sure that
# you have installed the server correctly (see above) so it reads this
# file.
#
# server_type=1
[mysqld]

# The next three options are mutually exclusive to SERVER_PORT below.
# skip-networking
# enable-named-pipe
# shared-memory

# shared-memory-base-name=MYSQL

# The Pipe the MySQL Server will use
# socket=MYSQL

# The TCP/IP Port the MySQL Server will listen on
port=3306

# Path to installation directory. All paths are usually resolved relative to this.
# basedir="C:/Program Files/MySQL/MySQL Server 8.0/"

# Path to the database root
datadir=T:/Data

# The default character set that will be used when a new schema or table is
# created and no character set is defined
# character-set-server=

# The default authentication plugin to be used when connecting to the server
default_authentication_plugin=mysql_native_password

# The default storage engine that will be used when create new tables when
default-storage-engine=INNODB

# Set the SQL mode to strict
# sql-mode="STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION"
sql-mode=""

# General and Slow logging.
log-output=FILE
general-log=0
general_log_file="DB1.log"
slow-query-log=1
slow_query_log_file="DB1-slow.log"
long_query_time=10

# Binary Logging.
log-bin="DB1-bin"

# Error Logging.
log-error="DB1.err"

# Server Id.
server-id=1

# Specifies the on how table names are stored in the metadata.
# If set to 0, will throw an error on case-insensitive operative systems
# If set to 1, table names are stored in lowercase on disk and comparisons are not case sensitive.
# If set to 2, table names are stored as given but compared in lowercase.
# This option also applies to database names and table aliases.
# NOTE: Modify this value after Server initialization won't take effect.
lower_case_table_names=1

# Secure File Priv.
secure-file-priv="C:/ProgramData/MySQL/MySQL Server 8.0/Uploads"

# The maximum amount of concurrent sessions the MySQL server will
# allow. One of these connections will be reserved for a user with
# SUPER privileges to allow the administrator to login even if the
# connection limit has been reached.
max_connections=151

# The number of open tables for all threads. Increasing this value
# increases the number of file descriptors that mysqld requires.
# Therefore you have to make sure to set the amount of open files
# allowed to at least 4096 in the variable "open-files-limit" in
# section [mysqld_safe]
table_open_cache=2000

# Maximum size for internal (in-memory) temporary tables. If a table
# grows larger than this value, it is automatically converted to disk
# based table This limitation is for a single table. There can be many
# of them.
tmp_table_size=312M

# How many threads we should keep in a cache for reuse. When a client
# disconnects, the client's threads are put in the cache if there aren't
# more than thread_cache_size threads from before.  This greatly reduces
# the amount of thread creations needed if you have a lot of new
# connections. (Normally this doesn't give a notable performance
# improvement if you have a good thread implementation.)
thread_cache_size=10

#*** MyISAM Specific options
# The maximum size of the temporary file MySQL is allowed to use while
# recreating the index (during REPAIR, ALTER TABLE or LOAD DATA INFILE.
# If the file-size would be bigger than this, the index will be created
# through the key cache (which is slower).
myisam_max_sort_file_size=100G

# If the temporary file used for fast index creation would be bigger
# than using the key cache by the amount specified here, then prefer the
# key cache method.  This is mainly used to force long character keys in
# large tables to use the slower key cache method to create the index.
myisam_sort_buffer_size=611M

# Size of the Key Buffer, used to cache index blocks for MyISAM tables.
# Do not set it larger than 30% of your available memory, as some memory
# is also required by the OS to cache rows. Even if you're not using
# MyISAM tables, you should still set it to 8-64M as it will also be
# used for internal temporary disk tables.
key_buffer_size=8M

# Size of the buffer used for doing full table scans of MyISAM tables.
# Allocated per thread, if a full scan is needed.
read_buffer_size=64K

read_rnd_buffer_size=256K

#*** INNODB Specific options ***
# innodb_data_home_dir=

# Use this option if you have a MySQL server with InnoDB support enabled
# but you do not plan to use it. This will save memory and disk space
# and speed up some things.
# skip-innodb

# If set to 1, InnoDB will flush (fsync) the transaction logs to the
# disk at each commit, which offers full ACID behavior. If you are
# willing to compromise this safety, and you are running small
# transactions, you may set this to 0 or 2 to reduce disk I/O to the
# logs. Value 0 means that the log is only written to the log file and
# the log file flushed to disk approximately once per second. Value 2
# means the log is written to the log file at each commit, but the log
# file is only flushed to disk approximately once per second.
innodb_flush_log_at_trx_commit=1

# The size of the buffer InnoDB uses for buffering log data. As soon as
# it is full, InnoDB will have to flush it to disk. As it is flushed
# once per second anyway, it does not make sense to have it very large
# (even with long transactions).
innodb_log_buffer_size=1M

# InnoDB, unlike MyISAM, uses a buffer pool to cache both indexes and
# row data. The bigger you set this the less disk I/O is needed to
# access data in tables. On a dedicated database server you may set this
# parameter up to 80% of the machine physical memory size. Do not set it
# too large, though, because competition of the physical memory may
# cause paging in the operating system.  Note that on 32bit systems you
# might be limited to 2-3.5G of user level memory per process, so do not
# set it too high.
innodb_buffer_pool_size=8M

# Size of each log file in a log group. You should set the combined size
# of log files to about 25%-100% of your buffer pool size to avoid
# unneeded buffer pool flush activity on log file overwrite. However,
# note that a larger logfile size will increase the time needed for the
# recovery process.
innodb_log_file_size=48M

# Number of threads allowed inside the InnoDB kernel. The optimal value
# depends highly on the application, hardware as well as the OS
# scheduler properties. A too high value may lead to thread thrashing.
innodb_thread_concurrency=8

# The increment size (in MB) for extending the size of an auto-extend InnoDB system tablespace file when it becomes full.
innodb_autoextend_increment=64

# The number of regions that the InnoDB buffer pool is divided into.
# For systems with buffer pools in the multi-gigabyte range, dividing the buffer pool into separate instances can improve concurrency,
# by reducing contention as different threads read and write to cached pages.
innodb_buffer_pool_instances=8

# Determines the number of threads that can enter InnoDB concurrently.
innodb_concurrency_tickets=5000

# Specifies how long in milliseconds (ms) a block inserted into the old sublist must stay there after its first access before
# it can be moved to the new sublist.
innodb_old_blocks_time=1000

# It specifies the maximum number of .ibd files that MySQL can keep open at one time. The minimum value is 10.
innodb_open_files=300

# When this variable is enabled, InnoDB updates statistics during metadata statements.
innodb_stats_on_metadata=0

# When innodb_file_per_table is enabled (the default in 5.6.6 and higher), InnoDB stores the data and indexes for each newly created table
# in a separate .ibd file, rather than in the system tablespace.
innodb_file_per_table=1

# Use the following list of values: 0 for crc32, 1 for strict_crc32, 2 for innodb, 3 for strict_innodb, 4 for none, 5 for strict_none.
innodb_checksum_algorithm=0

# The number of outstanding connection requests MySQL can have.
# This option is useful when the main MySQL thread gets many connection requests in a very short time.
# It then takes some time (although very little) for the main thread to check the connection and start a new thread.
# The back_log value indicates how many requests can be stacked during this short time before MySQL momentarily
# stops answering new requests.
# You need to increase this only if you expect a large number of connections in a short period of time.
back_log=80

# If this is set to a nonzero value, all tables are closed every flush_time seconds to free up resources and
# synchronize unflushed data to disk.
# This option is best used only on systems with minimal resources.
flush_time=0

# The minimum size of the buffer that is used for plain index scans, range index scans, and joins that do not use
# indexes and thus perform full table scans.
join_buffer_size=256K

# The maximum size of one packet or any generated or intermediate string, or any parameter sent by the
# mysql_stmt_send_long_data() C API function.
max_allowed_packet=4M

# If more than this many successive connection requests from a host are interrupted without a successful connection,
# the server blocks that host from performing further connections.
max_connect_errors=100

# Changes the number of file descriptors available to mysqld.
# You should try increasing the value of this option if mysqld gives you the error "Too many open files".
open_files_limit=4161

# If you see many sort_merge_passes per second in SHOW GLOBAL STATUS output, you can consider increasing the
# sort_buffer_size value to speed up ORDER BY or GROUP BY operations that cannot be improved with query optimization
# or improved indexing.
sort_buffer_size=256K

# The number of table definitions (from .frm files) that can be stored in the definition cache.
# If you use a large number of tables, you can create a large table definition cache to speed up opening of tables.
# The table definition cache takes less space and does not use file descriptors, unlike the normal table cache.
# The minimum and default values are both 400.
table_definition_cache=1400

# Specify the maximum size of a row-based binary log event, in bytes.
# Rows are grouped into events smaller than this size if possible. The value should be a multiple of 256.
binlog_row_event_max_size=8K

# If the value of this variable is greater than 0, a replication slave synchronizes its master.info file to disk.
# (using fdatasync()) after every sync_master_info events.
sync_master_info=10000

# If the value of this variable is greater than 0, the MySQL server synchronizes its relay log to disk.
# (using fdatasync()) after every sync_relay_log writes to the relay log.
sync_relay_log=10000

# If the value of this variable is greater than 0, a replication slave synchronizes its relay-log.info file to disk.
# (using fdatasync()) after every sync_relay_log_info transactions.
sync_relay_log_info=10000

# Load mysql plugins at start."plugin_x ; plugin_y".
# plugin_load

# The TCP/IP Port the MySQL Server X Protocol will listen on.
loose_mysqlx_port=33060

character-set-server = utf8

Even slow consumer-level hard drives have more than adequate transfer rates for the amount of data being transferred during installs. SSD or HDD, it's not going to make a difference with a rebuild if mysql is optimal, and the example hardware you gave has more than adequate RAM to do so.

Some notable issues:

innodb_flush_log_at_trx_commit=1 - change this to 2. While 1 is a great setting for missing critical applications where every write can't be lost, it slows write performance. With a setting of 2, it's much faster, and at most you could lose 1 second of data if power is lost.

innodb_buffer_pool_size=8M - far too small. This setting is one of the single most important settings for Innodb performance. It needs to be at least as large as the total largest anticipated size of all Innodb tables, plus about 15%.

innodb_buffer_pool_instances = 8 - this is completely useless except for when the above innodb_buffer_pool_size size is multi-gigabytes. General rule of thumb is 1 buffer pool instance per gigabyte of buffer pool size, up to about 32 instances (at that point go ahead and have each buffer pool larger than 1 GB). On high concurrency systems with plenty of CPU cores, it may be of benefit to use more instances than 32, but forums generally don't fit in this category except for very busy boards.

There are other issues in it as well, but I have a scheduled client task to get to, and those first two recommendations will go a long way towards speeding things up.
 
Even slow consumer-level hard drives have more than adequate transfer rates for the amount of data being transferred during installs. SSD or HDD, it's not going to make a difference with a rebuild if mysql is optimal, and the example hardware you gave has more than adequate RAM to do so.

Exactly what we were thinking, which is precisely why we've been quite baffled with the abnormally long installation times we've been getting.

Some notable issues:

innodb_flush_log_at_trx_commit=1 - change this to 2. While 1 is a great setting for missing critical applications where every write can't be lost, it slows write performance. With a setting of 2, it's much faster, and at most you could lose 1 second of data if power is lost.

innodb_buffer_pool_size=8M - far too small. This setting is one of the single most important settings for Innodb performance. It needs to be at least as large as the total largest anticipated size of all Innodb tables, plus about 15%.

innodb_buffer_pool_instances = 8 - this is completely useless except for when the above innodb_buffer_pool_size size is multi-gigabytes. General rule of thumb is 1 buffer pool instance per gigabyte of buffer pool size, up to about 32 instances (at that point go ahead and have each buffer pool larger than 1 GB). On high concurrency systems with plenty of CPU cores, it may be of benefit to use more instances than 32, but forums generally don't fit in this category except for very busy boards.

There are other issues in it as well, but I have a scheduled client task to get to, and those first two recommendations will go a long way towards speeding things up.

Cheers for the suggestions. We'll give these adjustments a shot to see what impact they have on our installation times. :)
 
Top