Server has high load

Marks534

Member
Hi,

I have quite a beefy machine, did some simple MySQL optimizations, tried caching, etc yet my machine has a high CPU usage and load. I have 700+- members at peaks. 25k registered and 880k posts. I do have elasticsearch enabled.

Xeon E5-1630 v3 @ 3.70GHz
RAM
4x 16384MB
Disks
2 x 480 GB SSD

So yeah, quite a machine yet I'm getting slow loading times and high CPU usage. I use MaxCDN which does a great job so far. See blow all my configuration files involved.

Code:
[mysqld]
skip-external-locking
query_cache_type = 0
query_cache_size = 0
query_cache_limit = 1M
query_cache_strip_comments = 0
thread_cache_size = 256
max_heap_table_size = 128M
tmp_table_size = 128M
connect_timeout = 10
max_connections = 500
interactive_timeout = 300
wait_timeout = 300
thread_stack = 256k
table_open_cache = 24000
open_files_limit = 24000
key_buffer_size = 768M
max_allowed_packet = 4M
sort_buffer_size = 1M
join_buffer_size = 512k
#net_buffer_length = 64K
read_buffer_size = 512K
#read_rnd_buffer_size = 512K
myisam_sort_buffer_size = 128M
innodb_buffer_pool_size = 8G
innodb_thread_concurrency = 7
innodb_buffer_pool_instances = 4
innodb_flush_method=O_DIRECT
innodb_file_per_table=1
innodb_log_buffer_size=8M

This is a configuration I've assembled from several threads on here and tweaked it to my server specifications.

Code:
[site]
listen = 127.0.0.1:9001
listen.allowed_clients = 127.0.0.1

user = admin
group = admin

pm = dynamic
pm.max_children = 2000
pm.start_servers = 50
pm.min_spare_servers = 50
pm.max_spare_servers = 90

env[HOSTNAME] = $HOSTNAME
env[PATH] = /usr/local/bin:/usr/bin:/bin
env[TMP] = /tmp
env[TMPDIR] = /tmp
env[TEMP] = /tmp


Let me know if you guys need any other configuration files.

The setup is a VestaCP installation. As stated above, i've tried caching systems and none really improved load significantly so I removed them temporarily until someone can help me to get it configured and setup properly.

htop

qCVQCOo.png
 
Try this for PHP-FPM:
Code:
pm = dynamic
pm.max_children = 32
pm.start_servers = 8
pm.min_spare_servers = 4
pm.max_spare_servers = 16
pm.max_requests = 1000

With similar PHP-FPM config, on one large forum I have.
Online stats last 15 minutes: Total: 4,477 (members: 1,162, guests: 3,242)
Average system load: 2% / 0.17 0.28 0.27

For Mysql:
Code:
innodb_io_capacity = 3000
innodb_read_io_threads = 64
innodb_write_io_threads = 64
innodb_flush_neighbors = 0

For server, I highly recommend PHP 7.0 w/ Zend Opcache
Memcached or Redis for Data Caching that should help a lot!

For Nginx and Guest Caching: https://xenforo.com/community/threads/110806/


I have 700+- members at peaks. 25k registered and 880k posts. I do have elasticsearch enabled.
This is really small board and can be hosted on Linode 4GB plan.
I have mine large a bit than this just hosted on Linode 4GB/$40 plan only and works very fine.
Current Average system load: 10% 0.40 0.38 0.41 (Just a bit high than normal usage as I just upgraded XF and restarted PHP-FPM and all Cache)
 
Last edited:
Try this for PHP-FPM:
Code:
pm = dynamic
pm.max_children = 32
pm.start_servers = 8
pm.min_spare_servers = 4
pm.max_spare_servers = 16
pm.max_requests = 1000

With similar PHP-FPM config, on one large forum I have.
Online stats last 15 minutes: Total: 4,477 (members: 1,162, guests: 3,242)
Average system load: 2% / 0.17 0.28 0.27

For Mysql:
Code:
innodb_io_capacity = 3000
innodb_read_io_threads = 64
innodb_write_io_threads = 64
innodb_flush_neighbors = 0

For server, I highly recommend PHP 7.0 w/ Zend Opcache
Memcached or Redis for Data Caching that should help a lot!

For Nginx and Guest Caching: https://xenforo.com/community/threads/110806/



This is really small board and can be hosted on Linode 4GB plan.
I have mine large a bit than this just hosted on Linode 4GB/$40 plan only and works very fine.
Current Average system load: 10% 0.40 0.38 0.41 (Just a bit high than normal usage as I just upgraded XF and restarted PHP-FPM and all Cache)


Thanks for your help. I've applied the php-fpm config options and I can notice a huge improvement on performance. As for the MySQL config, can you check what I currently have is okay? Mainly the innodb settings. For some reason, when I applied your settings mariadb refused to restart.
 
Use this instead:
Code:
innodb_io_capacity = 3000
innodb_read_io_threads = 64
innodb_write_io_threads = 64
 
Ditch a lot of that RAM, upgrade to a better processor (hex core) and add 2 more SSD's to move to RAID10 and you will see a big improvement. 64GB RAM is overkill for a server with 700 online. You should only need 8GB RAM at very most. Upgrading processor to a hex core, will give you more CPU power then a E5 by a long shot.
 
Use this instead:
Code:
innodb_io_capacity = 3000
innodb_read_io_threads = 64
innodb_write_io_threads = 64

Applied the fixes but don't see any direct improvements via htop.

Ditch a lot of that RAM, upgrade to a better processor (hex core) and add 2 more SSD's to move to RAID10 and you will see a big improvement. 64GB RAM is overkill for a server with 700 online. You should only need 8GB RAM at very most. Upgrading processor to a hex core, will give you more CPU power then a E5 by a long shot.

I don't think my server is the cause of bad performance, it's probably just configuration.
 
What database is causing the load? How large is it? Did you run mysqlchk on it to assure no crashed tables?
 
DDOS issue?
Code:
netstat -ntu | grep :80 | grep -v LISTEN | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn | grep -v 127.0.0.1
and see how many active connections from IP's you have... I've seen similar happen on a system that was having a layer 7 DDOS attack.
If running SSL, change :80 to :443.
 
DDOS issue?
Code:
netstat -ntu | grep :80 | grep -v LISTEN | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn | grep -v 127.0.0.1
and see how many active connections from IP's you have... I've seen similar happen on a system that was having a layer 7 DDOS attack.
If running SSL, change :80 to :443.
That was also my first thought however I think it's very unlikey to get hit with Layer 7 attacks 24/7. I have a normal amount of connections.

I'm seriously out of ideas.
 
Top Bottom