• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

Server has high load

#1
Hi,

I have quite a beefy machine, did some simple MySQL optimizations, tried caching, etc yet my machine has a high CPU usage and load. I have 700+- members at peaks. 25k registered and 880k posts. I do have elasticsearch enabled.

Xeon E5-1630 v3 @ 3.70GHz
RAM
4x 16384MB
Disks
2 x 480 GB SSD

So yeah, quite a machine yet I'm getting slow loading times and high CPU usage. I use MaxCDN which does a great job so far. See blow all my configuration files involved.

Code:
[mysqld]
skip-external-locking
query_cache_type = 0
query_cache_size = 0
query_cache_limit = 1M
query_cache_strip_comments = 0
thread_cache_size = 256
max_heap_table_size = 128M
tmp_table_size = 128M
connect_timeout = 10
max_connections = 500
interactive_timeout = 300
wait_timeout = 300
thread_stack = 256k
table_open_cache = 24000
open_files_limit = 24000
key_buffer_size = 768M
max_allowed_packet = 4M
sort_buffer_size = 1M
join_buffer_size = 512k
#net_buffer_length = 64K
read_buffer_size = 512K
#read_rnd_buffer_size = 512K
myisam_sort_buffer_size = 128M
innodb_buffer_pool_size = 8G
innodb_thread_concurrency = 7
innodb_buffer_pool_instances = 4
innodb_flush_method=O_DIRECT
innodb_file_per_table=1
innodb_log_buffer_size=8M

This is a configuration I've assembled from several threads on here and tweaked it to my server specifications.

Code:
[site]
listen = 127.0.0.1:9001
listen.allowed_clients = 127.0.0.1

user = admin
group = admin

pm = dynamic
pm.max_children = 2000
pm.start_servers = 50
pm.min_spare_servers = 50
pm.max_spare_servers = 90

env[HOSTNAME] = $HOSTNAME
env[PATH] = /usr/local/bin:/usr/bin:/bin
env[TMP] = /tmp
env[TMPDIR] = /tmp
env[TEMP] = /tmp


Let me know if you guys need any other configuration files.

The setup is a VestaCP installation. As stated above, i've tried caching systems and none really improved load significantly so I removed them temporarily until someone can help me to get it configured and setup properly.

htop

 

RoldanLT

Well-known member
#2
Try this for PHP-FPM:
Code:
pm = dynamic
pm.max_children = 32
pm.start_servers = 8
pm.min_spare_servers = 4
pm.max_spare_servers = 16
pm.max_requests = 1000
With similar PHP-FPM config, on one large forum I have.
Online stats last 15 minutes: Total: 4,477 (members: 1,162, guests: 3,242)
Average system load: 2% / 0.17 0.28 0.27

For Mysql:
Code:
innodb_io_capacity = 3000
innodb_read_io_threads = 64
innodb_write_io_threads = 64
innodb_flush_neighbors = 0
For server, I highly recommend PHP 7.0 w/ Zend Opcache
Memcached or Redis for Data Caching that should help a lot!

For Nginx and Guest Caching: https://xenforo.com/community/threads/110806/


I have 700+- members at peaks. 25k registered and 880k posts. I do have elasticsearch enabled.
This is really small board and can be hosted on Linode 4GB plan.
I have mine large a bit than this just hosted on Linode 4GB/$40 plan only and works very fine.
Current Average system load: 10% 0.40 0.38 0.41 (Just a bit high than normal usage as I just upgraded XF and restarted PHP-FPM and all Cache)
 
Last edited:
#3
Try this for PHP-FPM:
Code:
pm = dynamic
pm.max_children = 32
pm.start_servers = 8
pm.min_spare_servers = 4
pm.max_spare_servers = 16
pm.max_requests = 1000
With similar PHP-FPM config, on one large forum I have.
Online stats last 15 minutes: Total: 4,477 (members: 1,162, guests: 3,242)
Average system load: 2% / 0.17 0.28 0.27

For Mysql:
Code:
innodb_io_capacity = 3000
innodb_read_io_threads = 64
innodb_write_io_threads = 64
innodb_flush_neighbors = 0
For server, I highly recommend PHP 7.0 w/ Zend Opcache
Memcached or Redis for Data Caching that should help a lot!

For Nginx and Guest Caching: https://xenforo.com/community/threads/110806/



This is really small board and can be hosted on Linode 4GB plan.
I have mine large a bit than this just hosted on Linode 4GB/$40 plan only and works very fine.
Current Average system load: 10% 0.40 0.38 0.41 (Just a bit high than normal usage as I just upgraded XF and restarted PHP-FPM and all Cache)

Thanks for your help. I've applied the php-fpm config options and I can notice a huge improvement on performance. As for the MySQL config, can you check what I currently have is okay? Mainly the innodb settings. For some reason, when I applied your settings mariadb refused to restart.
 

Mike Edge

Well-known member
#7
Ditch a lot of that RAM, upgrade to a better processor (hex core) and add 2 more SSD's to move to RAID10 and you will see a big improvement. 64GB RAM is overkill for a server with 700 online. You should only need 8GB RAM at very most. Upgrading processor to a hex core, will give you more CPU power then a E5 by a long shot.
 
#8
Use this instead:
Code:
innodb_io_capacity = 3000
innodb_read_io_threads = 64
innodb_write_io_threads = 64
Applied the fixes but don't see any direct improvements via htop.

Ditch a lot of that RAM, upgrade to a better processor (hex core) and add 2 more SSD's to move to RAID10 and you will see a big improvement. 64GB RAM is overkill for a server with 700 online. You should only need 8GB RAM at very most. Upgrading processor to a hex core, will give you more CPU power then a E5 by a long shot.
I don't think my server is the cause of bad performance, it's probably just configuration.
 

Tracy Perry

Well-known member
#18
DDOS issue?
Code:
netstat -ntu | grep :80 | grep -v LISTEN | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn | grep -v 127.0.0.1
and see how many active connections from IP's you have... I've seen similar happen on a system that was having a layer 7 DDOS attack.
If running SSL, change :80 to :443.
 
#20
DDOS issue?
Code:
netstat -ntu | grep :80 | grep -v LISTEN | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn | grep -v 127.0.0.1
and see how many active connections from IP's you have... I've seen similar happen on a system that was having a layer 7 DDOS attack.
If running SSL, change :80 to :443.
That was also my first thought however I think it's very unlikey to get hit with Layer 7 attacks 24/7. I have a normal amount of connections.

I'm seriously out of ideas.