Server has high load

[Marks534]

Hi,

I have quite a beefy machine, did some simple MySQL optimizations, tried caching, etc yet my machine has a high CPU usage and load. I have 700+- members at peaks. 25k registered and 880k posts. I do have elasticsearch enabled.

Spoiler: Server Specifications

Xeon E5-1630 v3 @ 3.70GHz
RAM
4x 16384MB
Disks
2 x 480 GB SSD

So yeah, quite a machine yet I'm getting slow loading times and high CPU usage. I use MaxCDN which does a great job so far. See blow all my configuration files involved.

Spoiler: my.cnf

[mysqld]
skip-external-locking
query_cache_type = 0
query_cache_size = 0
query_cache_limit = 1M
query_cache_strip_comments = 0
thread_cache_size = 256
max_heap_table_size = 128M
tmp_table_size = 128M
connect_timeout = 10
max_connections = 500
interactive_timeout = 300
wait_timeout = 300
thread_stack = 256k
table_open_cache = 24000
open_files_limit = 24000
key_buffer_size = 768M
max_allowed_packet = 4M
sort_buffer_size = 1M
join_buffer_size = 512k
#net_buffer_length = 64K
read_buffer_size = 512K
#read_rnd_buffer_size = 512K
myisam_sort_buffer_size = 128M
innodb_buffer_pool_size = 8G
innodb_thread_concurrency = 7
innodb_buffer_pool_instances = 4
innodb_flush_method=O_DIRECT
innodb_file_per_table=1
innodb_log_buffer_size=8M

This is a configuration I've assembled from several threads on here and tweaked it to my server specifications.

Spoiler: php-fpm configuration

[site]
listen = 127.0.0.1:9001
listen.allowed_clients = 127.0.0.1

user = admin
group = admin

pm = dynamic
pm.max_children = 2000
pm.start_servers = 50
pm.min_spare_servers = 50
pm.max_spare_servers = 90

env[HOSTNAME] = $HOSTNAME
env[PATH] = /usr/local/bin:/usr/bin:/bin
env[TMP] = /tmp
env[TMPDIR] = /tmp
env[TEMP] = /tmp

Let me know if you guys need any other configuration files.

The setup is a VestaCP installation. As stated above, i've tried caching systems and none really improved load significantly so I removed them temporarily until someone can help me to get it configured and setup properly.

htop

 

[rdn]

Try this for PHP-FPM:

pm = dynamic
pm.max_children = 32
pm.start_servers = 8
pm.min_spare_servers = 4
pm.max_spare_servers = 16
pm.max_requests = 1000

With similar PHP-FPM config, on one large forum I have.
Online stats last 15 minutes: Total: 4,477 (members: 1,162, guests: 3,242)
Average system load: 2% / 0.17 0.28 0.27

For Mysql:

innodb_io_capacity = 3000
innodb_read_io_threads = 64
innodb_write_io_threads = 64
innodb_flush_neighbors = 0

For server, I highly recommend PHP 7.0 w/ Zend Opcache
Memcached or Redis for Data Caching that should help a lot!

For Nginx and Guest Caching: https://xenforo.com/community/threads/110806/

I have 700+- members at peaks. 25k registered and 880k posts. I do have elasticsearch enabled.

This is really small board and can be hosted on Linode 4GB plan.
I have mine large a bit than this just hosted on Linode 4GB/$40 plan only and works very fine.
Current Average system load: 10% 0.40 0.38 0.41 (Just a bit high than normal usage as I just upgraded XF and restarted PHP-FPM and all Cache)

 

[Marks534]

Try this for PHP-FPM:

pm = dynamic
pm.max_children = 32
pm.start_servers = 8
pm.min_spare_servers = 4
pm.max_spare_servers = 16
pm.max_requests = 1000

With similar PHP-FPM config, on one large forum I have.
Online stats last 15 minutes: Total: 4,477 (members: 1,162, guests: 3,242)
Average system load: 2% / 0.17 0.28 0.27

For Mysql:

innodb_io_capacity = 3000
innodb_read_io_threads = 64
innodb_write_io_threads = 64
innodb_flush_neighbors = 0

For server, I highly recommend PHP 7.0 w/ Zend Opcache
Memcached or Redis for Data Caching that should help a lot!

For Nginx and Guest Caching: https://xenforo.com/community/threads/110806/



This is really small board and can be hosted on Linode 4GB plan.
I have mine large a bit than this just hosted on Linode 4GB/$40 plan only and works very fine.
Current Average system load: 10% 0.40 0.38 0.41 (Just a bit high than normal usage as I just upgraded XF and restarted PHP-FPM and all Cache)

Thanks for your help. I've applied the php-fpm config options and I can notice a huge improvement on performance. As for the MySQL config, can you check what I currently have is okay? Mainly the innodb settings. For some reason, when I applied your settings mariadb refused to restart.

 

[rdn]

MariaDB version your using?
My current config is use on 10.1.11.

 

[Marks534]

MariaDB version your using?
My current config is use on 10.1.11.

5.5.44-MariaDB

 

[rdn]

Use this instead:

innodb_io_capacity = 3000
innodb_read_io_threads = 64
innodb_write_io_threads = 64

 

[Mike Edge]

Ditch a lot of that RAM, upgrade to a better processor (hex core) and add 2 more SSD's to move to RAID10 and you will see a big improvement. 64GB RAM is overkill for a server with 700 online. You should only need 8GB RAM at very most. Upgrading processor to a hex core, will give you more CPU power then a E5 by a long shot.

 

[Marks534]

Use this instead:

innodb_io_capacity = 3000
innodb_read_io_threads = 64
innodb_write_io_threads = 64

Applied the fixes but don't see any direct improvements via htop.

Ditch a lot of that RAM, upgrade to a better processor (hex core) and add 2 more SSD's to move to RAID10 and you will see a big improvement. 64GB RAM is overkill for a server with 700 online. You should only need 8GB RAM at very most. Upgrading processor to a hex core, will give you more CPU power then a E5 by a long shot.

I don't think my server is the cause of bad performance, it's probably just configuration.

 

[Marks534]

@RoldanLT Still high CPU usage

 

[Mike Edge]

What database is causing the load? How large is it? Did you run mysqlchk on it to assure no crashed tables?

 

[rdn]

I highly recommend PHP 7.0 w/ Zend Opcache
Memcached or Redis for Data Caching that should help a lot!

Your PHP version? Opcode caching use?

 

[Marks534]

Your PHP version? Opcode caching use?

PHP 5.6.17 with OpCache 7.0.6-dev

 

[Marks534]

@RoldanLT It really seems like a MySQL issue. I sometimes get all my MySQL instances hitting +200% CPU

 

[rdn]

Try to upgrade your MariaDB to 10.1.11 first.

 

[Joeychgo]

Try to upgrade your MariaDB to 10.1.11 first.

That much better than 5.5.48-MariaDB?

 

[Marks534]

That much better than 5.5.48-MariaDB?

I honestly think it would improve performance a little however I don't think that's causing my CPU usage to hit +200%.

 

[Joeychgo]

I honestly think it would improve performance a little however I don't think that's causing my CPU usage to hit +200%.

I was asking because I'm running 5.5.48-MariaDB and things seem a little slow. Don't want to hijack your thread though.

 

[TPerry]

DDOS issue?

netstat -ntu | grep :80 | grep -v LISTEN | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn | grep -v 127.0.0.1

and see how many active connections from IP's you have... I've seen similar happen on a system that was having a layer 7 DDOS attack.
If running SSL, change :80 to :443.

 

[rdn]

That much better than 5.5.48-MariaDB?

Based on experience, Yes.

 

[Marks534]

DDOS issue?

netstat -ntu | grep :80 | grep -v LISTEN | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn | grep -v 127.0.0.1

and see how many active connections from IP's you have... I've seen similar happen on a system that was having a layer 7 DDOS attack.
If running SSL, change :80 to :443.

That was also my first thought however I think it's very unlikey to get hit with Layer 7 attacks 24/7. I have a normal amount of connections.

I'm seriously out of ideas.