Cpu's lock up at 100% when 200+ hit site.

[andrewkm]

Alright quick introduction; I run a forum for a gaming server.
Our board statistics are not too big:

Forum Statistics

Discussions:
35,385
Messages:
162,477

There are times when we have 200-250 users hit the forums simultaneously (literally at the same time) - either when we run a competition with information on the website; or when our game server goes down for specific updates. All players insta hit the site. At this point I monitor our usage and cpus lock up at 100% and site just doesn't load.

Can anyone give me any advice on what would be best to update to prevent this from happening.
When run on a VPS with 2 vcpus and 4gb of ram. Just 2 days ago i bumped it to 3 vcpus and nothing... same lock-up affect... Do I need 4? Do I need to perhaps look into a different setting ? Is something configured wrong?

Someone shed some light

PS: We are running on a 100mbit line.

 

[Digital Doctor]

100% cpu

what addons are you using ?

 

[andrewkm]

what addons are you using ?

Installed Add-ons

Add Username Style to Last Post 1.0.1
Awaiting Email Users listing in AdminCP 0.9.1
Bookmarks 1.1.2
Conversation Essentials 1.1.4
Custom BBCode Manager 1.3.4
Easy ban users 1.0.1
KeyCAPTCHA 2.1
Nodes As Tabs 1.1.3
Post Ratings 1.4.1
TaigaChat 0.5.5
Template Modification System 1.2
Thread Ban 1.0.2
User Essentials 1.1.1
Xenplaza - Number Viewing a Forum 1.0.0
[8wayRun.Com] XenPorta (Portal) 1.5.5
[8wayRun.Com] XenUtiles (Tools) 1.2.2
[bd] Medal 1.4
[bd] Tag Me 1.6.5
[******] Custom Node Icon 0.9

 

[Digital Doctor]

None of those are obvious culprits.
It's probably not an addon issue.
[8wayRun.Com] XenPorta (Portal) 1.5.5 - might slow things down with higher traffic.
[******] Custom Node Icon - 1.0 is available.
http://xenforo.com/community/resources/custom-node-icon-icon-for-category.632/

Any errors logged ?
Social networking (Facebook especially) can slow things down.
Do you have the server logs from a time when problems were happening ?

 

[Digital Doctor]

what caching programs are you using ? APC ?
what php version ?
what are you server settings ?
does disk I/O peak when your CPU goes to 100% ?

 

[Luke F]

Is your mysql server optimised?
Have you considered moving to nginx+php-fpm if you haven't already?
Where is the cpu usage primarily coming from (php/webserver or mysqld)?

 

[andrewkm]

Hm - I'll get my devs to look into all this shortly and reply to all those questions. Need to wait for another gaming peak time to have 200+ hit the site at the same time again and I'll take more informed reports when its happening.

We're running on
PHP: 5.4.7
Mysql: 5.5.28
Running nginx

 

[andrewkm]

Mysql settings:

[mysqld]
performance_schema

skip-external-locking
sysdate-is-now

log-error = /var/log/mysql.log

default-storage-engine = InnoDB

innodb_file_per_table = 1
innodb_file_format = barracuda

innodb_buffer_pool_size = 512M
innodb_additional_mem_pool_size = 10M
innodb_log_buffer_size = 8M
innodb_flush_log_at_trx_commit = 1
innodb_lock_wait_timeout = 50
innodb_fast_shutdown = 0
innodb_flush_method = O_DSYNC
innodb_buffer_pool_instances = 4
innodb_max_dirty_pages_pct = 50
innodb_read_io_threads = 8
innodb_write_io_threads = 4
transaction_isolation = READ-COMMITTED

max_connections = 100
connect_timeout = 86400
wait_timeout = 86400
interactive_timeout = 86400
max_allowed_packet = 64M

open_files_limit = 2048
table_cache = 2048
net_buffer_length = 8K
query_cache_type = 1
query_cache_limit = 1M
query_cache_size = 16M
thread_cache = 100
thread_stack = 512K
tmp_table_size = 64M

key_buffer_size = 128M
sort_buffer_size = 512K
net_buffer_length = 8K
read_buffer_size = 256K
read_rnd_buffer_size = 512K
myisam_sort_buffer_size = 8M

[mysqldump]
quick
max_allowed_packet = 16M

[mysql]
no-auto-rehash

[myisamchk]
key_buffer_size = 20M
sort_buffer_size = 20M
read_buffer = 2M
write_buffer = 2M

 

[Slavik]

There are times when we have 200-250 users hit the forums simultaneously (literally at the same time) - either when we run a competition with information on the website; or when our game server goes down for specific updates. All players insta hit the site. At this point I monitor our usage and cpus lock up at 100% and site just doesn't load.

You say 100% cpu, whats using it? httpd, mysql etc?

When your under heavy load, have your admins SSH in and run the top command to find out.