Server load excessively high (three figures!!) after migration

[CTXMedia]

I can restart my server and run CycleChat XF as a closed board and it runs just fine. No problems.

As soon as I open it and my members begin using it the server load starts rising and rising and it takes longer and longer for pages/posts/actions to complete, to the point it becomes useless.

Any ideas about what I might have missed, or misconfigured?

Thanks,
Shaun

 

[AzzidReign]

Since I'm in the process of trying to convert things over, I'm interested in this answer. But can I ask you how many concurrent users you have on and what time frame you use? And what are your server specs? Do you manage your own server or do you outsource it to a sysadmin group?

 

[Paul B]

Did you re-enable the various indexes/keys you disabled when doing the import and search rebuild?

 

[CTXMedia]

Own server - Linux 2.6.26-2-686-bigmem on i686 - AMD Athlon(tm) 64 X2 Dual Core Processor 4400+, 2 cores - 8GB RAM.

Works fine if I disable board and just me and a handful of mods are on it, but once I open it up it just goes downhill.

 

[CTXMedia]

Did you re-enable the various indexes/keys you disabled when doing the import and search rebuild?

Thanks, but yes, AFAIK. Only disabled them on the post and search tables and have enabled on both since.

 

[optrex]

caching configure correctly?

 

[CTXMedia]

caching configure correctly?

Yup, AFAIK.

 

[Mike]

Have you tuned InnoDB settings? If you haven't touched these, then you're going to have some serious problems: http://www.mysqlperformanceblog.com/2007/11/01/innodb-performance-optimization-basics/

Particularly innodb_buffer_pool_size and innodb_flush_log_at_trx_commit.

 

[CTXMedia]

my.cnf:

[client]
port             = 3306
socket           = /var/run/mysqld/mysqld.sock

[safe_mysqld]
err-log          = /var/log/mysql/mysql.err

[mysqld]
user             = mysql
pid-file         = /var/run/mysqld/mysqld.pid
socket           = /var/run/mysqld/mysqld.sock
port             = 3306
max_connections  = 500
basedir          = /usr
datadir          = /var/lib/mysql
tmpdir           = /tmp
language         = /usr/share/mysql/english
log-slow-queries = /var/log/mysql-slow.log

skip-locking

key_buffer              = 512M
myisam_sort_buffer_size = 32M
max_allowed_packet      = 32M
table_cache             = 3000
join_buffer_size        = 2M
sort_buffer_size        = 4M
read_buffer_size        = 2M
read_rnd_buffer_size    = 2M
thread_cache_size       = 64
query_cache_limit       = 16M
query_cache_size        = 256M
query_cache_type        = 1
tmp_table_size          = 128M
max_heap_table_size     = 128M
long_query_time         = 2

wait_timeout = 360
interactive_timeout = 360
connect_timeout = 10
net_read_timeout = 30
net_write_timeout = 60
concurrent_insert = 2

innodb_log_buffer_size = 8M
innodb_log_file_size = 16M
innodb_flush_log_at_trx_commit = 2
innodb_flush_method = O_DIRECT
innodb_thread_concurrency = 0
innodb_additional_mem_pool_size = 8M
innodb_buffer_pool_size = 1024M

ft_min_word_len = 3


[mysqldump]
quick
max_allowed_packet = 16M

[mysql]

[isamchk]
key_buffer = 64M
sort_buffer = 64M
read_buffer = 16M
write_buffer = 16M
ft_min_word_len = 3

[myisamchk]
key_buffer = 64M
sort_buffer = 64M
read_buffer = 16M
write_buffer = 16M
ft_min_word_len = 3

 

[Mike]

APC installed?

 

[CTXMedia]

I've increased a couple of the values (to the max my server will accept without balking at mysql start).

 

[CTXMedia]

APC installed?

Memcached + eAccelerator

 

[Mike]

It's loading quickly now (and I can actually see some forums as a guest ). I can try to login to the server at some point and see if I can find anything obvious, though if it's fine now, then I likely won't see anything.

 

[Kevin]

Seem to be loading fine for me as well now on our corporate WAN. Earlier the sidebar was loading but absolutely nothing in the main content area.

 

[CTXMedia]

Could it be a simple matter of the volume of logins (triggering account upgrades, trophies, etc.) combined with no entries in the caches, creating a bottleneck that will simply be cleared with use?

 

[lazer]

Looks like it's coming together Shaun Congrats on your move!

Site is loading pretty quick too...

 

[CTXMedia]

It's loading quickly now (and I can actually see some forums as a guest ). I can try to login to the server at some point and see if I can find anything obvious, though if it's fine now, then I likely won't see anything.

Seem to be loading fine for me as well now on our corporate WAN. Earlier the sidebar was loading but absolutely nothing in the main content area.

... seems to have settled itself down - but then I suspect a lot of the regulars have given up for the evening so we'll see how it goes under a heavier load tomorrow.

I still haven't managed to get all the privs setup for the various usergroups yet - guests included ...

I'm pretty brain-fried at the moment as I haven't had much sleep in the last few days, so some of it may have to wait until tomorrow; and my laptop needs a break it's been on solidly since Saturday night ...

 

[CTXMedia]

I wonder if the innodb settings changes have made a difference? Seems to be at a good low server load figure now and working nice and fast.

<crosses fingers!!>

 

[Kevin]

I wonder if the innodb settings changes have made a difference? Seems to be at a good low server load figure now and working nice and fast.

<crosses fingers!!>

Though whatever you did, don't change it! Both sites are coming up fast for me now.

 

[AzzidReign]

I wonder if the innodb settings changes have made a difference? Seems to be at a good low server load figure now and working nice and fast.

<crosses fingers!!>

GOOD STUFF! I wish you knew exactly what was causing your issues in the event I ran into them with my user load (roughly 8-15k people online within a 20 min period...I'm scurred).