XF 1.5 Larger Forums and Stability

[Louise Williams]

Apologies if this is in the wrong area.

Hoping someone can help clarify something for me - we are on xenforo 1.5 we have just under 1.5mil posts and 116k threads (and 28k members) DB is nearly 4GB

My co-owner seems sure he has read in numerous places that this is too big to be stable. Our server is plenty big enough with 64GB RAM and 1TB storage etc - but the site does keep freezing/slowing down intermittently (as well as a couple of outright crashes).

Is this likely to be to do with being "too big for the software"? My co-owner thinks the answer is to delete the older half of the threads - and if it is the answer then okay - but I did see that there are way way bigger forums than ours running XF so I just thought I would find out if he has read something erroneous somewhere.

Any feedback would be appreciated. Thanks!

 

[Paul B]

There are many forums with many more posts than that - some have tens of millions: https://xenforo.com/community/threads/the-large-xenforo-forums-list.87974/
You shouldn't be suffering from server stability issues.

It's possible that there are other processes running on the server which are causing the issues, or it could be the server needs tuning and set up properly specifically for MyISAM and InnoDB.

Here are a couple of links which may help:
http://www.percona.com/blog/2013/09/20/innodb-performance-optimization-basics-updated/
https://tools.percona.com/wizard

It's going to mean monitoring the server to see which process are running at the time of the slowdowns/crashes and which, if any, are running out of resources.

The server error log may also hold some clues.

 

[Louise Williams]

Thanks that is really helpful. Our server has no cache modules installed (apparently) my developer asked about caching this morning so I am trying to understand caching - do you recommend having a caching module?

We tried to delete about 3k posts the other day and the site froze and threw up an error about "not enough MySQL connections" - apparently these have now been increased but I have no idea what that really means. The hosting company said that the freeze and crash was due to the spike in PHP processes from running the batch delete - but it didn't seem like that should seriously be enough to crash the site when we are running on a 64GB RAM dedicated server? Not that I really know about this stuff.

I will have a read of those links thank you

 

[Paul B]

Caching is generally always useful.

A lack of MySQL connections is a good example of the server not being tuned specifically for the installation.

 

[Louise Williams]

Thanks.

Sorry just one more question - should I get elastisearch? We rebuilt all the caches yesterday (took hours and the site was useless the entire time we were doing it - over a minute to load a page) so all the posts are finally indexed in the search I believe.

 

[Paul B]

In general, the cut off for MySQL search is deemed to be around 1 million posts.

Forums with more posts than that tend to use the ES add-on.

 

[Louise Williams]

Lovely thanks

 

[m1ne]

Try PHP7 with Opcache and MariaDB 10. There's a bunch of optimization guides out there for MySQL.
Consider Cloudflare too.

 

[Mike Edge]

>> 1TB storage

I'm guessing based on 1TB, your using SATA drives. A database your size would offer far better performance on SSD instead. The slowness during your peak time could very likely be I/O bottle necking MySQL processes..

 

[rdn]

Changing your Server Disk to SSD will change everything .
250gb+ SSD in raid 1 is already sufficient and will improve your site performance a lot.

Consider using PHP7 also with Zend Opcache.
Memcached or Redis for data caching.

 

[ForestForTrees]

If you can afford it, getting a XF consultant with server expertise might help. As someone who has never hired a consultant and has toughed it out myself, I can say that hiring a consultant sounds like a good option. RolandLT's suggestions seemed good and a consultant would help. There are some consultants who have a good reputation on this forum, and that could help you in selecting one, should you choose to go that way.

With that said, I'm self-taught and managed a transition to a multithreaded Apache MPM, Opcache and memcached with some help from our (high quality) ISP (LiquidWeb). It took some work, but now our VPS flies. I suspect that with a well tuned server and good caching, XF can laugh at anything you throw at it. That's certainly how I feel about my server. See the link that Brogan provided in his first post for details on scaling.

 

[rdn]

DB is nearly 4GB

The two large DB table consuming much from this are:
xf_bb_code_parse_cache (can be prune & disabled)
xf_search_index (can be prune when you use ES)

 

[Mouth]

we have just under 1.5mil posts and 116k threads (and 28k members) DB is nearly 4GB. My co-owner seems sure he has read in numerous places that this is too big to be stable.

Yup, as pointed out through posts above, it's not XF and your content that's too big. It's your server and services not being configured correctly. If you don't have the expertise to do this, get someone to do it for you.