I've been running a fairly high-traffic cluster for 10+ years at this point, so I'll share what my setup looks like in case it helps you (or anyone else). Over the years I've found that for some things, less is more (less things that can go wrong). At one point I had redundant load balancers that would route traffic to web servers and I also had database setup that involved a master with a bunch of slaves. Getting away from this was the single best thing I did... just too many issues where under high load transaction locking would cause replication lag and it just didn't scale well enough for SQL writes (things like Galera Cluster and InnoDB Cluster were a little better, but you are still bottlenecked by using a disk-based database system). Anyway, this is what I run currently:
- MySQL Cluster with 8 data nodes (ndbcluster storage engine). I can't stress how good ndbcluster is. You can scale into billions of SQL reads per second and 10s of millions of SQL writes per second. You end up having zero down time for users (even through server reboots, software updates, backups are non-locking). Good video here:
- All 8 servers have Nginx running, but Nginx is so efficient, there's only 1 live and the other 7 are just on standby. There's a cron job that checks the status of web server every 60 seconds and if it's not available (for any reason) it makes an API call to Cloudflare to just route traffic through the next available web server. Basically using Cloudflare as the load balancer and cutting out latency load balancers add. A 60 second max failover isn't a big deal because in the last 10 years, there's been exactly 0 times where an unplanned failover needed to happen. The systemd management of the nginx daemon automatically switches the Cloudflare DNS entry for the live web server before the process is stopped/restarted (so no need to remember to switch the DNS if you are doing maintenance). This is done with the ExecStop entry in the nginx systemd config.
- I have 2 servers running memcached. I've thought about switching to Redis for some of the more advanced things it does (like the ability to not lose the cache on a server reboot), but memcached works so well and has been so stable, I haven't really gone down that road. If I was starting fresh, it's something I'd look at.
- All 8 servers form an Elasticsearch cluster (2 copies of data across all the shards)
- The 8 servers formed a GlusterFS partition with lots of redundancy (4 copies of everything spread across 8 servers). This is NOT used for static files (PHP files, templates, etc.) The overhead of using a network file system for files that rarely change isn't worth it. Specifically, gluster is used for the sub-directories within internal_data and data (but NOT internal_data/temp or internal_data/code_cache).
Code:
twin1:/home/sites/rlqry.com/web # ls -al *data
data:
total 12
drwxrwxrwx 2 root root 4096 Jul 15 2019 .
drwxr-xr-x 10 root root 4096 Oct 14 2019 ..
lrwxrwxrwx 1 root root 42 Jul 15 2019 attachments -> /gluster/sites/rlqry.com/data/attachments/
lrwxrwxrwx 1 root root 38 Jul 15 2019 avatars -> /gluster/sites/rlqry.com/data/avatars/
lrwxrwxrwx 1 root root 48 Jul 15 2019 imported_reactions -> /gluster/sites/rlqry.com/data/imported_reactions
-rwxrwxrwx 1 root root 1 Nov 22 2017 index.html
lrwxrwxrwx 1 root root 35 Jul 15 2019 video -> /gluster/sites/rlqry.com/data/video
internal_data:
total 28
drwxrwxrwx 4 root root 4096 Jul 20 2019 .
drwxr-xr-x 10 root root 4096 Oct 14 2019 ..
lrwxrwxrwx 1 root root 50 Jul 15 2019 addon_batch -> /gluster/sites/rlqry.com/internal_data/addon_batch
lrwxrwxrwx 1 root root 51 Jul 15 2019 attachments -> /gluster/sites/rlqry.com/internal_data/attachments/
drwxrwxrwx 5 root root 4096 Jun 16 10:57 code_cache
lrwxrwxrwx 1 root root 50 Jul 15 2019 file_check -> /gluster/sites/rlqry.com/internal_data/file_check/
-rwxrwxrwx 1 root root 31 Nov 22 2017 .htaccess
lrwxrwxrwx 1 root root 50 Jul 15 2019 image_cache -> /gluster/sites/rlqry.com/internal_data/image_cache
lrwxrwxrwx 1 root root 51 Jul 15 2019 imported_xml -> /gluster/sites/rlqry.com/internal_data/imported_xml
-rwxrwxrwx 1 root root 1 Nov 22 2017 index.html
-rwxrwxrwx 1 wwwrun www 86 Jul 20 2019 install-lock.php
lrwxrwxrwx 1 root root 51 Jul 15 2019 oembed_cache -> /gluster/sites/rlqry.com/internal_data/oembed_cache
lrwxrwxrwx 1 root root 48 Jul 15 2019 sitemaps -> /gluster/sites/rlqry.com/internal_data/sitemaps/
drwxrwxrwx 2 wwwrun www 4096 Aug 7 15:57 temp
There is a custom addon that triggers a
csync2 -x command when things like templates are written to the file system. Again, csync2 is used to keep stuff in sync that rarely change (and typically only by an administrator action). In case you missed it before, running live PHP files on a networked file system isn't a good idea because (normally) PHP is constantly checking if the files have changed (better to keep them local on each server).
Hardware-wise, all servers are running RAID-6 with 6 hard drives (any two drives can fail without downtime, usable space is the size of 4 hard drives), have 1TB of RAM and are interconnected with 54Gbit Infiniband.
