XF 1.5 - Attachment folder size, Database bottlenecks, and 503 errors.

Mr Shap

Member
Hey everyone. Looking for a bit of advice:

We are a larger board (1.3M+ member accounts, almost 20 years online). We are currently running Xenforo 1.5, and we are busy working on a migration to the current 2.5.x version.

One major problem we are contending with while this happens in the meantime however, is that our board is very content heavy, and we are sitting at 42,000+ folders of content in our attachments directory.

Obviously this is a major problem because it far exceeds the 10,000 max we should have. We are running into issues with our database choking during peak times due to the number of folders / objects in that folder tree, creating a bottleneck on the overall site with the IO listing lags dragging everything down.

In the meantime while we get to a version of XenForo that can be scaled, our site seems to be dying a slow and steady death, with more content being added each day, and this issue becoming worse as time goes by.

We are wondering if this has come up before with some big boards, and if there is any small fix that can help address this to get us through while we wait for Xen 2 and the path to be clear to start making more robust development moves to scale our site for the long run.

The situation is getting a little urgent, so any advice would be greatly appreciated. :)

Thanks
 
We currently have over 1.2m attachments, using ~1.5TB space, and ~1PB/m transfer.

Once you're on 2.x you can use an S3 compatible storage provider to offload your attachments from your main webserver - https://xenforo.com/community/resou...or-amazon-s3-for-file-storage-in-xf-2-x.6805/ (Wasabi + Cloudflare would be much cheaper than DigitalOcean or Amazon).

On 1.x we used this in the past: https://xenforo.com/community/resources/bd-attachment-store.1779/ but IMO it'd be best to wait until you upgrade to 2.x and use the officially supported method, it'll cause a lot less headaches when it comes to upgrading Xenforo.

What is your hosting setup? Have you looked at moving/upgrading your setup to give you more IO throughput? Are you using a CDN or Cloudflare?
 
Top Bottom