Solution for Xenforo forum with 100+ GB of images?

[phong.vt]

My forum will be a image-heavy one, with 100,000+ images will be uploaded. Does anyone have any experience working on a similar forum?

I'm using Amazon S3 to host all external data of XF and begin uploading all the images & content as well. But I'm afraid the cost of AWS will be too high when we open the forum to public.

Please share your experience. Thank you all!

 

[rdn]

DigitalOcean Spaces | S3-Compatible Object Storage

S3-compatible object storage with a built-in CDN that makes scaling easy, reliable, and affordable. $5/mo.

www.digitalocean.com

Using DigitalOcean Spaces or Amazon S3 for file storage in XF 2.1+

XenForo Version Compatibility The download is only compatible with XenForo 2.1 and above. Why this guide? Since XenForo 2.0.0 we have supported remote file storage using an abstracted file system named Flysystem. It's called an abstracted file...

xenforo.com

 

[Slavik]

My forum will be a image-heavy one, with 100,000+ images will be uploaded. Does anyone have any experience working on a similar forum?

I'm using Amazon S3 to host all external data of XF and begin uploading all the images & content as well. But I'm afraid the cost of AWS will be too high when we open the forum to public.

Please share your experience. Thank you all!

I have recently set up a forum for a customer similar to yours using digitalocean spaces, the cost is very reasonable.

 

[woody]

100,000....cute My forum has over 1.5 million...for reference, data transfer each month is about 6TB.

S3 is on my list once the 2.1 migration occurs.

 

[Sim]

We have over 360,000 images in our photo gallery and currently run on a rather large Linode to get the required disk space (currently consuming 145GB of space).

We will shortly be migrating to a new (smaller) server on Linode (moving it to a new region to be closer to most of our members) and implementing Linode Block Storage - which gives you dynamically resizeable volumes you can attach to your Linode - they appear just like a local drive to your server, so no special code or functionality is required to use this storage.

This means we can keep all images locally on our server and not worry about how much space they consume, nor having to set up automated tasks to move files to a remote storage server - it should be completely transparent to XenForo.

 

[shawn]

265,000+ here

Never even thought to look until this thread asked the question. We just run on a regular old VPS.

 

[delicatebobster]

745,000+ images here and my data transfer each month is 1tb

 

[Dakis]

I have 280,000 images and just use an OVH server that @Brent W setup for me, and it just works fine, no issues with the images at all.

I don't think you need to worry for 100k images, even if you use aws the cost will be extremely low.

 

[fingon]

I have 300GB attachments(images and zip&rars) with 3-4TB traffic per month
Do NOT store them in S3, your wallet will be empty very soon.

Get a dedicated server with raid for storage is a better/cheaper solution.

 

[x3sphere]

I use S3+Cloudflare. Costs have been reasonable for me even storing around 150GB of data, however all our images are public and I've configured CF page rules to use 'cache everything'. So most of the time, a visit to the site doesn't even hit S3 since the images are cached on CF.

This approach might not work if you require access checks for your images, etc. Cloudflare ends up saving us around 60-70% of bandwidth usage each month.

 

[duderuud]

We also have 1mln+ images on our forum. Using multiple VPS servers from TransIP.nl with a "Big Storage" add-on. Works great!

 

[dethfire]

Wow! I have around 400k images but limited image size and compressed them hard and they only take up 10GB!

 

[phong.vt]

Wow my site feels so tiny compared with all of you . Here's my situation and I have some questions. I do really appreciate if I can get advice from everyone. I ain't a tech guy, so it's very difficult to manage a dedicated server, I can only use easy-to-use services.

1. Now I'm using AWS S3 with the Using DigitalOcean Spaces or Amazon S3 for file storage in XF 2.x add on. I notice that all the images stored on S3 is set with ".data" extension, not ".jpg" - I store the whole data (image, thumbnail, pdf ..) there.

So I'm thinking in the future, if cost of using S3 is too high, is it possible to move to a private server with those ".data" files? Or at least move to DigitalOcean Spaces.

2. From my understanding, AWS S3 calculates the cost by storage space, bandwidth and number of put/get requests. And DO Spaces has only 2 factors: 5$/month for 250 GB of storage, 1 TB of outbound transfer, Unlimited uploads, Unlimited Spaces. So I can understand that DO Spaces is a little bit better option compared to S3. Now my bill at S3 from Jan 1, 2019 is 3$ with 7GB storage / 12GB bandwidth / 40k put request / 260k get request.

Anyone can please share your experience with DO Spaces? Should I move to DO Spaces? And is it possible to transfer all current data from S3 to Spaces?

3. Is anyone using any solution for optimizing images on the server, with a reasonable costs? Is it a Xenforo Addon or you use third-party solution?

Thank you very much everyone to share your experience. Wish I can understanding everything you guys are sharing. Again, I'm not a tech guy

 

[cosmin85]

Wow my site feels so tiny compared with all of you . Here's my situation and I have some questions. I do really appreciate if I can get advice from everyone. I ain't a tech guy, so it's very difficult to manage a dedicated server, I can only use easy-to-use services.

1. Now I'm using AWS S3 with the Using DigitalOcean Spaces or Amazon S3 for file storage in XF 2.x add on. I notice that all the images stored on S3 is set with ".data" extension, not ".jpg" - I store the whole data (image, thumbnail, pdf ..) there.

So I'm thinking in the future, if cost of using S3 is too high, is it possible to move to a private server with those ".data" files? Or at least move to DigitalOcean Spaces.

2. From my understanding, AWS S3 calculates the cost by storage space, bandwidth and number of put/get requests. And DO Spaces has only 2 factors: 5$/month for 250 GB of storage, 1 TB of outbound transfer, Unlimited uploads, Unlimited Spaces. So I can understand that DO Spaces is a little bit better option compared to S3. Now my bill at S3 from Jan 1, 2019 is 3$ with 7GB storage / 12GB bandwidth / 40k put request / 260k get request.

Anyone can please share your experience with DO Spaces? Should I move to DO Spaces? And is it possible to transfer all current data from S3 to Spaces?

3. Is anyone using any solution for optimizing images on the server, with a reasonable costs? Is it a Xenforo Addon or you use third-party solution?

Thank you very much everyone to share your experience. Wish I can understanding everything you guys are sharing. Again, I'm not a tech guy

Why not a dedicated server ? Eg 2x2TB HDD
I think that are a cheap option.

online.net
kimsufi.com (ovh)
soyoustart.com (ovh)
hetzner.com/cloud
Or oneprovider.com .

 

[Sam9]

I'd suggest using Backblaze B2, it's S3 compatible, and considerably cheaper than both AWS S3 and DigitalOcean Spaces. Plus, you can put Cloudflare infront of it so bandwidth is 0 rated due to their bandwidth alliance. You'll only be paying $5 per TB storage, and $0 for transfer.

Cloudflare is pretty awesome...

 

[Chromaniac]

backblaze itself say that they do things a bit differently from s3. i could not find any thread where someone managed to use the S3 addon with backblaze (one thread has unanswered queries). did it work for you? if yes, that would be awesome. would be great if you can provide the code changes required to get it to work!

 

[Sim]

We've just passed 400,000 photos on ZooChat, consuming 150GB+ of storage space.

I run the site on an 8GB Linode ($40pm) with 200GB of block storage volume added ($20pm)

I wrote a tutorial on How to use Linode Block Storage for your XenForo attachments - it was really easy to set up.

 

[Sam9]

backblaze itself say that they do things a bit differently from s3. i could not find any thread where someone managed to use the S3 addon with backblaze (one thread has unanswered queries). did it work for you? if yes, that would be awesome. would be great if you can provide the code changes required to get it to work!

Our setup and use case is quite unique, we don't use Xenforo's flysystem based classes for uploading. However, you should be able to make it work using this: https://github.com/gliterd/flysystem-backblaze .

Another option is Wasabi which I've used in the past with Xenforo, they are fully S3 compatible, have similar pricing to B2, and in Cloudflare's bandwidth alliance as well. However, I'd say Backblaze is more reliable and a better option if you can get it working using the above flysystem adapter.

 

[Chromaniac]

i just started using s3 today. my user base is pretty small so i assume that i should be fine with s3 pricing. would watch for a month. if it goes beyond usd 5... would migrate to DO Spaces before looking at other alternatives. B2 would have been nice as I already use it for backup. But getting it to work probably goes beyond my level of expertise without a xenforo specific guide from someone who got it working! thanks!

 

[VersoBit]

i just started using s3 today. my user base is pretty small so i assume that i should be fine with s3 pricing. would watch for a month. if it goes beyond usd 5... would migrate to DO Spaces before looking at other alternatives. B2 would have been nice as I already use it for backup. But getting it to work probably goes beyond my level of expertise without a xenforo specific guide from someone who got it working! thanks!

Personally I would look at DigitalOcean as a serious contender for a reliable and more cost effective s3.

We are on kubernetes with 3 nodes, all of our internal data and external data is located in separate spaces, this is to balance the rate limit.

you’ll want to ensure all of your droplets and the spaces you create are within the free transfer zone. Add in a healthy nginx proxy cache with a decent storage volume, you’ll have yourself a cost effective option.

we’re pushing 250gb of data on our forum, and it’s growing monthly. With this solution I only need to account for the storage cost monthly, and bandwidth is handled by my droplets; being as I have more than just the severs for this website, my bandwidth pooling is shared which provides me more than enough bandwidth to supply all of my sites.