XF 2 (Paid) Need someone who knows how to use multiple dedicated servers to extend storage

[John917]

I want to learn more about increasing storage space by using multiple remote dedicated servers/vps. If you are experienced in this please message me.
Would like to set up a remote elastic search too.

Preferably using something popular like Network File System (NFS) or the like.

I'm running ubuntu, nginx, mysql.

 

[Arantor]

Honestly, that's about the most expensive way I can imagine to do it. Why not use any of the big providers' and their block storage solutions? AWS have S3, Linode have one, DigitalOcean have one - and you can add many GBs of storage for far less than paying full fat VPS rates.

See https://xenforo.com/community/threa...-amazon-s3-for-file-storage-in-xf-2-1.156282/ which is written by one of the XF devs on how to set this up.

 

[John917]

Honestly, that's about the most expensive way I can imagine to do it. Why not use any of the big providers' and their block storage solutions? AWS have S3, Linode have one, DigitalOcean have one - and you can add many GBs of storage for far less than paying full fat VPS rates.

See https://xenforo.com/community/threa...-amazon-s3-for-file-storage-in-xf-2-1.156282/ which is written by one of the XF devs on how to set this up.

Yes that manual is similar to what I had in mind. VPS block or object servers are good rates but there are dedicated servers with more harddrive space for a smaller price.

Despite price, do you think S3 Flysystem is more efficient than NFS system? It "seems" NFS looks more easier to setup than the S3.
I wonder if that S3 tutorial is only supported for vps that have things preconfigured already, or if I can use that tutorial for dedicated servers too.

 

[Sim]

Exactly how much storage space do you need?

 

[John917]

Exactly how much storage space do you need?

Hi Sim, I'm planning ahead but if my planning is right I'll eventually need more than 6tb of hd. A lot of image submissions.

 

[Sim]

Surely it can't be that difficult to find a dedicated server which will have 6TB of storage space on board? No need for complicated NFS setups?

Or just find someone with a NAS solution, which should be largely plug-and-play.

 

[John917]

Surely it can't be that difficult to find a dedicated server which will have 6TB of storage space on board? No need for complicated NFS setups?

Or just find someone with a NAS solution, which should be largely plug-and-play.

Well I was hoping to save on costs in the beginning if the NFS setup was easy enough.

NAS solution seems really cool. For that I would have to get a main dedicated server from the provider and hope that the same provider has NAS servers offered right? From there I can just order more HD space as needed from the same provider?

Only downside is that some popular providers seem to charge a heavy fee for NAS servers.

 

[Arantor]

Well, Flysystem doesn’t have a permanent connection to the networked resource so you’re not dealing with the stability issues that can arise if the connection isn’t good (NFS can really mess you about if you’re not careful)

Am curious to see which dedicated server providers will give you 6TB of storage cheaper than any of the folks who literally offer storage as a product, that would seem to completely blow their offering out the water.

 

[John917]

Well, Flysystem doesn’t have a permanent connection to the networked resource so you’re not dealing with the stability issues that can arise if the connection isn’t good (NFS can really mess you about if you’re not careful)

Am curious to see which dedicated server providers will give you 6TB of storage cheaper than any of the folks who literally offer storage as a product, that would seem to completely blow their offering out the water.

Thanks for that Arantor. Does Flysystem have it where if one storage server were to go down, the website won't completely shutdown? I guess certain images won't load if that were to happen? Completely new to this.

There's a lot of super cheap dedicated servers in Kansas that I use for backup. Joe's datacenter. I have custom pricing and specs that they hooked me up with so it's cheaper than what's listed.

 

[briansol]

I also recommend using S3 with cloudfront and cloudflare in front of it for caching.
see: https://xenforo.com/community/threa...hared-or-vps-suggestions.207196/#post-1582089

 

[Kirby]

Am curious to see which dedicated server providers will give you 6TB of storage cheaper than any of the folks who literally offer storage as a product, that would seem to completely blow their offering out the water.

Dedicated Server
Hetzner PX 62
2x 8 TB HDD
20 TB Traffic
=> 88 € / month

Cloud offers (Block / Object storage)
AWS S3 Standard Tier (Europe / Frankfurt)
Storage: 8000 GB * 0,0245 $ = 196 $
And that is just storage, not including traffic or requests - even just storage on Infrequent access tier would be > 100 $
Could be combined with CloudFront (or any other CDN) to reduce traffic / requests, but not storage costs.

Digital Ocean Spaces
Storage: 5 $ + 7750 GB * 0,02 $ = 160 $
Again just for storage - not including traffic

Digital Ocean Volumes
1000 GB * 8 * 100 $ = 800 $
Again just for storage - your would need a server instance as well at Digital Ocean to attach the volume

Linode Object Storage
5 TB (100 $) + 3000 MB * 0,02 $ = 120 $
Again just for storage - not including traffic

Linode Block Storage
2560 GB (256 $) + 5120 GB (512 $) => ~ 7680 GB for 768 $
Again just for storage - your would need a server instance as well at Linode to attach the volume

Cloud storage (especially object storage) is easy, but definitly not cheap.

PS: If I got smth. wrong please correct me

 

[briansol]

Pairing S3 with cloudflare and a bandwidth alliance partner main host is critical to financial success with using cloud or the egress charges will get you.

see the link in my sig for the Alliance companies. I use Vultr, and they are in. There are many others.


I haven't set this up yet, but it's on my to do list to pilot, to move from S3 standard to the infrequent access tier.
It's basically half the price of standard s3.

S3 Standard - Infrequent Access** - For long lived but infrequently accessed data that needs millisecond access
All Storage / Month $0.0125 per GB

So, pairing infrequent access with a good caching front end (cloudfront and cloudflare) is likely possible.

Consider most of the active threads will be actively cached, while old archived threads will not be accessed often, if at all if there's no search traffic coming in to it.

Another approach i'm considering is leaving the thumbnails in regular tier and the full size attachments in the infrequent access. Some % of images might not be expanded as people browse the thread. Of course, that fails in concept if a lot of folks paste in the full size image instead of the attachment, and that is common for my site.

 

[Arantor]

To answer the question about Flysystem, it’s really just a fancy API connector to S3-compatible storage in this case. If S3 or whatever goes down it’s not like the whole site goes with it.

The other thing I’d factor in to costing out dedicated servers for that is that you probably need to talk about having two of them because backups are a thing to consider and that will change the dynamics, unless you’re going to somehow manage the backups of that 6TB yourself (which is another thing again)