Converting Big Site To XF - Server Specs?

I have put my attachments on Amazon S3 for the purpose of using different machines with one file system. On S3 you can store unlimited data and its with 13 USD / TB / month not so expensive. You would have to pay around 60 USD then.
 
I REALLY appreciate all the ideas and support!

Regarding the 4TB: I think my previous software/company stored full-size images AND multiple resized versions of each image, so it's possible that there is a lot of duplication. The problem is that the various image sizes have already been used and referenced in posts and articles, so I doubt I'd be able to get rid of them since all the references point to those various files.

Regarding AWS S3: Yes, it's super cheap to store the files, but wouldn't the volume of bandwidth, requests, etc. kill me in cost?
 
Unless I'm mistaken, the pricing I see for transfer is about $0.09 per GB (Data Transfer OUT From Amazon S3 To Internet): https://aws.amazon.com/s3/pricing/

I have no idea what my bandwidth for images will be, but I'm guessing it will be high... maybe about 15 TB per month, so around $1,000 a month with AWS S3?
 
Unless I'm mistaken, the pricing I see for transfer is about $0.09 per GB (Data Transfer OUT From Amazon S3 To Internet): https://aws.amazon.com/s3/pricing/

I have no idea what my bandwidth for images will be, but I'm guessing it will be high... maybe about 15 TB per month, so around $1,000 a month with AWS S3?
You need to consider leveraging services like cloudflare too, that would help reduce bandwidth from your server.

You can start free or $20 a month for a pro account.
 
Unless I'm mistaken, the pricing I see for transfer is about $0.09 per GB (Data Transfer OUT From Amazon S3 To Internet): https://aws.amazon.com/s3/pricing/

I have no idea what my bandwidth for images will be, but I'm guessing it will be high... maybe about 15 TB per month, so around $1,000 a month with AWS S3?

No...you are not mistaken. S3 is UNGODLY expensive as far as bandwidth is concerned, which is actually quite funny, since bandwidth costs have drammatically decreased over the last couple years. Basically $90/TB. Bandwidth in the absolutely best datacenters in the country doesn't cost that much.
 
Unless I'm mistaken, the pricing I see for transfer is about $0.09 per GB (Data Transfer OUT From Amazon S3 To Internet): https://aws.amazon.com/s3/pricing/
I have no idea what my bandwidth for images will be, but I'm guessing it will be high... maybe about 15 TB per month, so around $1,000 a month with AWS S3?
Firstly, you should find out from your current provider what your monthly data transfer is.
You wouldn't get users requesting directly from S3, that would just be for your storage. You'd use Cloudfront for CDN, populated from your S3 storage bucket, so only the Cloudfront edge servers would be retrieving from S3. https://aws.amazon.com/cloudfront/pricing/ and scroll down to the bottom for 'reserved capacity pricing' that would apply to your usage requirement, if it is 10+ TB per month. I've heard of 20TB data transfer from Cloudfront for only around $250 p/mth.
 
It'll be better to have a cluster of machines (preferably VPS') all running the same software / setup, which are clustered and load balanced.

For image storage, slap up a Amazon Web Services S3 bucket, post the images there, then use Cloudflare with an aggressive caching rule and CNAME images.yourdomain.com to yourbucketid.s3.amazonaws.com and this'll reduce your bandwidth costs.

I'd advise you don't use Cloudfront like @Mouth suggested above, mainly as you have to pay for bandwidth -- and bandwidth isn't cheap via AWS. Let alone, if you get an L7 attack which requests lots of images, AWS wont care -- you'll still be charged.

Best of luck.
 
None of these suggestions really matter until we know what post XenForo conversion is going to look like. I prefer the keep it simple method. If you can keep it on one server, then do that imo. CloudFlare is going to save you a lot of resources.
 
None of these suggestions really matter until we know what post XenForo conversion is going to look like. I prefer the keep it simple method. If you can keep it on one server, then do that imo. CloudFlare is going to save you a lot of resources.

CloudFlare isn't going to do a thing for his resources. The biggest resource is disk space. What is CF going to do for that? Sure, it's going to save some bandwith, but as I said, bandwith is so cheap these days that it really doesn't matter. Well....it's cheap everywhere except Amazon. Their fiber lines must be solid gold or something. ;)

Keeping it on one server is going to result in a pair of RAID arrays (at the least)...one for the SSD drives and the other for the storage array. You're looking at a beefy server with at least 6 drives, when you could simply split it up amongst a pair of servers and be finished with it. Can't say that multiple servers is really going to complicate matters. In fact, it's an incredibly easy setup.
 
CloudFlare isn't going to do a thing for his resources. The biggest resource is disk space. What is CF going to do for that? Sure, it's going to save some bandwith, but as I said, bandwith is so cheap these days that it really doesn't matter. Well....it's cheap everywhere except Amazon. Their fiber lines must be solid gold or something. ;)

Keeping it on one server is going to result in a pair of RAID arrays (at the least)...one for the SSD drives and the other for the storage array. You're looking at a beefy server with at least 6 drives, when you could simply split it up amongst a pair of servers and be finished with it. Can't say that multiple servers is really going to complicate matters. In fact, it's an incredibly easy setup.

You don't even know what his diskspace requirements are going to be after the conversion so save me on the lecture ;) CloudFlare does more than just save bandwidth. It saves processing power on serving all static content that is cached as well.

Like I said, you have no idea what his server requirements will be and neither does he now. He is coming from a proprietary software.

You can keep making blind recommendations for him if you'd like but I'd prefer to see what his actual requirements are going to be.
 
You don't even know what his diskspace requirements are going to be after the conversion so save me on the lecture ;)

Yeah....

He has 4TB worth of images and attachments. Where do you think those are all going to go? Are those magically going to become 100GB after the conversion? Let's apply a little bit of common sense here.


It saves processing power on serving all static content that is cached as well.

Serving static content uses a miniscule amount of processing power. The connections are going to have a much bigger impact on RAM than on the processor.
 
Yeah....

He has 4TB worth of images and attachments. Where do you think those are all going to go? Are those magically going to become 100GB after the conversion? Let's apply a little bit of common sense here.

What you don't know is that Huddler ballooned those attachments. Before his conversion to Huddler his attachment size was 15GB. You also don't know how Huddler handles attachments. Most of that size could realistically be reduced if sizes are created by huddler that aren't needed by XenForo.

So how about you apply some common sense and don't recommend a setup to him without knowing what he will be dealing with ok? ;) Unless you have the information above. Then by all means, let us know what the end conversion attachment size will be for him.

Serving static content uses a miniscule amount of processing power. The connections are going to have a much bigger impact on RAM than on the processor.

Which is fine but everything helps in the end. Especially at his volume level.
 
15GB of attachments were ballooned to 4TB? No.

When I moved from Huddler 5 years ago, we only had around 15 GB of images. Even considering growth and larger image size support, I don't understand how it ballooned to 3.7 TB! I'm wondering if they duplicate and store multiple sizes of each image... that's my only guess.

But continue making recommendations based off non existent data if you wish. At the end of the day, you don't know what he is going to be working with so use common sense and wait. Or continue to respond. Either way, I don't care. I've made my point and will be more than happy to revisit this thread when we have actual XenForo conversion data to go on.
 
Back
Top Bottom