Converting Big Site To XF - Server Specs?
- Thread starter ProCom
- Start date
ProCom
Well-known member
I REALLY appreciate all the ideas and support!
Regarding the 4TB: I think my previous software/company stored full-size images AND multiple resized versions of each image, so it's possible that there is a lot of duplication. The problem is that the various image sizes have already been used and referenced in posts and articles, so I doubt I'd be able to get rid of them since all the references point to those various files.
Regarding AWS S3: Yes, it's super cheap to store the files, but wouldn't the volume of bandwidth, requests, etc. kill me in cost?
Regarding the 4TB: I think my previous software/company stored full-size images AND multiple resized versions of each image, so it's possible that there is a lot of duplication. The problem is that the various image sizes have already been used and referenced in posts and articles, so I doubt I'd be able to get rid of them since all the references point to those various files.
Regarding AWS S3: Yes, it's super cheap to store the files, but wouldn't the volume of bandwidth, requests, etc. kill me in cost?
CarpCharacin
Well-known member
ProCom
Well-known member
Unless I'm mistaken, the pricing I see for transfer is about $0.09 per GB (Data Transfer OUT From Amazon S3 To Internet): https://aws.amazon.com/s3/pricing/
I have no idea what my bandwidth for images will be, but I'm guessing it will be high... maybe about 15 TB per month, so around $1,000 a month with AWS S3?
I have no idea what my bandwidth for images will be, but I'm guessing it will be high... maybe about 15 TB per month, so around $1,000 a month with AWS S3?
You need to consider leveraging services like cloudflare too, that would help reduce bandwidth from your server.
You can start free or $20 a month for a pro account.
CarpCharacin
Well-known member
You could go with 1 dedicated server for now. You can always add a second one later if you need it.
WSWD
Well-known member
No...you are not mistaken. S3 is UNGODLY expensive as far as bandwidth is concerned, which is actually quite funny, since bandwidth costs have drammatically decreased over the last couple years. Basically $90/TB. Bandwidth in the absolutely best datacenters in the country doesn't cost that much.
Mouth
Well-known member
Firstly, you should find out from your current provider what your monthly data transfer is.
You wouldn't get users requesting directly from S3, that would just be for your storage. You'd use Cloudfront for CDN, populated from your S3 storage bucket, so only the Cloudfront edge servers would be retrieving from S3. https://aws.amazon.com/cloudfront/pricing/ and scroll down to the bottom for 'reserved capacity pricing' that would apply to your usage requirement, if it is 10+ TB per month. I've heard of 20TB data transfer from Cloudfront for only around $250 p/mth.
JamesJ
Member
It'll be better to have a cluster of machines (preferably VPS') all running the same software / setup, which are clustered and load balanced.
For image storage, slap up a Amazon Web Services S3 bucket, post the images there, then use Cloudflare with an aggressive caching rule and CNAME images.yourdomain.com to yourbucketid.s3.amazonaws.com and this'll reduce your bandwidth costs.
I'd advise you don't use Cloudfront like @Mouth suggested above, mainly as you have to pay for bandwidth -- and bandwidth isn't cheap via AWS. Let alone, if you get an L7 attack which requests lots of images, AWS wont care -- you'll still be charged.
Best of luck.
For image storage, slap up a Amazon Web Services S3 bucket, post the images there, then use Cloudflare with an aggressive caching rule and CNAME images.yourdomain.com to yourbucketid.s3.amazonaws.com and this'll reduce your bandwidth costs.
I'd advise you don't use Cloudfront like @Mouth suggested above, mainly as you have to pay for bandwidth -- and bandwidth isn't cheap via AWS. Let alone, if you get an L7 attack which requests lots of images, AWS wont care -- you'll still be charged.
Best of luck.
WSWD
Well-known member
CloudFlare isn't going to do a thing for his resources. The biggest resource is disk space. What is CF going to do for that? Sure, it's going to save some bandwith, but as I said, bandwith is so cheap these days that it really doesn't matter. Well....it's cheap everywhere except Amazon. Their fiber lines must be solid gold or something.
Keeping it on one server is going to result in a pair of RAID arrays (at the least)...one for the SSD drives and the other for the storage array. You're looking at a beefy server with at least 6 drives, when you could simply split it up amongst a pair of servers and be finished with it. Can't say that multiple servers is really going to complicate matters. In fact, it's an incredibly easy setup.
Brent W
Well-known member
CloudFlare isn't going to do a thing for his resources. The biggest resource is disk space. What is CF going to do for that? Sure, it's going to save some bandwith, but as I said, bandwith is so cheap these days that it really doesn't matter. Well....it's cheap everywhere except Amazon. Their fiber lines must be solid gold or something.
Keeping it on one server is going to result in a pair of RAID arrays (at the least)...one for the SSD drives and the other for the storage array. You're looking at a beefy server with at least 6 drives, when you could simply split it up amongst a pair of servers and be finished with it. Can't say that multiple servers is really going to complicate matters. In fact, it's an incredibly easy setup.
You don't even know what his diskspace requirements are going to be after the conversion so save me on the lecture
CloudFlare does more than just save bandwidth. It saves processing power on serving all static content that is cached as well.Like I said, you have no idea what his server requirements will be and neither does he now. He is coming from a proprietary software.
You can keep making blind recommendations for him if you'd like but I'd prefer to see what his actual requirements are going to be.
WSWD
Well-known member
You don't even know what his diskspace requirements are going to be after the conversion so save me on the lecture
Yeah....
He has 4TB worth of images and attachments. Where do you think those are all going to go? Are those magically going to become 100GB after the conversion? Let's apply a little bit of common sense here.
Serving static content uses a miniscule amount of processing power. The connections are going to have a much bigger impact on RAM than on the processor.
Brent W
Well-known member
What you don't know is that Huddler ballooned those attachments. Before his conversion to Huddler his attachment size was 15GB. You also don't know how Huddler handles attachments. Most of that size could realistically be reduced if sizes are created by huddler that aren't needed by XenForo.
So how about you apply some common sense and don't recommend a setup to him without knowing what he will be dealing with ok?
Unless you have the information above. Then by all means, let us know what the end conversion attachment size will be for him.Which is fine but everything helps in the end. Especially at his volume level.
CarpCharacin
Well-known member
Huddler stores multiple sizes of the same images.
Brent W
Well-known member
But continue making recommendations based off non existent data if you wish. At the end of the day, you don't know what he is going to be working with so use common sense and wait. Or continue to respond. Either way, I don't care. I've made my point and will be more than happy to revisit this thread when we have actual XenForo conversion data to go on.
Brent W
Well-known member
Which is easy to do when you have the actual data
Of course but it puts an end to the endless circle argument.
