Scalability of avatar/attachment storage

[digitalpoint]

I'm curious if any "big boards" have run into issues with how avatars and attachments are stored in the web server file system?

Currently I run a setup with 4 load balanced web servers that all independent of each other (that way there is no single point of failure with things like a SAN failing). Let's say I had 200,000 users that had avatars plus 5,000,000 attachments. 200k avatars becomes 600k files because there are 3 sizes that are stored.

Of course you can use something like csync2 to keep files in sync across multiple servers, but I wonder if it's really worth it to try and keep millions of attachment/avatar files in sync every time someone uploads a new one. You also start having issues that need to be addressed if you have a web server that is down (either failed or taken down intentionally for maintenance).

On top of it, I'm probably going to be moving to a bigger cluster of 12 web servers sometime soon. It just seems that storage of attachments and avatars doesn't scale well as you bring on more web servers. It's more time trying to keep everything in sync every time someone uploads something, but also at the extreme you start running into disk space issues. For sake of argument, let's say I had 100 web servers... do I really want to store attachments/avatars 100 times each? Csync2 is extremely fast, but even if it took "only" 1 second to sync millions of files to a server (it would be longer), you are talking 12 seconds of delay every time someone uploaded something.

On a massive scale, imagine if Facebook was trying to store user avatars locally and sync all their web servers with those files every time someone uploaded a new one.

In addition to scalability issues, there are a few additional things I don't like about how avatars are stored... When using Imagemagick, you can't reject animated avatars. And all avatars are also stored in the file system as .jpeg files (even GIF/PNGs which makes the web server kick out invalid mime types for GIF or PNG avatars... so you are more or less relying on the web browser to assume site owners are stupid and correct invalid file types).

I'm thinking I may need to just rewriting parts of the Model_Avatar and Model_Attachment class to address all my issues and store stuff in the database rather than file system (1 query to read it from the database really isn't that much overhead as long as you use the proper HTTP headers so browsers aren't checking if it's changed since the last time it downloaded it).

 

[Deebs]

If you invest in a decent SAN then the problem goes away. I recommend 3PAR, the system I have invested in has 4 redundant power supplies. It powers the core of the business. 3PAR has recently been bought out by HP after a bidding war with Dell.

 

[digitalpoint]

I extended XenForo_Model_Avatar::uploadAvatar() to kick out an error to the user if they try to upload an animated avatar now. That was fairly simple.

 

[digitalpoint]

If you invest in a decent SAN then the problem goes away. I recommend 3PAR, the system I have invested in has 4 redundant power supplies. It powers the core of the business. 3PAR has recently been bought out by HP after a bidding war with Dell.

Definitely not going the route of a SAN. Have had bad experiences in the past (even with ones that were "redundant"). We had a daughter board go out on one that handled the power supplies... so even though we had redundant power supplies, the thing that controlled them died internally. The only way to do it and truly not have a single point of failure would be to get multiple independent SANs that are fed through different SAN switches and have multiple fiber channel controllers on each server. And that starts becoming massively expensive and ends up being a very complex solution to a simple problem.

The way we have things setup up now, we have no single point of failure... to the point that if we needed to physically move servers/equipment to an entirely different data center, users would experience no down time.

 

[Deebs]

Definitely not going the route of a SAN. Have had bad experiences in the past (even with ones that were "redundant"). We had a daughter board go out on one that handled the power supplies... so even though we had redundant power supplies, the thing that controlled them died internally. The only way to do it and truly not have a single point of failure would be to get multiple independent SANs that are fed through different SAN switches and have multiple fiber channel controllers on each server. And that starts becoming massively expensive and ends up being a very complex solution to a simple problem.

The way we have things setup up now, we have no single point of failure... to the point that if we needed to physically move servers/equipment to an entirely different data center, users would experience no down time.

Can you seriously prove that? We have 3 SANs from 3PAR, everything point is redundant, multiple power supplies, multiple fabric connections, multiple datacentres, multiple data points.All reliant on a single "box".

What I see is a bad experience on a **** SAN... SAN's are meant and designed to be unbreakable.

 

[Deebs]

Just one more thing to add, Facebook uses 3PAR SANs

 

[digitalpoint]

Can you seriously prove that? We have 3 SANs from 3PAR, everything point is redundant, multiple power supplies, multiple fabric connections, multiple datacentres, multiple data points.All reliant on a single "box".

What I see is a bad experience on a **** SAN... SAN's are meant and designed to be unbreakable.

Prove what, that ours failed? I guess you could go back in time about 3 years and try to get to our website which was down for about 6 hours because of it. To be fair, Dell was VERY responsive to getting it resolved. Had a call in to them around 9pm, and by 2am, they had a technician that drove about 2 1/2 hours to our data center from their Los Angeles warehouse with the replacement part, and we were back online by about 3am. In the end, it was a daughter board in the unit that was about 3"x3"... and apparently it's what detects when a power supply fails and does the switching when one does. But yeah... IT failed, so *no* power supplies were being routed to the chassis. lol

Either way, that was when I decided that in the future we are going to stay away from anything that shares a physical chassis... at least as far as considering as being able to never fail. We still have some blade chassis' that contain 10 blades per... they have been fine, but same deal... when they get replaced, not going with stuff that shares a single chassis.

 

[Deebs]

Prove what, that ours failed? I guess you could go back in time about 3 years and try to get to our website which was down for about 6 hours because of it. To be fair, Dell was VERY responsive to getting it resolved. Had a call in to them around 9pm, and by 2am, they had a technician that drove about 2 1/2 hours to our data center from their Los Angeles warehouse with the replacement part, and we were back online by about 3am. In the end, it was a daughter board in the unit that was about 3"x3"... and apparently it's what detects when a power supply fails and does the switching when one does. But yeah... IT failed, so *no* power supplies were being routed to the chassis. lol

Either way, that was when I decided that in the future we are going to stay away from anything that shares a physical chassis... at least as far as considering as being able to never fail. We still have some blade chassis' that contain 10 blades per... they have been fine, but same deal... when they get replaced, not going with stuff that shares a single chassis.

That just shows an issue tho. Dell created a SAN that relied on a single board to function. 3PAR use a system where every node is ACTIVE and as long as it receives power it works and provides data, why do you think it caused a bidding war between HP and Dell? I spoke to my HP account manager last year and they rate 3PAR as their top level system into data storage followed by EVA.

 

[digitalpoint]

Just one more thing to add, Facebook uses 3PAR SANs

Yep... and I would guess they have multiple redundant/independent SANs. If one *were* to fail, I suspect Facebook would not go down.

I'm not against SANs, I'm just not willing to use ONE... if I was willing to invest in at least TWO that ran independent of each other, then sure... they are great then.

 

[agison]

So The easy solution here is a suggestion to have an option to store Avatars/Attachments in Database. One more request per file is not a bad idea.

Here another idea to use csync with lsyncd: https://www.axivo.com/community/threads/lightning-fast-synchronization-with-csync2-and-lsyncd.121/

 

[digitalpoint]

Yeah... I already use lsyncd with csync2 for the normal stuff that is synced (it's great for things like PHP/site files). I'm just not terribly fond of the idea of needing to replicate gigs and gigs of data to every web server.

I'll probably have database storage for avatars working soon... I just spent a little time with it and now have the avatars being added to and deleted from the database as needed (when a user uploads/deletes theirs). Now really just a matter of making a PHP file that outputs the image and using mod_rewrite to intercept those requests.

If this works, then maybe tomorrow I'll do attachments too.

 

[agison]

Yeah... I already use lsyncd with csync2 for the normal stuff that is synced (it's great for things like PHP/site files). I'm just not terribly fond of the idea of needing to replicate gigs and gigs of data to every web server.

I'll probably have database storage for avatars working soon... I just spent a little time with it and now have the avatars being added to and deleted from the database as needed (when a user uploads/deletes theirs). Now really just a matter of making a PHP file that outputs the image and using mod_rewrite to intercept those requests.

If this works, then maybe tomorrow I'll do attachments too.

Great!
Will you release it as an Add-on?

 

[digitalpoint]

Maybe... right now just focused on getting everything I need for my own site before I start spending time trying to wrap up installer packages and supporting people.

My avatars are totally out of the filesystem now, PNG and GIF images send the right mime-type from the server, animated GIFs are disallowed automatically and we are forcing a "Cache-Control" header that tells browsers to don't even try checking if the image has changed until a year from now (if an image HAS actually changed it will update since the URL is different with the timestamp in it).

The only thing I don't like about it all is I decided to not touch the avatar file names... so they still end with .jpg even if they are PNG or GIF files, but at least they aren't sending an image/jpeg mime type header (the right image type header is sent).

I just decided it was probably more work than it was worth trying to find all the places XF generates the avatar file name, and I'm instead just using mod_rewrite to intercept the requests for avatars and routing them to my PHP script that will output the image...

RewriteRule ^data/avatars/(.)/.*/(.*).jpg$ data/avatar.php?user_id=$2&size=$1 [NC,L]

Have not even looked at the Attachment class, but I assume it should be similarly easy to move into the database.

 

[Mert]

The best solution is either storing attachments/avatars on a separate server or on CDN. I modified vBulletin so now our big board is hosting attachments on a separate server. Huge improvement and highly suggest doing so.

 

[digitalpoint]

The best solution is either storing attachments/avatars on a separate server or on CDN. I modified vBulletin so now our big board is hosting attachments on a separate server. Huge improvement and highly suggest doing so.

I have trust issues. lol I don't trust a third party CDN to be better than the mini one I already have/built myself specifically for my site.

dpstatic.com is my cookieless "CDN" domain for that sort of thing (CSS, JS and images all served through it for my current site).

Plus to make any sort of *real* impact performance-wise, the cost for us is something stupid like ~$25,000/month simply because of how many static files we are currently serving (somewhere around 1,000/second... obviously most of it nothing to do with the forum).

 

[Mert]

I also considered setting a separate server for static files but that will not match CDN performance for end user. CDN delivers files with geo ip targeting so files are loaded way faster.

 

[digitalpoint]

I also considered setting a separate server for static files but that will not match CDN performance for end user. CDN delivers files with geo ip targeting so files are loaded way faster.

I wouldn't say *way* faster... certainly not $25k/month faster (in my case). As long as your static files are properly setup to do things like instruct the web browser to never check if the file has changed, it might be an extra 0.2 seconds on the first load, but after that it's fast. Unless you can move your dynamic content into geographical proximity with multiple in-sync data centers it's really not that great in my opinion... and if you were doing that, you might as well have multiple of your own geographical static content servers.

Either way, that's a bit off topic...

As far as the original topic, I finished up making a XF addon that makes attachments live in the database (and did the one for avatars earlier today), so I guess for me, it has all become a moot point.

 

[zaiger]

Our wiki serves about that and the forum images are on the same server and it runs fine. Our forum is only 2% of our traffic, but it's not a small forum by any means.

 

[Digital Doctor]

Our wiki serves about that and the forum images are on the same server and it runs fine. Our forum is only 2% of our traffic, but it's not a small forum by any means.

What wiki do you use ?

 

[zaiger]

I guess I didn't read the OP well last night. Sorry for that.

What wiki do you use ?

We use MediaWiki. We get upwards of 100 requests per second with like 5-10 static images per page (on the wiki, many more than that on the forum obviously) and everything runs great, no major complications concerning images. We use Cloudflare though, which lessens the load a lot. By more than half usually, and static images are cached for a long time.

I mean we are not by any means as large as digitalpoint.com. They are just over 300 Alexa ranked and we are over 13k (5k in the US tho ). I guess what I'm getting at is that as a site that gets a couple of million views a day, we don't have any problems with Xenforo and syncing images across multiple servers (we have 7 servers for this particular site).