Option to store avatars in database

digitalpoint

Well-known member
Not sure what it's going to be like trying to keep avatars files within data/avatars in sync across multiple web servers... Especially when you get a ton of them.

csync2 works fairly well for maintaining cluster synchronization, but I can see where it might start getting obnoxious when it's trying to calculate exactly what files changed across 4 or 5 web servers when you have a ton of avatars (say you have 600,000 users and even of only 100,000 have avatars... 3 sizes saved, it's 300,000 files it has to check on every web server every time someone changes their avatar). It could even be the source of a user-triggered DDoS attack by changing their avatar over and over and your syncing mechanism not able to sync as fast as new avatars are coming in.

The database load for pulling avatars from the database is pretty minimal if you use Cache-Control headers properly... 1 query per avatar seen by the end user and then force their browser to not even bother checking if the image changed for 180 days or something (if the avatar does change, it will be reflected instantly since the timestamp of when it was last updated is appended to the avatar URL already).

Thankfully it doesn't look terribly hard to make an addon to make it database based if I need to extend Model_Avatar to do so... but still would be a nice option for bigger sites that have multiple web servers.
 
Upvote 8
I would not mind having a avatar limiter plugin that lets a user change their avata twice per minute, once per 6, 12 or 24 hours ..

This prevents a dos attack, but if they choose incorrectly, they can still change it.
 
I was using a SAN at one point, but got rid of it a few years ago after it failed... decided to go the route that anything in my setup can fail and the site isn't affected. Upstream router, any power supply, network cable, load balancers, server, etc. can be unplugged or just fail and we don't go down as far as end users are concerned. The only way to do that with a SAN is to have 2 independent SANs each feeding into independent SAN controllers on each server. And the cost involved for that is just silly.

For distribution of user supplied files (things like avatars and attachments) I'm probably just going to make a class to store them in the database. And then later when I get my beefy MySQL Cluster rig going (details), it should be able to do more than 5,000,000 SQL queries/second. Along with 43TB of drive space spread around multiple servers, it seems logical to use MySQL Cluster's data node partitioning to transparently replicate and store at least n copies of everything across multiple machines.
 
Yep... currently I'm storing everything in the database... avatars, profile pics, attachments, etc. The web server does not have write access to any directory/file on the existing site. And yeah, was playing a bit with the lsyncd (listens for the kernel notifications). I just think trying to keep hundreds of thousands of static files in sync across many web servers starts to become a little silly and overly complex.

We'll see though... plenty of other things to work on while I mull the options for avatars (and anything else the web server needs to write to the file system).
 
This is a suggestion thread, not a support thread.

If you wish to receive support, you will need to to associate your license with your forum account and post in the relevant customer support forum.
 
Yep... currently I'm storing everything in the database... avatars, profile pics, attachments, etc. The web server does not have write access to any directory/file on the existing site. And yeah, was playing a bit with the lsyncd (listens for the kernel notifications). I just think trying to keep hundreds of thousands of static files in sync across many web servers starts to become a little silly and overly complex.

We'll see though... plenty of other things to work on while I mull the options for avatars (and anything else the web server needs to write to the file system).
Did you resolve this, mind sharing how?
 
I made an addon to do it.
Understood, did you have to modify any core files?.

Are you considering releasing it, or should I look around for someone to develop something with similar functionality? I'm running into a similar issue where I’m tired of syncing attachments and avatars every time I scale servers up and down, and nfs or similar looks too precarious to me.
 
Well it was more or less built as bare-bones with only stuff I need... Specifically it does NOT have a function to move existing data to/from file system. So if you have existing attachments or avatars in the file system, they would all be lost.
 
Well it was more or less built as bare-bones with only stuff I need... Specifically it does NOT have a function to move existing data to/from file system. So if you have existing attachments or avatars in the file system, they would all be lost.
Fair enough.

Thank you for the quick response.
 
@digitalpoint are you still storing avatars and attachments in the database? Isn't that against performance?
It depends on your setup I suppose... if you are already database query bottlenecked and one extra query per second would cause problems, yes... it would be an issue. But *anything* would be an issue really.

But yes... we still store avatars and attachments in our database, and have no plans to change that.
 
I also want an option to store at least avatars in the filesystem

Avatars are already stored in the file system.

I mean in the database.

@digitalpoint on my current vBulletin installation I'm storing both avatars and attachments in the database without any lag nor issue so that's why I want to keep storing these in the DB in xenForo. Do you have any plans to release your add-on?
 
I mean in the database.

@digitalpoint on my current vBulletin installation I'm storing both avatars and attachments in the database without any lag nor issue so that's why I want to keep storing these in the DB in xenForo. Do you have any plans to release your add-on?
Not in the short-term, no... too much other stuff to do right now.
 
This was a suggestion for XenForo 1.0. Definitely not needed anymore since XenForo 2 has abstracted filesystem. Someone could store avatars in the database with the existing abstracted file system (or other places like AWS S3/Cloudflare R2). Direct support for storage in the database is not a relevant suggestion anymore.
 
Top Bottom