1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Option to store avatars in database

Discussion in 'XenForo Suggestions' started by digitalpoint, Oct 29, 2011.

  1. digitalpoint

    digitalpoint Well-Known Member

    Not sure what it's going to be like trying to keep avatars files within data/avatars in sync across multiple web servers... Especially when you get a ton of them.

    csync2 works fairly well for maintaining cluster synchronization, but I can see where it might start getting obnoxious when it's trying to calculate exactly what files changed across 4 or 5 web servers when you have a ton of avatars (say you have 600,000 users and even of only 100,000 have avatars... 3 sizes saved, it's 300,000 files it has to check on every web server every time someone changes their avatar). It could even be the source of a user-triggered DDoS attack by changing their avatar over and over and your syncing mechanism not able to sync as fast as new avatars are coming in.

    The database load for pulling avatars from the database is pretty minimal if you use Cache-Control headers properly... 1 query per avatar seen by the end user and then force their browser to not even bother checking if the image changed for 180 days or something (if the avatar does change, it will be reflected instantly since the timestamp of when it was last updated is appended to the avatar URL already).

    Thankfully it doesn't look terribly hard to make an addon to make it database based if I need to extend Model_Avatar to do so... but still would be a nice option for bigger sites that have multiple web servers.
  2. Floris

    Floris Guest

    I would not mind having a avatar limiter plugin that lets a user change their avata twice per minute, once per 6, 12 or 24 hours ..

    This prevents a dos attack, but if they choose incorrectly, they can still change it.
    ankurs likes this.
  3. AlexT

    AlexT Well-Known Member

  4. digitalpoint

    digitalpoint Well-Known Member

    I was using a SAN at one point, but got rid of it a few years ago after it failed... decided to go the route that anything in my setup can fail and the site isn't affected. Upstream router, any power supply, network cable, load balancers, server, etc. can be unplugged or just fail and we don't go down as far as end users are concerned. The only way to do that with a SAN is to have 2 independent SANs each feeding into independent SAN controllers on each server. And the cost involved for that is just silly.

    For distribution of user supplied files (things like avatars and attachments) I'm probably just going to make a class to store them in the database. And then later when I get my beefy MySQL Cluster rig going (details), it should be able to do more than 5,000,000 SQL queries/second. Along with 43TB of drive space spread around multiple servers, it seems logical to use MySQL Cluster's data node partitioning to transparently replicate and store at least n copies of everything across multiple machines.
  5. AlexT

    AlexT Well-Known Member

  6. digitalpoint

    digitalpoint Well-Known Member

    Yep... currently I'm storing everything in the database... avatars, profile pics, attachments, etc. The web server does not have write access to any directory/file on the existing site. And yeah, was playing a bit with the lsyncd (listens for the kernel notifications). I just think trying to keep hundreds of thousands of static files in sync across many web servers starts to become a little silly and overly complex.

    We'll see though... plenty of other things to work on while I mull the options for avatars (and anything else the web server needs to write to the file system).
  7. Hannan

    Hannan New Member

    Hi this is cool discussion and guide going on here. I have a problem here: http://extreme-reptiles.com/forum/threads/latrodectus-sp-widow-spiders.53/

    I could not post my reply to any comment don't knwo why there is any issue gets raised when I go to reply any comment/ article.

    <login details removed by Brogan>

    Also could you tell me where Avatar is assigned / marked / allocated to each user in database?
  8. Brogan

    Brogan XenForo Moderator Staff Member

    This is a suggestion thread, not a support thread.

    If you wish to receive support, you will need to to associate your license with your forum account and post in the relevant customer support forum.
  9. whyweprotest

    whyweprotest Well-Known Member

    Did you resolve this, mind sharing how?
  10. digitalpoint

    digitalpoint Well-Known Member

    I made an addon to do it.
  11. whyweprotest

    whyweprotest Well-Known Member

    Understood, did you have to modify any core files?.

    Are you considering releasing it, or should I look around for someone to develop something with similar functionality? I'm running into a similar issue where I’m tired of syncing attachments and avatars every time I scale servers up and down, and nfs or similar looks too precarious to me.
  12. digitalpoint

    digitalpoint Well-Known Member

    Well it was more or less built as bare-bones with only stuff I need... Specifically it does NOT have a function to move existing data to/from file system. So if you have existing attachments or avatars in the file system, they would all be lost.
  13. whyweprotest

    whyweprotest Well-Known Member

    Fair enough.

    Thank you for the quick response.
  14. imthebest

    imthebest Formerly Super120

    I also want an option to store at least avatars in the filesystem.

    @digitalpoint are you still storing avatars and attachments in the database? Isn't that against performance?
  15. Jeremy

    Jeremy Well-Known Member

    Avatars are already stored in the file system.
  16. digitalpoint

    digitalpoint Well-Known Member

    It depends on your setup I suppose... if you are already database query bottlenecked and one extra query per second would cause problems, yes... it would be an issue. But *anything* would be an issue really.

    But yes... we still store avatars and attachments in our database, and have no plans to change that.
  17. imthebest

    imthebest Formerly Super120

    I mean in the database.

    @digitalpoint on my current vBulletin installation I'm storing both avatars and attachments in the database without any lag nor issue so that's why I want to keep storing these in the DB in xenForo. Do you have any plans to release your add-on?
  18. digitalpoint

    digitalpoint Well-Known Member

    Not in the short-term, no... too much other stuff to do right now.

Share This Page