Reply to thread

The problem:


You want to store attachments and avatars in a non-local filesystem.


The solution:


There are currently 3 locations you can override filesystem adaptors:  data, internal-data and code-cache.  So to solve "the problem" you can override data and internal-data to use something like AWS S3.  So far, so good...


The issue:


Overwriting the data and internal-data filesystem is (for the most part) an "all or nothing" thing.  So when I really just wanted to store internal_data/attachments and data/avatars in the cloud, now I'm forced to also store a whole slew of other things in the cloud:


WhereWhat
data/attachmentsAttachment thumbnails
data/audioUser uploaded audio
data/imported_reactionsReactions imported from other forum software
data/profile_bannersUser uploaded banners
data/stylesStyles imported from an archive (why is this in data rather than internal_data?)
data/videoUser uploaded video
internal_data/addon_batchData only used when installing add-ons from an archive
internal_data/file_checkInfo about nightly file checks
internal_data/image_cacheImages from image proxy
internal_data/imported_xmlImported smiles
internal_data/keysEmail DKIM key
internal_data/oembed_cacheMedia embeds
internal_data/sitemapsSitemaps for search engines


Examples of why you wouldn't want to store some of these items in the cloud:

  • Making cloud API calls on the backend every time a user views a reaction on a post (or is given an opportunity to use that reaction) just to get the reaction image.  Not efficient.
  • Reading and writing what ultimately is temporary data to the cloud when you install an add-on.  Not efficient.
  • Reading proxied images via cloud API calls every time someone views a proxied image (at least partially) defeats the purpose of the image proxy.
  • Making cloud API calls to read what is a singular DKIM key every time an email is sent?  This one is SO bad.  We've now slowed down outgoing mails and added an actual cost to send emails (cloud API calls aren't terribly expensive, but if you are sending a ton of emails from XenForo, which large sites will be doing, 10,000,000 individually cheap emails is a real cost).  Real-world example:  Let's say the API call only takes 1/4 of a second to fully complete.  You send a mass email to 10,000 users.  Doing the same API call once for each email (to get the same key each time), and you just added 42 minutes of extra time to send those 10,000 emails on top of paying for each API call.

There's already a suggestion thread specifically related to the DKIM keys, but I think it would be better to have that simply be part of what could be a more granular system:


[URL unfurl="true"]https://xenforo.com/community/threads/dkim-change-location-of-key-material.205277/post-1571144[/URL]


I don't think it would be terribly hard to do and also maintain backward compatibility (see my post in the above thread).


Back
Top Bottom