Object Storage Adapter

Object Storage Adapter 0.4.2

No permission to download
Overview FAQ Updates (2) Reviews (2) History Discussion
Don Daniello

Don Daniello

Member
Don Daniello submitted a new resource:

Object Storage Adapter - Use cloud object storage like OpenStack Swift or S3-compatible to offload your attachments

Store your attachments in the cloud efficiently. Using this addon you can save bandwidth and storage space by separating your heavy attachments from the rest of the forum.

Yeah, seriously, you can pay as little as $1 a month to store 100 GB of attachments that your users will happily upload as soon as you let them.

Under the hood, this addon allows you to mount internal_data/ directory (or "filesystem", since it's powered by the...
Click to expand...

Read more about this resource...
 
This looks neat - i like the idea of not streaming attachments through the server... i really don't have any private areas that get attachments (eg, mod forums) so i'm not too concerned about it.

Any thoughts about how running cloudflare in front of s3 changes anything with streaming attachments?
 
briansol said:
Any thoughts about how running cloudflare in front of s3 changes anything with streaming attachments?
Click to expand...
Yes - it depends. If you have lots of images and images only as your attachments, then CloudFlare would almost get the job done. However, it actually does not do anything because all attachments are using private Cache-Control, disallowing CloudFlare from caching them. This addon changes that - that's what the pass-through option is for.

The optimal configuration depends on how fast your cloud storage is for direct end-user access (maybe it's a CDN already and not worth caching by CloudFlare) and how frequently accessed your attachments are (if they're infrequently displayed, CloudFlare will evict your images from cache before they are ever served again). Also, if you happen to have other files that are non-cachable (not images), then CloudFlare also won't help at all.
 
Assuming that the 'manual' way was already in production, should we comment out our config file?

Code: 
//private data
$config['fsAdapters']['internal-data'] = function() use($s3)
{
   return new \League\Flysystem\AwsS3v3\AwsS3Adapter($s3(), 'bucket', 'internal_data');
};
 
getting all kinds of errors...

needs more testing/instruction for those of us who did it manually.


old images don't work now.

Code: 
An exception occurred: [League\Flysystem\FileNotFoundException] File not found at path: attachments/26/26020-73050a00acd23f866f08e88aa1138171.data in src/vendor/league/flysystem/src/Filesystem.php on line 389

[LIST=1]
[*]League\Flysystem\Filesystem->assertPresent() in src/vendor/league/flysystem/src/Filesystem.php at line 194
[*]League\Flysystem\Filesystem->readStream()
[*]call_user_func_array() in src/vendor/league/flysystem-eventable-filesystem/src/EventableFilesystem.php at line 431
[*]League\Flysystem\EventableFilesystem\EventableFilesystem->callFilesystemMethod() in src/vendor/league/flysystem-eventable-filesystem/src/EventableFilesystem.php at line 395
[*]League\Flysystem\EventableFilesystem\EventableFilesystem->delegateMethodCall() in src/vendor/league/flysystem-eventable-filesystem/src/EventableFilesystem.php at line 154
[*]League\Flysystem\EventableFilesystem\EventableFilesystem->readStream() in src/vendor/league/flysystem/src/MountManager.php at line 345
[*]League\Flysystem\MountManager->readStream() in src/addons/SwiftAttachmentStore/XF/Pub/View/Attachment/View.php at line 49
[*]SwiftAttachmentStore\XF\Pub\View\Attachment\View->renderRaw() in src/XF/Mvc/Renderer/AbstractRenderer.php at line 91
[*]XF\Mvc\Renderer\AbstractRenderer->renderViewObject() in src/XF/Mvc/Renderer/Raw.php at line 41
[*]XF\Mvc\Renderer\Raw->renderView() in src/XF/Mvc/Dispatcher.php at line 458
[*]XF\Mvc\Dispatcher->renderView() in src/XF/Mvc/Dispatcher.php at line 440
[*]XF\Mvc\Dispatcher->renderReply() in src/XF/Mvc/Dispatcher.php at line 400
[*]XF\Mvc\Dispatcher->render() in src/XF/Mvc/Dispatcher.php at line 58
[*]XF\Mvc\Dispatcher->run() in src/XF/App.php at line 2300
[*]XF\App->run() in src/XF.php at line 488
[*]XF::runApp() in index.php at line 20
[/LIST]
 
If I'm currenlty using the original solution with an existent object storage, how should I install the product? I have already deleted the attachment files from /internal_data and everything is currently stored in DO Spaces.
 
briansol said:
Assuming that the 'manual' way was already in production, should we comment out our config file?

Code: 
//private data
$config['fsAdapters']['internal-data'] = function() use($s3)
{
   return new \League\Flysystem\AwsS3v3\AwsS3Adapter($s3(), 'bucket', 'internal_data');
};
Click to expand...
You can comment it out but I believe the addon's overrides will take precedence and it won't matter what you have in config for internal-data adapter.

briansol said:
041 installs now, but it shows as 040
Click to expand...
That unfortunately happened to me with some other addons when first installation failed. You can check that the version_string in addon.json in the archive is showing 0.4.1, so I suspect some bug with XenForo.

JulianD said:
If I'm currenlty using the original solution with an existent object storage, how should I install the product? I have already deleted the attachment files from /internal_data and everything is currently stored in DO Spaces.
Click to expand...
Good question! Since you already have everything in S3, just configure the addon for S3. Simply installing it won't change anything, in its default configuration the internal-data adapter is not changed, so you won't break the attachments.

In case you used a snippet like this:
PHP: 
$config['fsAdapters']['internal-data'] = function() use($s3)
{
   return new \League\Flysystem\AwsS3v3\AwsS3Adapter($s3(), 'bucket', 'internal_data');
};

You should configure the Path Prefix to be internal_data. That's what I believe caused @briansol's "File not found at path" exceptions.
 
I must have a setting issue.

with the add on enabled and config commented:
new uploads load / old images do not

with the add on off and config on:
new uploads are not found / old images load

from an s3 perspective, i can see both files in the same bucket ad using same keys... so i don't know why it doesn't work both ways
 
briansol said:
with the add on enabled and config commented:
new uploads load / old images do not

with the add on off and config on:
new uploads are not found / old images load
Click to expand...
That's quite odd. It would suggest that the path/directory structure is somehow different in the container depending on which method is used. I don't have an installation where the "official guide" method was used. Would you mind sending me (by PM) a listing of objects in your container, including attachments uploaded using addon and previous method? I'm sure there's some reason it works as you described.
 
Ok i figured out my issue. My cdn url had data/ baked into it, so i was actually going to data/internal_data incorrectly when pulling the image.

I think it's working correctly now, but now i'm not sure how to test the improvement/ensure it's actually doing what it's supposed to do?
 
everything seems to be working but i'm logging errors like crazy.

Code: 
Server error log
[LIST]
[*]ErrorException: [E_WARNING] call_user_func_array() expects parameter 1 to be a valid callback, class 'XF\LocalFsAdapter' does not have a method 'getClient'
[*]src/addons/SwiftAttachmentStore/Flysystem/PathBasedAdapter.php:52
[*]Generated by: Unknown account
[*]Jan 31, 2021 at 11:36 AM
[/LIST]
[HEADING=2]Stack trace[/HEADING]
#0 [internal function]: XF::handlePhpError(2, '[E_WARNING] cal...', '/home/xxxxx/pub...', 52, Array)
#1 src/addons/SwiftAttachmentStore/Flysystem/PathBasedAdapter.php(52): call_user_func_array(Array, Array)
#2 src/addons/SwiftAttachmentStore/Flysystem/PluginAdapters/S3UrlAdapter.php(45): SwiftAttachmentStore\Flysystem\PathBasedAdapter->__call('getClient', Array)
#3 src/addons/SwiftAttachmentStore/Flysystem/GetTemporaryUrlPlugin.php(70): SwiftAttachmentStore\Flysystem\PluginAdapters\S3UrlAdapter->getTemporaryUrl('attachments/9/9...', Array)
#4 [internal function]: SwiftAttachmentStore\Flysystem\GetTemporaryUrlPlugin->handle('attachments/9/9...', Array)
#5 src/vendor/league/flysystem/src/Plugin/PluggableTrait.php(72): call_user_func_array(Array, Array)
#6 src/vendor/league/flysystem/src/Plugin/PluggableTrait.php(88): League\Flysystem\Filesystem->invokePlugin('getTemporaryUrl', Array, Object(League\Flysystem\EventableFilesystem\EventableFilesystem))
#7 [internal function]: League\Flysystem\Filesystem->__call('getTemporaryUrl', Array)
#8 src/vendor/league/flysystem/src/MountManager.php(283): call_user_func_array(Array, Array)
#9 src/vendor/league/flysystem/src/MountManager.php(168): League\Flysystem\MountManager->invokePluginOnFilesystem('getTemporaryUrl', Array, 'internal-data')
#10 src/addons/SwiftAttachmentStore/Helpers.php(206): League\Flysystem\MountManager->__call('getTemporaryUrl', Array)
#11 src/addons/SwiftAttachmentStore/XF/Pub/View/Attachment/View.php(34): SwiftAttachmentStore\Helpers::url('internal-data:/...', Array)
#12 src/XF/Mvc/Renderer/AbstractRenderer.php(91): SwiftAttachmentStore\XF\Pub\View\Attachment\View->renderRaw()
#13 src/XF/Mvc/Renderer/Raw.php(41): XF\Mvc\Renderer\AbstractRenderer->renderViewObject('XF:Attachment\\V...', '', Array)
#14 src/XF/Mvc/Dispatcher.php(458): XF\Mvc\Renderer\Raw->renderView('XF:Attachment\\V...', '', Array)
#15 src/XF/Mvc/Dispatcher.php(440): XF\Mvc\Dispatcher->renderView(Object(XF\Mvc\Renderer\Raw), Object(XF\Mvc\Reply\View))
#16 src/XF/Mvc/Dispatcher.php(400): XF\Mvc\Dispatcher->renderReply(Object(XF\Mvc\Renderer\Raw), Object(XF\Mvc\Reply\View))
#17 src/XF/Mvc/Dispatcher.php(58): XF\Mvc\Dispatcher->render(Object(XF\Mvc\Reply\View), 'raw')
#18 src/XF/App.php(2326): XF\Mvc\Dispatcher->run()
#19 src/XF.php(488): XF\App->run()
#20 index.php(20): XF::runApp('XF\\Pub\\App')
#21 {main}
[HEADING=2]Request state[/HEADING]
array(4) {
  ["url"] => string(36) "/attachments/dxb18bwiring-pdf.12192/"
  ["referrer"] => bool(false)
  ["_GET"] => array(0) {
  }
  ["_POST"] => array(0) {
  }
}
 
actually, the pattern here is that they are all pdf's trying to load causing the errors.
i added pdf to the list on Pass-through Extensions and no difference, still throwing errors.

edit, mp3's throwing the issue too.

seems to not work for non-image attachments.

disabling once again
 
Problem: Images embedded as attachments are specifically never cached by the browser.
Click to expand...
This is not correct. Images embedded as attachments are being cached by the browser, but they might should be revalidated upon request.

img-revalidate.PNG

Problem: internal_data/ is used not just for attachments but also for... other internal data. Specifically, there is code_cache/,
Click to expand...
This is not fully correct. code_cache can (and in case of cloud-storage "must") be configured indepentently.
 
Is anyone using this and it works well? Looking for a remote storage option that is easy to get going.
 
doesn't work at all for anything non-image in my testing. pdfs, etc don't work.

just use the official s3 plugin and the config file change. it's easy.
setting up the s3 bucket/perms is the hard part
 
You must log in or register to reply here.

Similar threads

frm
  • Question Question
XenForo with Vultr Object Storage
Replies
4
Views
153
frm
frm
Fullmental
Object Storage/S3 users, how is your I/O performance?
Replies
14
Views
2K
eva2000
eva2000
Back
Top Bottom