[DigitalPoint] App for Cloudflare®

[DigitalPoint] App for Cloudflare® 1.8.2

No permission to download
If you don't have an internal_data/xfmg folder, you don't need to do anything.

Now I'm wondering if that new option is even needed if XFMG doesn't store anything in internal_data. Honestly I was just going off what some users told me because I don't use XFMG or have a license for it myself.

Although this post implies it should be there (maybe):


This one too:


Either way, if YOU don't have anything in internal_data/xfmg, nothing you need to move. Definitely don't move data to internal_data.
Yeah man.. listen I have nothing in internal_data/xfmg and I use it.. theres no folder for it.. I think this may be a mistake

I'm not sure how some people have something in internal_data but I do not.. I'm in the process of rcopying it back to the right PUBLIC data bucket, because it does look like I broke something. Fortunately its only a couple MB of images.
 
Yeah man.. listen I have nothing in internal_data/xfmg and I use it.. theres no folder for it.. I think this may be a mistake

I'm not sure how some people have something in internal_data but I do not.. I'm in the process of rcopying it back to the right PUBLIC data bucket, because it does look like I broke something. Fortunately its only a couple MB of images.
Ya, I pinged @Chris D on it. I assume he should know definitively. Maybe it's just used sometimes somehow? Want to ask before I gut the feature out (and then find out later it is needed). haha
 
@Chris D said internal_data/xfmg is a valid location, but it's not always used. Something about it keeping unwatermarked originals there.

So the location is valid, but also not abnormal if an install isn't using it.
 
Ya, was thinking about adding an “Easy Mode” button that does some auto config for people.
So I ended up implementing this for the next version. It's a one-click "Easy config" button that sets 22 different Cloudflare settings based on what I've found to work well for XenForo sites. I'm sure some people will argue about which settings it should include, but also the people that understand individual Cloudflare settings enough to even be making those arguments probably shouldn't be "Easy mode" configuring their sites anyway. :)

1674619245764.webp

You can still set any individual settings like you can already.
 
digitalpoint updated [DigitalPoint] App for Cloudflare® with a new update entry:

"Easy config" option and some additional handling of some "what ifs"

  • Added "Easy config" button to settings page to automatically optimize some settings for XenForo (currently it sets 22 Cloudflare settings in one go that work well for XenForo)
  • Don't use getContentUrl() method since it's only in XenForo 2
  • Force path and type keys when getting Metadata via R2 API
  • Better handling of non-existent R2 objects trying to be read by internal processes (throw FileNotFoundException)
View attachment 280531

Read the rest of this update entry...
 
I don't use watermarks, so maybe thats why.
It doesn't hurt to just set it to use the same bucket that you are using for attachments (both are in the internal-data abstracted filesystem) in case you do ever end up with files there or a future version of XFMG uses it for something new. But up to you...
 
For those affected by Cloudflare's API issue yesterday, one thing that I do like about Cloudflare is that when they do have issues, they disclose, document and correct. Which is a way better way of handling issues vs. some other companies that pretend nothing happened.

In case anyone is curious about what happened on their end and what they are doing to make sure it doesn't happen again:

 
Is it possible to defer or chunk-upload a large file to CF R2?
I'm hitting the 100-second timeout limit imposed by CF when testing 2GB+ files.
 
Does this add-on, with the buckets created, keep attachments in hidden/private forums private to those navigating those forums? Or can those attachments be accessed by the public?
 
What specific Nginx configuration do I need to configure in order for the Cloudflare backup feature to work?
Right now, it only downloads admin.php as a blank file.

Thank you.
 
Is it possible to defer or chunk-upload a large file to CF R2?
I'm hitting the 100-second timeout limit imposed by CF when testing 2GB+ files.
Not sure how you are testing, but here's some answers for different ways...

If you are testing outside of XenForo (just uploading a file on your own), yes... you can upload an individual file/object that is up to 4.995 TB, with the maximum size per chunk being 5 GB (see the exact limits here).

If you are uploading via an HTTP connection (like a XenForo attachment), and using this addon, you will be limited to 5 GB because the addon isn't doing chunked uploads to R2. If you use Cloudflare as a proxy to your site, you will also be limited to 100 seconds of backend processing time for anything (has nothing to do with R2) because that's simply the time limit Cloudflare has for HTTP requests to website while working as a proxy.

Honestly, I'm not sure the default way that XenForo manages attachments is going to be the best method for massive files like that. XenForo reads attachments and they passes them through because of the way the permission system works so that's not really efficient for massive files. It also expects to be able to return something in a reasonable amount of time after the file is uploaded (AJAX stuff showing the attachment info after it's processed).

You might want to consider using a different upload/download mechanism for files that are so large because XenForo isn't really optimized for that (regardless of the backend storage system you are wanting to use).

Does this add-on, with the buckets created, keep attachments in hidden/private forums private to those navigating those forums? Or can those attachments be accessed by the public?
It's a backend data store so it will adhere to the same permission system XenForo uses for everything. If your setup has permissions setup for private forums it will continue to work that way. It's not a system where attachments are thrown some place public and then a link is given that anyone can use. Everything still passes through XenForo's permission system. The only real change is where XenForo is retrieving data when it needs to pass something along to a user based on their permissions.

What specific Nginx configuration do I need to configure in order for the Cloudflare backup feature to work?
Right now, it only downloads admin.php as a blank file.

Thank you.
You don't need any special config beyond the standard config. Not even sure what could be messed up if you have XenForo working. Maybe you are trying to treat requests differently based on their mime type or file extension with Nginx and that's mucking it? Basically if you have anything beyond the standard config, look at that maybe?


If you are using Cloudflare, you might want to try testing it while not in proxy mode (orange cloud). Maybe you have some Cloudflare config setting somewhere on your Cloudflare account that is affecting it. A good place to check considering Cloudflare is literally processing and manipulating HTTP requests in realtime, so maybe it's something you are doing at the network level rather than the web server level?
 
Testing a small video file produce this error.
PHP:
Server error log
League\Flysystem\FileNotFoundException: File not found at path: video/0/5-6f8de3b7719e9ec3d9c5b107dcc92b9c.mp4 src/addons/DigitalPoint/Cloudflare/League/Flysystem/Adapter/R2.php:166
Generated by: user101 Jan 25, 2023 at 5:37 PM

Stack trace
#0 src/addons/DigitalPoint/Cloudflare/League/Flysystem/Adapter/R2.php(83): DigitalPoint\Cloudflare\League\Flysystem\Adapter\R2->getMetadata('video/0/5-6f8de...')
#1 src/vendor/league/flysystem/src/Filesystem.php(57): DigitalPoint\Cloudflare\League\Flysystem\Adapter\R2->has('video/0/5-6f8de...')
#2 [internal function]: League\Flysystem\Filesystem->has('video/0/5-6f8de...', Array)
#3 src/vendor/league/flysystem-eventable-filesystem/src/EventableFilesystem.php(431): call_user_func_array('League\\Flysyste...', Array)
#4 src/vendor/league/flysystem-eventable-filesystem/src/EventableFilesystem.php(395): League\Flysystem\EventableFilesystem\EventableFilesystem->callFilesystemMethod('has', Array)
#5 src/vendor/league/flysystem-eventable-filesystem/src/EventableFilesystem.php(128): League\Flysystem\EventableFilesystem\EventableFilesystem->delegateMethodCall('has', Array)
#6 src/vendor/league/flysystem/src/Filesystem.php(118): League\Flysystem\EventableFilesystem\EventableFilesystem->has('video/0/5-6f8de...')
#7 [internal function]: League\Flysystem\Filesystem->putStream('video/0/5-6f8de...', Resource id #57, Object(League\Flysystem\Config))
#8 src/vendor/league/flysystem-eventable-filesystem/src/EventableFilesystem.php(431): call_user_func_array('League\\Flysyste...', Array)
#9 src/vendor/league/flysystem-eventable-filesystem/src/EventableFilesystem.php(395): League\Flysystem\EventableFilesystem\EventableFilesystem->callFilesystemMethod('putStream', Array)
#10 src/vendor/league/flysystem-eventable-filesystem/src/EventableFilesystem.php(71): League\Flysystem\EventableFilesystem\EventableFilesystem->delegateMethodCall('putStream', Array)
#11 src/vendor/league/flysystem/src/MountManager.php(615): League\Flysystem\EventableFilesystem\EventableFilesystem->putStream('video/0/5-6f8de...', Resource id #57, Array)
#12 src/XF/Util/File.php(200): League\Flysystem\MountManager->putStream('video/0/5-6f8de...', Resource id #57, Array)
#13 src/XF/Service/Attachment/Preparer.php(78): XF\Util\File::copyFileToAbstractedPath('/var/www/html/x...', 'data://video/0/...')
#14 src/XF/Service/Attachment/Preparer.php(38): XF\Service\Attachment\Preparer->insertDataFromFile(Object(XF\FileWrapper), 1, Array)
#15 src/XF/Attachment/Manipulator.php(199): XF\Service\Attachment\Preparer->insertAttachment(Object(XFMG\Attachment\Media), Object(XF\FileWrapper), Object(XFMG\XF\Entity\User), '51a8378308b1d03...')
#16 src/XF/Pub/Controller/Attachment.php(91): XF\Attachment\Manipulator->insertAttachmentFromUpload(Object(XF\Http\Upload), NULL)
#17 src/addons/ChunkedUploads/XF/Pub/Controller/Attachment.php(48): XF\Pub\Controller\Attachment->actionUpload()
#18 src/XF/Mvc/Dispatcher.php(352): ChunkedUploads\XF\Pub\Controller\Attachment->actionUpload(Object(XF\Mvc\ParameterBag))
#19 src/XF/Mvc/Dispatcher.php(259): XF\Mvc\Dispatcher->dispatchClass('XF:Attachment', 'Upload', Object(XF\Mvc\RouteMatch), Object(ChunkedUploads\XF\Pub\Controller\Attachment), NULL)
#20 src/XF/Mvc/Dispatcher.php(115): XF\Mvc\Dispatcher->dispatchFromMatch(Object(XF\Mvc\RouteMatch), Object(ChunkedUploads\XF\Pub\Controller\Attachment), NULL)
#21 src/XF/Mvc/Dispatcher.php(57): XF\Mvc\Dispatcher->dispatchLoop(Object(XF\Mvc\RouteMatch))
#22 src/XF/App.php(2483): XF\Mvc\Dispatcher->run()
#23 src/XF.php(524): XF\App->run()
#24 index.php(20): XF::runApp('XF\\Pub\\App')
#25 {main}

Request state
array(4) {
  ["url"] => string(96) "/attachments/upload?type=xfmg_media&context[category_id]=1&hash=51a8378308b1d03f1ef8d0d4e11e2010"
  ["referrer"] => string(39) "https://xf2.com/media/categories/1/add"
  ["_GET"] => array(4) {
    ["/attachments/upload"] => string(0) ""
    ["type"] => string(10) "xfmg_media"
    ["context"] => array(1) {
      ["category_id"] => string(1) "1"
    }
    ["hash"] => string(32) "51a8378308b1d03f1ef8d0d4e11e2010"
  }
  ["_POST"] => array(11) {
    ["_xfToken"] => string(8) "********"
    ["_xfResponseType"] => string(4) "json"
    ["_xfWithData"] => string(1) "1"
    ["flowChunkNumber"] => string(1) "4"
    ["flowChunkSize"] => string(8) "10485760"
    ["flowCurrentChunkSize"] => string(8) "10485760"
    ["flowTotalSize"] => string(9) "100450390"
    ["flowIdentifier"] => string(42) "100450390-Sample-Video-File-For-Testingmp4"
    ["flowFilename"] => string(33) "Sample-Video-File-For-Testing.mp4"
    ["flowRelativePath"] => string(33) "Sample-Video-File-For-Testing.mp4"
    ["flowTotalChunks"] => string(2) "10"
  }
}
 
Are you able to replicate the error outside of XFMG? I don't have XFMG myself, so I can't really do a test doing the same thing.
 
PHP:
Server error log
League\Flysystem\FileNotFoundException: File not found at path: video/0/6-6f8de3b7719e9ec3d9c5b107dcc92b9c.mp4 src/addons/DigitalPoint/Cloudflare/League/Flysystem/Adapter/R2.php:166
Generated by: user101 Jan 25, 2023 at 5:44 PM

Stack trace
#0 src/addons/DigitalPoint/Cloudflare/League/Flysystem/Adapter/R2.php(83): DigitalPoint\Cloudflare\League\Flysystem\Adapter\R2->getMetadata('video/0/6-6f8de...')
#1 src/vendor/league/flysystem/src/Filesystem.php(57): DigitalPoint\Cloudflare\League\Flysystem\Adapter\R2->has('video/0/6-6f8de...')
#2 [internal function]: League\Flysystem\Filesystem->has('video/0/6-6f8de...', Array)
#3 src/vendor/league/flysystem-eventable-filesystem/src/EventableFilesystem.php(431): call_user_func_array('League\\Flysyste...', Array)
#4 src/vendor/league/flysystem-eventable-filesystem/src/EventableFilesystem.php(395): League\Flysystem\EventableFilesystem\EventableFilesystem->callFilesystemMethod('has', Array)
#5 src/vendor/league/flysystem-eventable-filesystem/src/EventableFilesystem.php(128): League\Flysystem\EventableFilesystem\EventableFilesystem->delegateMethodCall('has', Array)
#6 src/vendor/league/flysystem/src/Filesystem.php(118): League\Flysystem\EventableFilesystem\EventableFilesystem->has('video/0/6-6f8de...')
#7 [internal function]: League\Flysystem\Filesystem->putStream('video/0/6-6f8de...', Resource id #56, Object(League\Flysystem\Config))
#8 src/vendor/league/flysystem-eventable-filesystem/src/EventableFilesystem.php(431): call_user_func_array('League\\Flysyste...', Array)
#9 src/vendor/league/flysystem-eventable-filesystem/src/EventableFilesystem.php(395): League\Flysystem\EventableFilesystem\EventableFilesystem->callFilesystemMethod('putStream', Array)
#10 src/vendor/league/flysystem-eventable-filesystem/src/EventableFilesystem.php(71): League\Flysystem\EventableFilesystem\EventableFilesystem->delegateMethodCall('putStream', Array)
#11 src/vendor/league/flysystem/src/MountManager.php(615): League\Flysystem\EventableFilesystem\EventableFilesystem->putStream('video/0/6-6f8de...', Resource id #56, Array)
#12 src/XF/Util/File.php(200): League\Flysystem\MountManager->putStream('video/0/6-6f8de...', Resource id #56, Array)
#13 src/XF/Service/Attachment/Preparer.php(78): XF\Util\File::copyFileToAbstractedPath('/var/www/html/x...', 'data://video/0/...')
#14 src/XF/Service/Attachment/Preparer.php(38): XF\Service\Attachment\Preparer->insertDataFromFile(Object(XF\FileWrapper), 1, Array)
#15 src/XF/Attachment/Manipulator.php(199): XF\Service\Attachment\Preparer->insertAttachment(Object(XFMG\XF\Attachment\Post), Object(XF\FileWrapper), Object(XFMG\XF\Entity\User), '48ffa9444bfb5dd...')
#16 src/XF/Pub/Controller/Attachment.php(91): XF\Attachment\Manipulator->insertAttachmentFromUpload(Object(XF\Http\Upload), NULL)
#17 src/addons/ChunkedUploads/XF/Pub/Controller/Attachment.php(48): XF\Pub\Controller\Attachment->actionUpload()
#18 src/XF/Mvc/Dispatcher.php(352): ChunkedUploads\XF\Pub\Controller\Attachment->actionUpload(Object(XF\Mvc\ParameterBag))
#19 src/XF/Mvc/Dispatcher.php(259): XF\Mvc\Dispatcher->dispatchClass('XF:Attachment', 'Upload', Object(XF\Mvc\RouteMatch), Object(ChunkedUploads\XF\Pub\Controller\Attachment), NULL)
#20 src/XF/Mvc/Dispatcher.php(115): XF\Mvc\Dispatcher->dispatchFromMatch(Object(XF\Mvc\RouteMatch), Object(ChunkedUploads\XF\Pub\Controller\Attachment), NULL)
#21 src/XF/Mvc/Dispatcher.php(57): XF\Mvc\Dispatcher->dispatchLoop(Object(XF\Mvc\RouteMatch))
#22 src/XF/App.php(2483): XF\Mvc\Dispatcher->run()
#23 src/XF.php(524): XF\App->run()
#24 index.php(20): XF::runApp('XF\\Pub\\App')
#25 {main}

Request state
array(4) {
  ["url"] => string(86) "/attachments/upload?type=post&context[node_id]=2&hash=48ffa9444bfb5dd65d6589f99b7039dc"
  ["referrer"] => string(37) "https://xf2.com/forums/2/post-thread"
  ["_GET"] => array(4) {
    ["/attachments/upload"] => string(0) ""
    ["type"] => string(4) "post"
    ["context"] => array(1) {
      ["node_id"] => string(1) "2"
    }
    ["hash"] => string(32) "48ffa9444bfb5dd65d6589f99b7039dc"
  }
  ["_POST"] => array(11) {
    ["_xfToken"] => string(8) "********"
    ["_xfResponseType"] => string(4) "json"
    ["_xfWithData"] => string(1) "1"
    ["flowChunkNumber"] => string(1) "2"
    ["flowChunkSize"] => string(8) "10485760"
    ["flowCurrentChunkSize"] => string(8) "10485760"
    ["flowTotalSize"] => string(9) "100450390"
    ["flowIdentifier"] => string(42) "100450390-Sample-Video-File-For-Testingmp4"
    ["flowFilename"] => string(33) "Sample-Video-File-For-Testing.mp4"
    ["flowRelativePath"] => string(33) "Sample-Video-File-For-Testing.mp4"
    ["flowTotalChunks"] => string(2) "10"
  }
}

Testing on a thread attachment.
 
Maybe an incompatibility with @JulianD's addon:
 
Probably... if that addon is assuming you are working in the local file system (which probably is a requirement for chunked uploads to be reassembled), I could see how that's going to be an issue when it checks to see if files exist in XenForo's abstracted filesystem (which is looking in R2) when really it's in the local filesystem.

However, even if you got that worked out, I still think you are going to run into other issues trying to store multi-gigabyte files as XenForo attachments simply because how XenForo is designed (like how it makes sure the user has permissions to view the file and then reads the file into the application itself just so it can pass it through). Having the application process such huge files every time someone simply wants to download it is going to cause other problems even if you solve the chunked upload thing by not using R2.

Honestly it feels to me like you are trying to force the use of XenForo attachment system to store massive files, which it's simply not designed for. You need something different where a user can upload direct to something (in R2 you could do presigned URLs for that) and then have something other than XenForo attachments for downloading them. Then you solved the problem of 100 second http request timeouts, you aren't fully processing massive files at the application level just to check a user's permissions, etc.

XenForo attachments (with or without R2) simply isn't the right tool for what you are trying to do.
 
Top Bottom