[DigitalPoint] App for Cloudflare®

[DigitalPoint] App for Cloudflare® 1.8.2

No permission to download
Overview FAQ Updates (51) Reviews (18) History Discussion
just the throttling errors. i tried uploading a fresh image. preview does not appear. but full attachment does load and appear just fine. probably shouldn't use this considering the throttling issues i continue to face on cloudflare.
 
Chromaniac said:
just the throttling errors. i tried uploading a fresh image. preview does not appear. but full attachment does load and appear just fine. probably shouldn't use this considering the throttling issues i continue to face on cloudflare.
Click to expand...
Well according to Cloudflare, R2 endpoints don't have any API throttling. No idea why they are throttling you for API calls, but R2 calls are supposedly not throttled at all (I checked with them on that). Which makes sense since they are more or less billable events.

1672945974208.webp

TL;DR

The limitation right now with R2 is going to be if you approach a billion class B operations per day.
 
Chromaniac said:
a direct link to attachment like this publicly accessible is also not working. my data folder is currently being copied so waiting for that to finish.
Click to expand...
Can you post a screenshot of your bucket with some of the file paths? Make sure they are similar to the screenshot I posted. Specifically, the path/filenames need to be similar (like no extra characters or leading slashes). Digging through XenForo's code, the error you see is when it can't find the object by the path XenForo gives.

The R2 page in XenForo admin has clickable operation numbers. If you go to the bucket in question, click the number for class B operations (reads), it will be a log from Cloudflare about the underlying API calls. If you see userError/404 it should show you the path being requested for an object when Cloudflare/R2 is saying it doesn't exist. That might give some insight as to what's going on.
 
Does anyone have any tips in regards to moving large amounts of data to R2? I have over 120GB that will need to be moved.
 
rclone seems to be the go-to tool (it's free). I haven't used it myself, so I can't be of much assistance with it, but seems to work well for everyone I've talked to when doing it. A lot of the people testing the add-on before it was released used it to move data into R2 quickly.

Rclone

Rclone syncs your files to cloud storage: Google Drive, S3, Swift, Dropbox, Google Cloud Storage, Azure, Box and many more.
rclone.org rclone.org
 
AndrewSimm said:
Does anyone have any tips in regards to moving large amounts of data to R2? I have over 120GB that will need to be moved.
Click to expand...

Use rclone copy command to copy all the files over, then run it again once you turn the option on in the add-on.

It works like rsync and will only copy over new files the second time. That will help minimise down time of images not working.
 
graham_w said:
Use rclone copy command to copy all the files over, then run it again once you turn the option on in the add-on.

It works like rsync and will only copy over new files the second time. That will help minimise down time of images not working.
Click to expand...
Once I copy the images over and turn on the add-on can I remove the images from my server or do they ned to remain?
 
You can remove them (they aren't used for anything at that point). It makes sense to do some sanity checks on that much data though... maybe keep them there for a bit while you do some spot checking that everything moved and the expected number of objects/files are in the respective buckets. Basically don't click the button to move all the data, delete all the source data and then check to see if it all moved okay (wrong order). haha

But once you are happy with how it looks, the source data/files aren't used for anything whatsoever.
 
One other thing with rclone, use the --transfers option to speed it up.

I'm not in front of my machine at the moment but I can post the command I used to copy my data over which may help.
 
digitalpoint said:
You can remove them (they aren't used for anything at that point). It makes sense to do some sanity checks on that much data though... maybe keep them there for a bit while you do some spot checking that everything moved and the expected number of objects/files are in the respective buckets. Basically don't click the button to move all the data, delete all the source data and then check to see if it all moved okay (wrong order). haha

But once you are happy with how it looks, the source data/files aren't used for anything whatsoever.
Click to expand...

I'm still checking things over a week later .... Too scared to delete the old files yet 🤣
 
I could tell you how to programmatically generate an Access Key ID and Secret Access Key from your existing token, but the easier thing is to go into Cloudflare -> R2, and in the upper right there's a Manage R2 API Tokens link there to generate yourself some for this (can delete them when you are done).

When you go in there, ignore your existing tokens (don't try to generate one for an existing token), instead just do Create API Token. I ran into an issue where I tried to generate a R2 token for an existing API token and it wiped out all the permissions on my existing token. So just keep them separate for this.
 
digitalpoint said:
I could tell you how to programmatically generate an Access Key ID and Secret Access Key from your existing token, but the easier thing is to go into Cloudflare -> R2, and in the upper right there's a Manage R2 API Tokens link there to generate yourself some for this (can delete them when you are done).

When you go in there, ignore your existing tokens (don't try to generate one for an existing token), instead just do Create API Token. I ran into an issue where I tried to generate a R2 token for an existing API token and it wiped out all the permissions on my existing token. So just keep them separate for this.
Click to expand...
Thanks, looks like rclone is working.
 
Looks like this is going to take a hot minute to complete.

Will new attachments only write to r2 once I turn this on?
 
AndrewSimm said:
Looks like this is going to take a hot minute to complete.

Will new attachments only write to r2 once I turn this on?
Click to expand...
Correct. You have to enable (and pick which bucket). Until it’s enabled, it will just work like it’s always worked (local file system or whatever you are using as your file system adapter).

And ya… 120GB is a lot of data to move around. Your bottleneck is most likely going to be the network connectivity of your server.
 
digitalpoint said:
Correct. You have to enable (and pick which bucket). Until it’s enabled, it will just work like it’s always worked (local file system or whatever you are using as your file system adapter).

And ya… 120GB is a lot of data to move around. Your bottleneck is most likely going to be the network connectivity of your server.
Click to expand...
Easy enough! Before I started copying I tested the connections for a few minutes and some attachments saved to r2. Sounds like I should be live with this tomorrow. This was perfect timing as I was filling up my storage and was looking at using S3 this weekend. My backups are going to be a lot smaller in another week or so!
 
I had a thought of making it so you didn’t even need to move files to R2 (it would do it behind the scenes transparently). Like when you read a file, check if it’s in R2, if it’s not read it from the old adapter (local server usually) and upload it to R2 at that point and then serve it normally…

But then I came to my senses and was like, “That would be cool, but also a lot of work for something I don’t need myself and in an addon that’s free. Probably not the best use of my time.” Hah
 
Upgraded to this.. and MANY of my image links (like in sig and then even uploading images) had issues. I do NOT use R2 functions.. but apparently once you install the latest version of the add-on it expects that you do? Removing the add-on resolves the issue.. but then after installing 1.4 you apparently cannot regress back to 1.3 as a new install after removing 1.4. But then doing a refresh allowed me to re-install it.
So there is apparently an issue present.

Code: 
[*]ErrorException: Failed to write files for DigitalPoint/Cloudflare action, including src/addons/DigitalPoint/Cloudflare/Setup.php
[*]src/XF/Error.php:77
[*]Generated by: Tracy
[*]January 6, 2023 at 5:34 AM

Stack trace
#0 src/XF.php(219): XF\Error->logError('Failed to write...', false)
#1 src/XF/Job/AddOnInstallBatch.php(158): XF::logError('Failed to write...')
#2 src/XF/Job/AddOnInstallBatch.php(79): XF\Job\AddOnInstallBatch->stepInit(Object(XF\Timer))
#3 src/XF/Job/Manager.php(260): XF\Job\AddOnInstallBatch->run(8)
#4 src/XF/Job/Manager.php(202): XF\Job\Manager->runJobInternal(Array, 8)
#5 src/XF/Job/Manager.php(118): XF\Job\Manager->runJobEntry(Array, 8)
#6 src/XF/Admin/Controller/Tools.php(122): XF\Job\Manager->runByIds(Array, 8)
#7 src/XF/Mvc/Dispatcher.php(352): XF\Admin\Controller\Tools->actionRunJob(Object(XF\Mvc\ParameterBag))
#8 src/XF/Mvc/Dispatcher.php(259): XF\Mvc\Dispatcher->dispatchClass('XF:Tools', 'RunJob', Object(XF\Mvc\RouteMatch), Object(SV\UserMentionsImprovements\XF\Admin\Controller\Tools), NULL)
#9 src/XF/Mvc/Dispatcher.php(115): XF\Mvc\Dispatcher->dispatchFromMatch(Object(XF\Mvc\RouteMatch), Object(SV\UserMentionsImprovements\XF\Admin\Controller\Tools), NULL)
#10 src/XF/Mvc/Dispatcher.php(57): XF\Mvc\Dispatcher->dispatchLoop(Object(XF\Mvc\RouteMatch))
#11 src/XF/App.php(2483): XF\Mvc\Dispatcher->run()
#12 src/XF.php(524): XF\App->run()
#13 admin.php(13): XF::runApp('XF\\Admin\\App')
#14 {main}
Request state
array(4) {
  ["url"] => string(24) "/admin.php?tools/run-job"
  ["referrer"] => string(153) "https://astrowhat.com/admin.php?tools/run-job&only=addOnInstallBatch44&_xfRedirect=%2Fadmin.php%3Fadd-ons%2Finstall-from-archive-complete%26batch_id%3D44"
  ["_GET"] => array(1) {
    ["tools/run-job"] => string(0) ""
  }
  ["_POST"] => array(3) {
    ["_xfRedirect"] => string(81) "https://astrowhat.com/admin.php?add-ons/install-from-archive-complete&batch_id=44"
    ["_xfToken"] => string(8) "********"
    ["only_ids"] => string(5) "39317"
 
Tracy Perry said:
Upgraded to this..
Click to expand...
IMPORTANT for existing users: The new R2 functions and control of new settings require some new permissions for the API Token you use. You can go to your Cloudflare API Tokens, edit the token you have and add the following permissions:
  • Account.Account Analytics: Read
  • Account.Workers R2 Storage: Edit
  • Zone.Bot Management: Edit
  • Zone.Cache Rules: Edit
You should have a total of 14 permissions for your API token at this point. If you don't have 14, you can check what you should have under XF Admin -> Options -> External service providers -> Cloudflare authentication
 
You must log in or register to reply here.

Similar threads

N
[DigitalPoint] App for Cloudflare® - FRENCH translation
Replies
8
Views
822
Ozzy47
Ozzy47
digitalpoint
App for Cloudflare (appforcf.com)
Replies
12
Views
1K
securedme
securedme
Top Bottom