![[DigitalPoint] App for Cloudflare®](/community/data/resource_icons/8/8750.jpg?1692633961){kind=icon}
[DigitalPoint] App for Cloudflare® 1.9.1.1
- Thread starter digitalpoint
- Start date
digitalpoint
Well-known member
It already does that.
Ivancas
Well-known member
During the R2 bucket creation for data bucket I got this error:
The bucket was created but not the record pointing to the R2 bucket. The API Token was created using the addon. There is no error at admin.php?logs/server-errors/ so I can't troubleshoot.
Code:
Server error: `POST https://api.cloudflare.com/client/v4/accounts/dcdd1aa52d8172728a5085d084214ab8/r2/buckets/data/custom_domains` resulted in a `500 Internal Server Error` response: {"success":false,"errors":[{"code":10001,"message":"We encountered an internal error. Please try again."}],"messages":[] (truncated...)The bucket was created but not the record pointing to the R2 bucket. The API Token was created using the addon. There is no error at admin.php?logs/server-errors/ so I can't troubleshoot.
digitalpoint
Well-known member
You can try again, but nothing we can do on our end with 5xx errors. 5xx means it’s a server-side error and changing the request wouldn’t help. 4xx errors are client-side (meaning we can change something to change the outcome). Maybe Cloudflare was having issues on their end?
digitalpoint
Well-known member
FWIW, I went and tried it (creating an R2 bucket, assigning domain and all that) from within the addon and it's working as expected for me. So maybe whatever issue was going on with Cloudflare has since resolved itself?
Ivancas
Well-known member
Yup, seems to be a temporary error. Nothing on the status page. I'll just manually create the record.
digitalpoint
Well-known member
Ya... like I said, not really much we can do about 5xx errors beyond what we are already doing (if a 5xx error happens, the addon will automatically retry the request). Usually that's enough to "fix" whatever it is since it's something on Cloudflare's side and a second try will usually work. So in your case the 2nd try also failed with a 5xx so it logged the error. Beyond that, not much we can do since it's Cloudflare's side (can't realistically just loop and try over and over and over until it finally works... not sure Cloudlfare would appreciate 10,000 API calls happening per second if they are actually having an issue).
ekool
Well-known member
I'm setting up R2 for the first time. I'm trying to move 5GB of attachments to R2. It started out at 11MB/s transferring. I'm at 35% and it's now down to 500KB/s -- and slowly dropping. Is this normal? At this rate it's going to take a lot longer... it's estimating 50 minutes but that will increase as the speed continues to drop.
If I stop the php cmd.php migrate -- will it know where to pick up if I restart it?
What causes it to slow down? The site is running fine, there is no load on the server.
Down to this speed now.
Edit: Well, that answers that question. I stopped it and restarted it, and it starts at the very beginning and still appears to be throttled. I don't know how a large site can do this if it can't pick up where it left off.
If you can't specify where to restart the CLI import seems a bit limited. If anything happens you'll have to nuke the bucket and all of its contents, then start again from scratch every time and pray that nothing happens or that you don't get throttled like it seems like I am?
If I stop the php cmd.php migrate -- will it know where to pick up if I restart it?
What causes it to slow down? The site is running fine, there is no load on the server.
Code:
9654/26834 [==========>-----------------] 35% 50 mins 485.3 KB/sDown to this speed now.
Code:
10458/26834 [==========>-----------------] 38% 1 hr 419.7 KB/sEdit: Well, that answers that question. I stopped it and restarted it, and it starts at the very beginning and still appears to be throttled. I don't know how a large site can do this if it can't pick up where it left off.
Code:
70/26834 [>---------------------------] 0% 1 min 110.8 KB/sIf you can't specify where to restart the CLI import seems a bit limited. If anything happens you'll have to nuke the bucket and all of its contents, then start again from scratch every time and pray that nothing happens or that you don't get throttled like it seems like I am?
Last edited:
digitalpoint
Well-known member
Have you tried the
--processes option?
ekool
Well-known member
I just restarted it with --processes 8 and its hovering around 460KB/s... we will see if that is consistent. Still quite a bit off from 10/11MB/s that I got with 1 process before it started to throttle down after about 10 minutes.
What is the recommended processes option?
I have a site that has over 60GB of attachments, I can't imagine waiting for that to finish at 500KB/s?
digitalpoint
Well-known member
Is it possible that your server doesn’t have enough available resources? Disk read speed, CPU cycles or just bandwidth available?
500KB/sec seems incredibly slow. When I ran my first test of the CLI migration tool it ran at 9.1MB/sec:
https://xenforo.com/community/threads/digitalpoint-app-for-cloudflare-r.206176/post-1616106
It’s never going to be insanely fast (there’s more to it than just pushing bytes around), but that’s still about 20x faster than what you are seeing.
500KB/sec seems incredibly slow. When I ran my first test of the CLI migration tool it ran at 9.1MB/sec:
https://xenforo.com/community/threads/digitalpoint-app-for-cloudflare-r.206176/post-1616106
It’s never going to be insanely fast (there’s more to it than just pushing bytes around), but that’s still about 20x faster than what you are seeing.
ekool
Well-known member
I did a couple of rsync's off this server earlier and got way higher speeds... the load isn't very high and there is no iowait going on.
Code:
22:19:38 up 863 days, 11:19, 3 users, load average: 2.57, 1.97, 1.67
Code:
avg-cpu: %user %nice %system %iowait %steal %idle
6.86 0.00 1.56 1.19 0.04 90.35
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 155.53 2334.83 543.45 174187669533 40543325800
sdd 1.69 202.26 97.48 15089514045 7272284952
sdb 0.05 0.09 0.23 6449572 17104008It's just weird that it started blazing and just throttled down. This is a Linode VM so perhaps they did some type of rate limiting behind the scenes. I didn't get any notices and there is nothing in the Linode CP alerting me to passing any limits or any throttling going on but that doesn't mean anything.
I will continue to investigate. Thank you for your assistance!
digitalpoint
Well-known member
Ya, does seem odd. There’s definitely nothing on this end that would cause it to throttle back as time progressed. Might want to play around with rclone and see if you see the same thing happening.
ekool
Well-known member
Since getting this going, I get this when browsing the Cloudflare settings in the AdminCP...
And this if I try to go to the CF Settings page in the Admin CP
Edit: I'm running the migrator again this time on the data directory. I see it won't follow symlinks
That's a real bummer LOL. I've got so much of this data symlinked to block storage or other drives.
Code:
971: Please wait and consider throttling your request speedAnd this if I try to go to the CF Settings page in the Admin CP
Code:
ErrorException: [E_WARNING] Trying to access array offset on value of type bool in src/addons/DigitalPoint/Cloudflare/Repository/Cloudflare.php at line 1081
[LIST=1]
[*]XF::handlePhpError() in src/addons/DigitalPoint/Cloudflare/Repository/Cloudflare.php at line 1081
[*]DigitalPoint\Cloudflare\Repository\CloudflareAbstract->getZoneSettings() in src/addons/DigitalPoint/Cloudflare/Repository/Cloudflare.php at line 787
[*]DigitalPoint\Cloudflare\Repository\CloudflareAbstract->organizeSettings() in src/addons/DigitalPoint/Cloudflare/Admin/Controller/Cloudflare.php at line 49
[*]DigitalPoint\Cloudflare\Admin\Controller\Cloudflare->actionIndex() in src/XF/Mvc/Dispatcher.php at line 352
[*]XF\Mvc\Dispatcher->dispatchClass() in src/XF/Mvc/Dispatcher.php at line 258
[*]XF\Mvc\Dispatcher->dispatchFromMatch() in src/XF/Mvc/Dispatcher.php at line 115
[*]XF\Mvc\Dispatcher->dispatchLoop() in src/XF/Mvc/Dispatcher.php at line 57
[*]XF\Mvc\Dispatcher->run() in src/XF/App.php at line 2485
[*]XF\App->run() in src/XF.php at line 524
[*]XF::runApp() in admin.php at line 13
[/LIST]Edit: I'm running the migrator again this time on the data directory. I see it won't follow symlinks
That's a real bummer LOL. I've got so much of this data symlinked to block storage or other drives.
Last edited:
digitalpoint
Well-known member
That's an API rate limit (for general API stuff, not R2). Cloudflare allows 1200 API requests per rolling 5 minute period. Normally that's going to be plenty, but if you are reloading the settings page over and over very quickly, you could get it I guess (there's 13 API requests made to read all the settings), so viewing the settings (as well as changing settings) takes 13 per instance. So if you make around 80 Cloudflare settings changes in a short period of time (5 minutes), you will need to wait a bit.Since getting this going, I get this when browsing the Cloudflare settings in the AdminCP...
Code:971: Please wait and consider throttling your request speed
And this if I try to go to the CF Settings page in the Admin CP
Code:ErrorException: [E_WARNING] Trying to access array offset on value of type bool in src/addons/DigitalPoint/Cloudflare/Repository/Cloudflare.php at line 1081 [LIST=1] [*]XF::handlePhpError() in src/addons/DigitalPoint/Cloudflare/Repository/Cloudflare.php at line 1081 [*]DigitalPoint\Cloudflare\Repository\CloudflareAbstract->getZoneSettings() in src/addons/DigitalPoint/Cloudflare/Repository/Cloudflare.php at line 787 [*]DigitalPoint\Cloudflare\Repository\CloudflareAbstract->organizeSettings() in src/addons/DigitalPoint/Cloudflare/Admin/Controller/Cloudflare.php at line 49 [*]DigitalPoint\Cloudflare\Admin\Controller\Cloudflare->actionIndex() in src/XF/Mvc/Dispatcher.php at line 352 [*]XF\Mvc\Dispatcher->dispatchClass() in src/XF/Mvc/Dispatcher.php at line 258 [*]XF\Mvc\Dispatcher->dispatchFromMatch() in src/XF/Mvc/Dispatcher.php at line 115 [*]XF\Mvc\Dispatcher->dispatchLoop() in src/XF/Mvc/Dispatcher.php at line 57 [*]XF\Mvc\Dispatcher->run() in src/XF/App.php at line 2485 [*]XF\App->run() in src/XF.php at line 524 [*]XF::runApp() in admin.php at line 13 [/LIST]
Edit: I'm running the migrator again this time on the data directory. I see it won't follow symlinks
It should follow symlinks... all it's doing to read is using XenForo's abstracted filesystem, so if XenForo can read them (however they are setup... symlinks or otherwise), it should work.
Does it work as expected if you disable R2? The only thing being used underneath it all is XenForo's abstracted filesystem, so if something isn't landing in R2, I'd suspect XenForo is rejecting it for whatever reason and not trying to send it. Maybe an invalid image or something? Either way, check with R2 disabled... if it's not working with R2 disabled, it's definitely not going to work with it enabled.
ekool
Well-known member
It gave me this error when there were symlinks... when I moved them it worked fine.
Code:
In NotSupportedException.php line 21:
Links are not supported, encountered link at /home/nginx/domains/XXX.net/public/data/ph
otosIs their API rate limit per "account" -- as we have quite a few Cloudflare sites under this one account. Probably about 25... and most of them have your plugin on it which also displays the stats in the Admin CP. We don't access the Admin CP on all the sites often or anything like that but there are definitely lots of sites on this Cloudflare account.
Edit: I'm also assuming that uploading a lot of these files while doing a migration eats up API calls?
digitalpoint
Well-known member
The error is coming from Flysystem (the abstracted filesystem that XenForo uses). Can't tell tell much else without the full error backtrace though.
Yes, it's a rate limit per account (which will include calls made with API tokens for the account as well as normal Cloudflare dashboard usage). It does not include R2 API calls (it's primarily reading/writing settings, purging cache, etc.)
If you are hitting the API limit, you will also not be able to use Cloudflare's normal dashboard either. That being said, 1200 per rolling 300 second window normally is plenty. One thing that could run up a lot of API calls is if you are using Guest Page Caching that this addon can do and you are proactively purging the parts of the Cloudflare cache whenever a post is created or deleted. If you are getting a TON of posts across all sites, you might want to disable the
Purge cache when post is created or deleted option. The only thing it will affect is guests (non-registered users) and how fast they would see new posts. Probably not going to matter much at the end of the day since you are only talking about guest users anyway.
