[DigitalPoint] App for Cloudflare® 1.8.2

[NealC]

I think the R2 location for XFMG is incorrect in your add-on:

R2 for /internal-data/xfmg/

I found the xfmg folder in the DATA folder, not internal_data

As R2 has a bucket for DATA do we need one for XFMG and if so how is it rooted? Should it be xfmg/... (i.e. there is a xfmg parent folder?)

I checked my site now complete migrated to Cloudflare and you can see how the XFMG works and it appears you don't need that third bucket as it's under DATA.

 

[digitalpoint]

I'm setting up the R2 config and see that you have a bucket for XFMG but what about the XFRM?

As far as I'm aware, there is no XFRM-specific folders inside internal_data, so it's not needed (and wouldn't do anything since it's not a thing).

Confirm after R2 is setup with your add-on I do not need to change anything in the config.php for attachments to get saved in R2 vs. local? I see posts on the site about S3 and Digital Ocean needing some config in the config.php and want to make sure I've done all setup needed using your add-on.

You don't need to do anything with your config.php, feel free to ignore other unrelated addons and how they do things.

I think the R2 location for XFMG is incorrect in your add-on:

R2 for /internal-data/xfmg/

I found the xfmg folder in the DATA folder, not internal_data

As R2 has a bucket for DATA do we need one for XFMG and if so how is it rooted? Should it be xfmg/... (i.e. there is a xfmg parent folder?)

I checked my site now complete migrated to Cloudflare and you can see how the XFMG works and it appears you don't need that third bucket as it's under DATA.

XFMG, primarily uses the data folder, but there are use cases where it does in fact use internal_data/xfmg (more for when you are dealing with original uploads and you have variations like watermarked images). So unless you are doing that sort of thing, the internal-data/xfmg bucket isn't going to do much (or anything). The bulk of it's files are the same as the resource manager (reside in the data folder... which you already have a universal bucket for all of the data folder).

Since the data bucket covers everything in the data directory, you don't need to do anything for individual folders in the data directory. XFMG, XFRM and other third-party addons that do anything with the data directory are covered already.

 

[NealC]

Thank you. I just wanted to make sure I didn't need to rclone anything up to the XFMG bucket. I'll leave it enabled but nothing will be done unless you state otherwise.

 

[digitalpoint]

Thank you. I just wanted to make sure I didn't need to rclone anything up to the XFMG bucket. I'll leave it enabled but nothing will be done unless you state otherwise.

You would only need to move stuff to the internal-data/xfmg bucket if you have anything in an internal_data/xfmg folder (you may not as it's not always used). Definitely don't move things from data to internal_data (or vice versa). The paths are correct/not a typo, so if you do that you are going to be in for a world of issues.

 

[NealC]

You would only need to move stuff to the internal-data/xfmg bucket if you have anything in an internal_data/xfmg folder (you may not as it's not always used). Definitely don't move things from data to internal_data (or vice versa). The paths are correct/not a typo, so if you do that you are going to be in for a world of issues.

Thanks. I have no such folder under internal_data.

 

[NealC]

Any opinions on the Mirage option in beta?

Mirage​

Improve load time for pages that include images on mobile devices with slow network connections.

 

[digitalpoint]

Not something I use myself, but it works if you want it. Doesn’t seem terribly useful to me as XenForo already stores and uses multiple image sizes and lazy loads some. Try it out and see if your site loads faster.

 

[digitalpoint]

@Mouth are you still getting any 499 errors or did the network work itself out? I have a version with the automatic retry mechanism if it sees a 499 if you want to test it.

 

[NealC]

Take a look at this thread on my forum now using Cloudflare and R2. Post #1 has my images full size inline and they are served from domain.com/attachments. Go down to post #4 and the attachment is served from data.domain.com (R2) but if you click the image it's served again from domain.com/attachments.

Is something incorrect with URL's here? Are the full size images coming from the local server and not Cloudflare R2?

 

[digitalpoint]

That’s normal. Full sized attachments are stored in R2, but served from XenForo. It’s done that way because XenForo checks permissions of users before allowing them to view/download attachments. If you made all the attachments public, the permission system could be bypassed. Nothing to do with the addon though, that’s just how XenForo works. Even without R2, you’d see thumbnails (which are intended to be public for anyone) served from data directory, and full images are served through attachment route so permissions can be checked.

You are seeing the difference between thumbnails and full images.

 

[NealC]

Thanks. I was hoping to not have to worry about image storage on my local server if R2 was going to handle it going forward and not have to worry about storage space in the future.

Also, so if an image is saved is it saved both local and in R2? I'm curious how the two stay in sync.

 

[digitalpoint]

Thanks. I was hoping to not have to worry about image storage on my local server if R2 was going to handle it going forward and not have to worry about storage space in the future.

You don’t… the underlying data is in R2. Attachment routes for full images has the application (XenForo) grabbing them from R2 and passing it through to the user. Again, it’s normal and how XenForo works with or without R2. Even if you use the local filesystem, the attachment route for full images is XenForo reading the file from the local path and passing it through. Users can’t access anything in internal_data directly (that’s why it’s “internal”). Either way, nothing is actually in your local file system if you are using R2 for storage.

 

[Mouth]

If you do a little digging on the errors you got, are they particularly large attachments? Large enough that the time it takes to transfer them from Cloudflare might be coming into play? It's more going to be network connectivity/bandwidth available to your server rather than the server itself being overloaded.

Been away for a few days, so haven't been able to monitor these errors closely or investigate. Will do so during the next couple of days.

What we are left with is stuff that is outside my control (or your control)... if the network connections are being routed through a peer or other network that is dropping/resetting the connection. So for example, say a data center a few hops away that is routing your traffic rebooted the router/switch that your traffic was going through. That might cause it (along with a ton of other things... network congestion somewhere along the route for example).

Cloudflare and my host (Linode/Akamai) are interconnected by one peering hop (Equinix). It would seem unlikely to be dropped/reset connections by them.

So I think I'm going to treat 499 as a 5xx (and do a single retry behind the scenes and hop it worked the second try before failing to the error log). Another option would be to suppress 499 errors from the error log, but not sure I like that idea... like if my data center is having networking issues, it's probably a good thing to know.

are you still getting any 499 errors or did the network work itself out? I have a version with the automatic retry mechanism if it sees a 499 if you want to test it.

Yes, still getting errors. Happy to test it out. Thanks.

I wonder if you have ipv6 active on your service? Is it possible file transfer 'chunks' are occurring on both ipv4 and ipv6 routes, with one getting a reset/timeout and thus the failed file transit?

 

[digitalpoint]

I do have some sites using IPv6 and some that don’t. Chunked transfers shouldn’t be an issue because the backend (where you are seeing the error) isn’t using chunked transfers.

Have you run trace routes from your server to R2 IPs by chance? Maybe there’s a mucked route happening somewhere?

Whatever is happen does appear to be somehow specific to your setup. I have yet to see a single 499 error myself and I suspect if others were getting them in their error log, they would be posting about it here.

I’ll send you the new version when I get back to a computer. Retrying doesn’t fix whatever the underlying issue is though, more of a band aid.

 

[Mouth]

Have you run trace routes from your server to R2 IPs by chance? Maybe there’s a mucked route happening somewhere?

I don't think so, it's a single network peering hop between hosting infrastructure and CF R2 ....

:~$ traceroute 6cba001c6e66f6a2962585edfe412c3f.r2.cloudflarestorage.com
traceroute to 6cba001c6e66f6a2962585edfe412c3f.r2.cloudflarestorage.com (104.18.9.90), 30 hops max, 60 byte packets
1 10.216.0.5 (10.216.0.5) 0.178 ms 0.122 ms 0.094 ms
2 10.216.35.1 (10.216.35.1) 0.257 ms 10.216.35.2 (10.216.35.2) 0.239 ms 0.258 ms
3 10.216.32.1 (10.216.32.1) 0.917 ms 10.216.32.2 (10.216.32.2) 0.895 ms 10.216.32.1 (10.216.32.1) 0.872 ms
4 lo0-0.gw1.syd1.au.linode.com (172.105.160.101) 0.672 ms 0.639 ms *
5 13335.syd.equinix.com (45.127.172.154) 1.020 ms 1.033 ms ae0-100.gw2.syd1.au.linode.com (172.105.160.9) 0.507 ms
6 172.69.60.3 (172.69.60.3) 1.150 ms 108.162.250.5 (108.162.250.5) 1.404 ms 13335.syd.equinix.com (45.127.172.154) 3.845 ms
7 104.18.9.90 (104.18.9.90) 0.691 ms 172.68.64.3 (172.68.64.3) 4.427 ms 4.405 ms


:~$ traceroute6 6cba001c6e66f6a2962585edfe412c3f.r2.cloudflarestorage.com
traceroute to 6cba001c6e66f6a2962585edfe412c3f.r2.cloudflarestorage.com (2606:4700::6812:85a), 30 hops max, 80 byte packets
1 2600:3c0f:16::5 (2600:3c0f:16::5) 0.786 ms 0.687 ms 0.654 ms
2 2600:3c0f:16:35::1 (2600:3c0f:16:35::1) 0.973 ms 0.955 ms 1.054 ms
3 2600:3c0f:16:32::2 (2600:3c0f:16:32::2) 1.333 ms 2600:3c0f:16:32::1 (2600:3c0f:16:32::1) 1.419 ms 1.351 ms
4 2400:8907:100::102 (2400:8907:100::102) 0.840 ms 2400:8907:100::101 (2400:8907:100::101) 0.937 ms 0.713 ms
5 2400:8907:16:5::2 (2400:8907:16:5::2) 1.796 ms 13335.syd.equinix.com (2001:de8:6::1:3335:1) 1.337 ms 2400:8907:16:5::2 (2400:8907:16:5::2) 1.729 ms
6 2400:cb00:26:3:: (2400:cb00:26:3::) 1.713 ms 2400:cb00:491:3:: (2400:cb00:491:3::) 1.857 ms 2400:cb00:493:3:: (2400:cb00:493:3::) 45.287 ms
7 2400:cb00:490:3:: (2400:cb00:490:3::) 1.592 ms 2400:cb00:491:3:: (2400:cb00:491:3::) 1.636 ms 2400:cb00:492:3:: (2400:cb00:492:3::) 17.499 ms
8 2400:cb00:491:1024::ac44:911c (2400:cb00:491:1024::ac44:911c) 0.923 ms 2400:cb00:492:1024::ac44:d10a (2400:cb00:492:1024::ac44:d10a) 1.281 ms 2400:cb00:491:1024::ac44:9121 (2400:cb00:491:1024::ac44:9121) 0.852 ms

I’ll send you the new version when I get back to a computer. Retrying doesn’t fix whatever the underlying issue is though, more of a band aid.

Thanks!

 

[NealC]

Had a scary situation today, I downloaded my custom theme so I could run the images through tinyPNG to optimize them. I then created the zip archive and uploaded it to my site overwriting the former style. After that was completed I had a mess, some images not showing, logo.svg missing, etc yet if I use the URL in the browser it worked. I tried clearing cache, no luck, both sides at Cloudflare for the site and my browser. I finally went to R2 and to my data directory and uploaded the images from my computer directly to R2. That fixed it. Also purged cache.

There seems to be an issue if you upload or import a theme that possibly those files through that process do not get to R2. Maybe the data URL for external providers is not being honored?

 

[digitalpoint]

Is it a custom process or something built into XenForo? I can’t image XenForo not using its own abstracted file system.. but something done by a third party theme maker, I could see that…

 

[NealC]

It's all native to XF. Just going to the styles area and importing one. I set the option to overwrite my existing style. Maybe some hiccup with R2 sync? I don't know. I did the same process on my other site and it seemed to work but I uploaded the changes directly to R2 just in case.

 

[digitalpoint]

There shouldn’t be a “hiccup”. If there was an issue transferring files to R2, you would have all sorts of stuff in the error log. If you aren’t seeing anything in the error log, then for some reason XenForo didn’t try to put them into R2 for whatever reason. If one site worked and another didn’t, I’d look around at the differences between the sites. Probably a stupid question, but you are certain R2 was enabled when it happened and you didn’t have all addons disabled via config or something?

 

[NealC]

No errors on either site, R2 is definitely in play and things have been performing nicely. It seemed as if the style import system may be looking for config.php settings instead of the external data provider? I don't know. The two sites are quite different in styles and one is actually closed, it may not have had a cache conflict, who knows. I can probably test someday in importing one style from one site to the other and see if it makes it to R2 or not. Just import a style as a new style name and check R2 data.