[DigitalPoint] App for Cloudflare® 1.8.2

[digitalpoint]

From the looks of the error, I really don’t think it would work with S3 either because they are always reading it from the local filesystem via file_get_contents(). Maybe they were making exceptions if S3 was being used, but if you were going to the trouble of making an exception, and one way uses XenForo’s abstracted filesystem, why wouldn’t you make all the ways use the abstracted filesystem? So that seems weird to me that it even could work with S3.

Reading/writing anything inside the data or internal_data folders should always use XenForo’s abstracted filesystem calls, which that addon is not doing.

 

[digitalpoint]

So, i) create the bucket on CF R2 first, ii) rclone contents of the dir from XF to R2, iii) visit /admin.php?cloudflare/r2 to switch on R2 integration and select to use an existing bucket. Is this correct?

Yep… as @puterfixer mentioned, you’ll need one bucket for data and one for internal_data if you are planning to move both areas.

 

[Mouth]

you’ll need one bucket for data and one for internal_data

Just a minor point @ /admin.php?cloudflare/r2 ...



... that should be an underscore for Internal_data, not a hypen.

 

[digitalpoint]

internal_data refers to the directory in the local file system, internal-data is the abstracted mounting point (so in the context of what it is, it's should have the dash, not the underscore)... I mean if we are splitting hairs.

 

[MattW]

From the looks of the error, I really don’t think it would work with S3 either because they are always reading it from the local filesystem via file_get_contents(). Maybe they were making exceptions if S3 was being used, but if you were going to the trouble of making an exception, and one way uses XenForo’s abstracted filesystem, why wouldn’t you make all the ways use the abstracted filesystem? So that seems weird to me that it even could work with S3.

Reading/writing anything inside the data or internal_data folders should always use XenForo’s abstracted filesystem calls, which that addon is not doing.

Andy has responded to a PM discussion on his site:

ask him if he is using the abstracted file system with his add-on

I’m sort of stuck in the middle here though.

 

[digitalpoint]

Andy has responded to a PM discussion on his site:


I’m sort of stuck in the middle here though.

Lol… it’s an easy fix, but I don’t have access to his stuff, so I can’t fix it. The error you posted shows his addon using file_get_contents(). If that function is being used in his addon, it’s not using the abstracted file system. Look on line 167 of his Controller/Post.php file.

Crazy to me that he thinks the error is coming from anything but his addon if you showed him the error message.

 

[MattW]

        // check condition
        if ($internalDataUrl instanceof \Closure)
        {
            // get internalDataUrl
            $internalDataUrl = $internalDataUrl(null, "canonical");

            // get sourceLink
            $sourceLink = $internalDataUrl . 'attachments/' . $lastFolder . '/' . $dataId . '-' . $fileHashOriginal . '.data';

            // get response
            try
            {
                $client = \XF::app()->http()->client();
                $response = $client->get($sourceLink);
            }
            catch (\GuzzleHttp\Exception\RequestException $e)
            {
                return;
            }

            // get contents
            $response = $response->getBody()->getContents();

            // get tempPathFull
            $tempPathFull = $tempPath . $dataId . '-' . $fileHashOriginal . '.data';

            // put file
            file_put_contents($tempPathFull, $response);
        }
        else
        {
            // get internalDataPath
            $internalDataPath = \XF::app()->config('internalDataPath');

            // get fullpath
            $fullpath = $internalDataPath . '/attachments/' . $lastFolder . '/' . $dataId . '-' . $fileHashOriginal . '.data';

            // get attachment
            $response = file_get_contents($fullpath);

            // get tempPathFull
            $tempPathFull = $tempPath . $dataId . '-' . $fileHashOriginal . '.data';

            // put file
            file_put_contents($tempPathFull, $response);
        }

Specifically line 167

$internalDataPath = \XF::app()->config('internalDataPath');

 

[puterfixer]

The abstraction layer is there to ensure that multiple plugins play together well. By not using it, the author hardwires the functionality to the backend, assuming that this will not break integration with other plugins. However, as we know from the brilliant quote from the otherwise dull Steven Seagal movie "Under Siege 2: Dark Territory (1995)",

[Ryback has been revealed as the intruder, supposedly dead]
Penn : When she shot the intruder, did you see the body?
Mercenary #1 : No, just a **** load of blood, and I figured if you get run over by a train...
[Penn smacks Merc 1 hard]
Penn : [slow and menacing] Did... you... see... the body?
Mercenary #1 : I ASSUMED he was DEAD!
Penn : Assumption is the MOTHER of all **** ups!

 

[digitalpoint]

        // check condition
        if ($internalDataUrl instanceof \Closure)
        {
            // get internalDataUrl
            $internalDataUrl = $internalDataUrl(null, "canonical");

            // get sourceLink
            $sourceLink = $internalDataUrl . 'attachments/' . $lastFolder . '/' . $dataId . '-' . $fileHashOriginal . '.data';

            // get response
            try
            {
                $client = \XF::app()->http()->client();
                $response = $client->get($sourceLink);
            }
            catch (\GuzzleHttp\Exception\RequestException $e)
            {
                return;
            }

            // get contents
            $response = $response->getBody()->getContents();

            // get tempPathFull
            $tempPathFull = $tempPath . $dataId . '-' . $fileHashOriginal . '.data';

            // put file
            file_put_contents($tempPathFull, $response);
        }
        else
        {
            // get internalDataPath
            $internalDataPath = \XF::app()->config('internalDataPath');

            // get fullpath
            $fullpath = $internalDataPath . '/attachments/' . $lastFolder . '/' . $dataId . '-' . $fileHashOriginal . '.data';

            // get attachment
            $response = file_get_contents($fullpath);

            // get tempPathFull
            $tempPathFull = $tempPath . $dataId . '-' . $fileHashOriginal . '.data';

            // put file
            file_put_contents($tempPathFull, $response);
        }

Specifically line 167

$internalDataPath = \XF::app()->config('internalDataPath');

There is so much wrong even in that little bit of code...

1. internalDataPath is not necessarily a publicly accessible location, that simply tells which folder inside the abstracted path to use (defaults to internal_data), and my addon does not change it.
2. internalDataUrl is not a thing. Trying to artificially generate a publicly accessible URL for something that is INTERNAL DATA and then accessing it from that URL... uh what? Why? Also, why would a site put their internal data on a publicly accessible URL unless they did something wrong?
3. Relying on $lastFolder is not going to be 100% reliable because adding attachments is a multi-threaded process (it will usually work, but maybe not if someone adds a bunch at the same time and the later ones roll over to a new folder... which happens every 1,0000 (why not use the $attachmentData entity to get the abstracted path? There's literally a method for that, so that code is making things way more complicated that it would be if it just used XenForo entities).
4. Using file_get_contents() to get anything inside an abstracted file system will fail if that file system is not local (or a publicly accessible URL).
5. file_put_contents() isn't the right way to write to XenForo's temp directory either because it's also an abstracted file system (usually it doesn't change, but you can't be assured of that).
6. Doing different things based on if a config setting is a Closure or not is silly too. The whole point of abstracted file system is that you don't need to write extra code for stuff. Just use the abstracted file system and it figures out where things are.... that's the whole point of an abstracted file system... it's abstracted for you.

#4 is the problem here. grep the contents of my addon and you will see exactly zero uses of file_get_contents() or file_put_contents(). Using those for anything that belongs inside the abstracted filesystem means you are not using the abstracted file system. The error log you posted clearly shows he's using file_get_contents (even without seeing the code), and for them to tell you that it must be my addon that isn't using the abstracted file system is so dumb, I don't even know where to begin with that one. The error shows it's in his file. So either he doesn't understand how to read an error or doesn't understand what an abstracted filesystem is. Either of those should scare you.

 

[digitalpoint]

Just a minor point @ /admin.php?cloudflare/r2 ...

View attachment 281520

... that should be an underscore for Internal_data, not a hypen.

So I was thinking about this a little more... even though the dash is correct for what it's trying to convey, the way it's formatted might not be ideal.

If it was presented like this, would it have made more sense to you? Just wondering if it makes more sense to me only because I'm a developer.

 

[Mouth]

If it was presented like this, would it have made more sense to you?

I understand it, but generically probably less I don't think it needs fixing/action, just a little thing I noticed, and your response explained it.

 

[Mouth]

So either he doesn't understand how to read an error or doesn't understand what an abstracted filesystem is. Either of those should scare you.

@MattW I'm very surprised you're using his addons. The risks are well known, and long running.

 

[Mouth]

and my rclone command:
/usr/bin/rclone sync r2z22se:z22se/data/ r2z22se:z22se-data/ --verbose --transfers 10

Thanks, this was helpful. I used;
internal_data/attachments# rclone sync -P --transfers=20 ./ CloudflareR2:<bucket name>

 

[Mouth]

After rclone of all my internal_data/attachments/* and turning on 'R2 for /internal-data/attachments/' only ('R2 for /data/' not yet active) within this addon, I'm getting many 404 errors for /attachments/* files, eg https://<my site>/attachments/img_5016-jpg.262521/



Firstly, is my assumption that /attachments/* files are from internal_data/attachments/* correct? And thus, the 404 errors are likely to be related to activating 'R2 for /internal-data/attachments/'?

I'm not sure on the file structure within internal_data/attachments/ but using the above 404 example, I see that the file appears to exist both on my server and within my R2 bucket ...

internal_data/attachments/262$ ls -la 262521*
-rw-r--r-- 1 www-data www-data 112331 Mar 13  2022 262521-abe9770693e8ef2ffa4d239643b5ec7e.data

I only have 1 bucket on R2, and it was selected as an existing bucket when activating R2 within this add-on.

internal_data/attachments$ find . -type f | wc -l
422176

All functions at /admin.php?cloudflare/r2[/*] are working, so I don't believe permissions are an issue ...

DigitalPoint CloudFlare Add-on API token summary
This API token will affect the below accounts and zones, along with their respective permissions

<redacted> Account - Turnstile:Edit, Workers R2 Storage:Edit, Account Analytics:Read, Access: Organizations, Identity Providers, and Groups:Read, Workers Scripts:Edit, Access: Apps and Policies:Edit
<my site> - Cache Rules:Edit, Bot Management:Edit, Zone Settings:Edit, Zone:Edit, SSL and Certificates:Edit, Cache Purge:Purge, Page Rules:Edit, Firewall Services:Edit, Analytics:Read
Client IP Address Filtering
Is in - <my server IPv4 and IPv6 redacted>

Appreciate tips/steps on how I can diagnose this further?
FYI, I'm not currently using CF for caching, just DNS (and now R2).

 

[digitalpoint]

Looks like you are missing the full path to the “files”… no attachments “folder”. See this post: https://xenforo.com/community/threads/digitalpoint-app-for-cloudflare®.206176/post-1605618

 

[Mouth]

Looks like you are missing the full path to the “files”… no attachments “folder”. See this post: https://xenforo.com/community/threads/digitalpoint-app-for-cloudflare®.206176/post-1605618

Urgghh, thanks!
Perhaps a RM FAQ entry for that requirement would make it more easily found? I missed that when going through the activation process.

Now I'm off to try and figure out if I can 'move' all R2 files into a 'subfolder'

 

[Moshe1010]

A new tool to migrate large chunks of data from Cloudflare (replacing rclone?):

R2 Migrator · Cloudflare R2 docs

R2 Migrator (also known as Super Slurper) allows you to quickly and easily copy objects from other cloud providers to an R2 bucket of your choice.

developers.cloudflare.com

 

[digitalpoint]

Nope... that's for moving from S3 or something compatible. Won't work moving local files. The summarized version is when the request for an object comes in, if it doesn't exist, it will go out and fetch it from S3 and upload it before it serves it. Basically it's a lazy way to migrate from another cloud provider. Lazy in that you need to keep the old cloud provider running/serving content so it can fetch the stuff it doesn't have yet.

...copy objects from other cloud providers...

 

[Wildcat Media]

I'm a bit baffled...

I'm serving fonts from the server. I've saved them in data://assets/fonts. On a test site without Cloudflare, they work fine. But with R2 on a live forum, I'm getting 404 errors for all of the fonts, the 404 error page generated by Cloudflare. And, this:

Access to font at 'https://forumdata.xxxxx.com/assets/fonts/Roboto-Italic.woff2' from origin 'https://forum.xxxxx.com' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.

First, one error I made myself--I had to create a 'fonts' folder under 'assets' in the R2 bucket, then upload all the font files there. (I forgot that files only appear in the bucket if XF initiates the file transfer--my manual uploading of files will not automagically make them appear in the R2 bucket.)

The font files are blocked because I need to create a CORS policy. And it looks like I can do this in the R2 bucket (the feature is in beta).

Configure CORS · Cloudflare R2 docs

Cross-Origin Resource Sharing (CORS) is a standardized method that prevents domain X from accessing the resources of domain Y. It does so by using …

developers.cloudflare.com

I could set AllowedOrigins to "*" but that is too hacky. I have my data bucket set to forumdata.xxxxx.com so I am not sure if I enter that URL, the URL of my server, or...?

And do I need to be adding any headers in Apache on my server? (I have mod_headers enabled if so.) I'm not exactly sure if I need to set this at the bucket or on my server, or both.

 

[digitalpoint]

It would be the URL of your server. For example if it was this site, you would want the access-control-allow-origin header returned by your R2 bucket to be https://xenforo.com. Honestly though, it's probably overkill if you are just wanting to serve a font. It's more for using pre-signed URLs with R2.

But ya... fonts are subject to CORS headers/responses while most other static resources are not.