![[DigitalPoint] App for Cloudflare®](/community/data/resource_icons/8/8750.jpg?1692633961){kind=icon}
[DigitalPoint] App for Cloudflare® 1.8.2
- Thread starter digitalpoint
- Start date
NealC
Well-known member
Strike that, it's not area specific. I went back to old posts with various attachments and see the issue there. Who really controls the content type, the requester or the host? For example, if you're going to request a file that ends in .PDF, shouldn't the caller by specifying the content type? If you know the extension of what you're calling can't you specify the content type? Thinking out loud....how can we resolve this?
NealC
Well-known member
Objects in R2 appear to have the correct content types. However, I believe something gets stored in XF with .data extensions and therefore R2 uses the application/octet-stream on those. Something must be wrong with the presigned URL code I'm guessing that's not decoding this properly? How can XF download the item ending in .data properly (content type) with presigned off if it's the same object from R2? My guess is XF knows the content type and is requesting it accordingly? Again...thinking out loud...
digitalpoint
Well-known member
It doesn't work that way.
If you are doing attachments "normally", the content type is set by XenForo.
Presigned URLs set the content type based on the content type that is set for the R2 object. You most DEFINITELY would not want to try and set a content type based on object key (what can look like a path/file name). All attachments are stored with a key ending in .data, but with a correct content type:
File extensions are never looked at for determining content type (it would be all sorts of problems if it worked that way). For example XernForo avatars universally have a jpg extension even if they are other formats. For example, look at the content types here:
Cloudflare will always determine the content type based on the content of the object when it's uploaded, EXCEPT for a case where you upload something with a specific content type set by the uploader (at that point, Cloudflare will treat it as the content type you are telling it to be, not the one Cloudflare thinks it is).
So in your case, it just sounds like you somehow uploaded existing content in some way where it was trying to be smarter than Cloudflare and instructing Cloudflare to treat it as a certain content type vs. what it actually is. The "right" thing to do would be to fix that (download them and just re-upload them to R2 without telling Cloudflare to force a certain content type). The right thing to do is not to try and build a bunch of one-offs and start carving out sub-directories or XenForo content types that presigned URLs do or don't work with. You are talking about massive added complexity and maintenance as it would surely cause other issues, when the correct thing to do is simply fix the issue of incorrect content types set by whatever you used to upload. Option B is simply to not use presigned URLs, then you aren't relying on the incorrect content type you have set on the R2 side.
TL;DR: you should never, EVER force set content type based on file extension with XenForo files. Even avatars have a .jpg extension 100% of the time if they are PNG files.
NealC
Well-known member
Thank you. I used that tool, discussed on this site or in your add-on description (can't remember) that mass uploads the files to R2. I don't recall an option to pass a content type and I guess the .data being generic R2 just put application/octet-stream on those. Not sure exactly how to fix but I'm not sure presigned URL's offer me that much savings to warrant this effort? I don't know... The question remains, how can this work when presigned URL's are OFF and the issue is only affected when ON?
digitalpoint
Well-known member
I don't use rclone myself, but I vaguely recall seeing somewhere it uses the mime-type definitions for the system you are running rclone on. The better thing to do is just not set the content type though (which is why things work better that were uploaded by the addon... new stuff).
Because XenForo sets the content-type as part of the request for the attachment. Presigned URLs bypass all that, you are redirecting users to the presigned URL which sets content type based on the content type set for the object in R2.
Wildcat Media
Well-known member
I used rclone and had no issues loading our files into R2.
digitalpoint
Well-known member
I believe that by default, rclone will attempt to override content type/mime type if the machine (server) running it has system-wide mime types for certain extensions. On Linux, that would be in the
So I can see how it would work fine for most people, but since rclone looks at system settings, there might be cases where it's trying to do something different on a certain server. Either way, the mime-type/content type being overridden is really only an issue when using presigned URLs (since there isn't an opportunity for XenForo to explicitly set it).
/etc/mime.types file. ".data" isn't a normal mime-type in there (I just checked my servers and it's not in there with a fairly vanilla/default file) , so I suspect someone on @NealC's server set .data to some specific mime type there, and then rclone decided to adopt that since that's what someone set it to system-wide.So I can see how it would work fine for most people, but since rclone looks at system settings, there might be cases where it's trying to do something different on a certain server. Either way, the mime-type/content type being overridden is really only an issue when using presigned URLs (since there isn't an opportunity for XenForo to explicitly set it).
digitalpoint
Well-known member
There certainly could be data that Cloudflare can't identify, but it never identifies based on object key (the part that looks like a path/filename/extension). You could certainly have a raw binary file of random bits and it wouldn't be able to identify it. But if it's something standard (image, PDF, zip file, video, audio, etc.) it will identify it.
Again... just to be clear even though the object keys look like path/filename/extension, they are actually not that. They are simply keys similar to how something like memcached works. On our side we pick a key that makes it easy to understand what it is, but as far as Cloudflare is concerned, the key is simply a random unique string and it will never, EVER try to assign a mime type/content type based on that object key. Cloudflare WILL try to "sniff" the content to figure out what the underlying content type is, UNLESS it's told to use a specific content type, then it will always use that instead.
As an example look at the avatars stored in R2 in this post, all the object keys end with
.jpg, but Cloudflare correctly assigned the right content type because it only looks at the actual content (not the key). XenForo saves all avatars with .jpg extension even if they are PNG (or other formats), so it's a good demo of Cloudflare only looking at the content (and never the key/"filename").The issue here is something (I assume rclone based on what you said) decided to try and be smarter than it needed to be and explicitly assign content types based on file extension (big no-no for XenForo stuff). If you uploaded avatars with rclone, I suspect they all got assigned
image/jpeg content types even if some are PNG or GIF. The good news for that is it's fairly common for images to get assigned the wrong extension on websites, so I suspect that for images, browsers will treat the image as the content it is rather than what it's told by the web server.
digitalpoint
Well-known member
Also, it wouldn't be terribly difficult to fix... All you would need to do is download the stuff you uploaded to R2 (like maybe something before a certain date?) There definitely are utilities to do that. And then upload them right back in a way that doesn't explicitly assign content type. The end result would just be the same file overwriting itself, but also allowing Cloudflare to do it's content sniffing magic. You wouldn't have any down time... users wouldn't know it was happening since it would just be overwriting what's already there.
digitalpoint
Well-known member
You could probably just use the CLI utility that this addon comes with (although it would do everything, not just a subset).
By default it goes from old to new abstracted filesystem, but there's an option to reverse that. Something like this would go from the new filesystem (R2) to the old (by default the local filesystem), doing 20 concurrently:
It will not delete anything, so you don't need to worry about that... then when it's done you could run the same command but without the
I'd probably just do internal_data/attachments, because the data never changes so you don't need to worry about users overwriting something while you are doing it. If you want to also do the data directory, just be aware that if you download them, then a user changes his avatar before it's uploaded back, their avatar would revert back to the one you uploaded back since you are then overwriting with what you downloaded after they changed it.
Code:
php cmd.php dp:migrate-data --helpBy default it goes from old to new abstracted filesystem, but there's an option to reverse that. Something like this would go from the new filesystem (R2) to the old (by default the local filesystem), doing 20 concurrently:
Code:
php cmd.php dp:migrate-data --new-to-old --processes=20It will not delete anything, so you don't need to worry about that... then when it's done you could run the same command but without the
--new-to-old flag (then it will take the stuff in the local filesystem to R2).I'd probably just do internal_data/attachments, because the data never changes so you don't need to worry about users overwriting something while you are doing it. If you want to also do the data directory, just be aware that if you download them, then a user changes his avatar before it's uploaded back, their avatar would revert back to the one you uploaded back since you are then overwriting with what you downloaded after they changed it.
Wildcat Media
Well-known member
If that were the case, wouldn't PNG images with .jpg extensions then be assigned an
image/jpg mime type?As I mentioned, everything worked properly when I used rclone, so I don't know if it would affect the mime type at all. My
/etc/mime.types don't even show a .data extension, although that was via a cursory glance at it so far.Rhetorical question, but why would rclone do anything to alter a mime type? Its function is to copy a file from one place to another. (Not that I'm an expert on it, but...)
digitalpoint
Well-known member
I’ve never personally used rclone, so can’t really answer any of it. I only know what I Google (which definitely could be wrong).
Just trying to help the best I can with something I’ve never used…
It does seem strange to me that rclone doesn’t at least have an option to disable that behavior. At least I couldn’t find it in their docs.
Semper Fidelis
Active member
The Install and Update AddOn reports your AddOn in perpetual need to be upgraded. Is there some versioning disconnect between the way you set your version and other AddOns that you are the only AddOn that I can't get this screen to disappear for?
digitalpoint
Well-known member
Don’t know anything about that addon, but not doing anything non-standard with versioning.
mjda
Well-known member
I just installed this add-on on a new site (fresh install, no posts, no other add-ons) and when I try to change a setting on the settings page, I get this error:
Might be worth noting that the API token is all added and I can add cache/page rules (haven't tested anything else), but I'm not able to change anything on the main settings page without getting that error.
Might be worth noting that the API token is all added and I can add cache/page rules (haven't tested anything else), but I'm not able to change anything on the main settings page without getting that error.
digitalpoint
Well-known member
Is it when trying to change a specific setting, or is it anything? Also, are you on a free or paid plan for the zone?
