Mr. Jinx
Well-known member
Absolutely, and I hope they will add more country TLDs.
I have many .nl domains which I would love to transfer to Cloudflare.
![[DigitalPoint] App for Cloudflare®](/community/data/resource_icons/8/8750.jpg?1692633961){kind=icon}
[DigitalPoint] App for Cloudflare® 1.8.2
- Thread starter digitalpoint
- Start date
I have many .nl domains which I would love to transfer to Cloudflare.
webroxau
Active member
I've been putting the finishing touches on the 1.4 version of this add-on, which supports Cloudflare R2 (similar to AWS S3). This allows you to move files off your server and into the cloud for yourdatafolder and/or yourinternal_data/attachmentssub-folder without any config file changes or any of the complexities that normally come with using XenForo's abstracted file system.
The cost involved should be free for most sites, and very inexpensive for huge sites. Cloudflare gives you 10GB of storage, 1,000,000 writes and 10,000,000 reads for free (and no bandwidth costs like S3). Additionally, if you configure cache rules for your public data folder, you could get substantially more than 10M reads for free (a read operation is only counted when it's not cached and needs to backhaul to R2). With proper caching rules and the way XenForo is setup, it wouldn't be out of the question to get 1M writes (people uploading things like new attachments or avatars) per month for free and 100M real-world reads per month (people viewing them) for free. At which point, most sites are simply left with storage cost.
Storage cost is very cheap ($0.015 per GB per month after the first 10GB [which is free]). As an example, you could offload 100GB of attachments and avatars from your server to R2 for $1.35 per month.
I've also built a bi-direction data migration tool that works with XenForo's CLI cmd.php system (you can not only move data to R2, but you can also move it from R2 if you decide it's not for you down the road).
Anyway, what I'm looking for people who want to give R2 a go for offloading files from your server (if you have less than 10GB of avatars/attachments, there would be no cost). I have it running on three of my live sites right now, but there's some fundamental changes to how XenForo's abstracted file system works, so looking for people who are willing to test it for me (it's also good to run it on different Cloudflare accounts that might be configured differently to see if any issues pop up). It wouldn't be a "here install this" thing, I'd want to coordinate a time that we are both available just in case anything unforeseen pops up so I can assist in realtime if needed. Basically I want to make sure this is fully tested before rolling it out for everyone.
For those that don't know what I'm talking about, R2 is a system that allows you to store objects (files in the case of XenForo) in the cloud, rather than your server and move that data closer to your users by leveraging Cloudflare's 275+ data centers around the world:
Cloudflare R2 | Zero Egress Fee Distributed Object Storage | Cloudflare
Cloudflare R2 is an S3-compatible, zero egress-fee, globally distributed object storage. Move data freely and build the multi-cloud architecture you desire.www.cloudflare.com
Ive come looking for a solution for this as I'm wanting to implement on one of my sites. This is perfect timing.
I don't have CloudFlare side of things set up yet (hoping to do something this weekend) but would be keen to help if you need any more testers.
digitalpoint
Well-known member
Sent it to you via conversation. Will be a good test for a site that doesn't have Cloudflare for anything to see if it's user friendly enough.
Calamity James
Active member
I'm also interested in testing the R2 stuff - I was trying to use Backblaze B2 but ran into a few issues with the AWS adaptor. Already having Cloudflare do the caching for my site I'm happy to use them for image hosting etc.
My
My
internal_data/attachments folder is 42GB if that matters!
MRaburn
Active member
I've been putting the finishing touches on the 1.4 version of this add-on, which supports Cloudflare R2 (similar to AWS S3). This allows you to move files off your server and into the cloud for yourdatafolder and/or yourinternal_data/attachmentssub-folder without any config file changes or any of the complexities that normally come with using XenForo's abstracted file system.
The cost involved should be free for most sites, and very inexpensive for huge sites. Cloudflare gives you 10GB of storage, 1,000,000 writes and 10,000,000 reads for free (and no bandwidth costs like S3). Additionally, if you configure cache rules for your public data folder, you could get substantially more than 10M reads for free (a read operation is only counted when it's not cached and needs to backhaul to R2). With proper caching rules and the way XenForo is setup, it wouldn't be out of the question to get 1M writes (people uploading things like new attachments or avatars) per month for free and 100M real-world reads per month (people viewing them) for free. At which point, most sites are simply left with storage cost.
Storage cost is very cheap ($0.015 per GB per month after the first 10GB [which is free]). As an example, you could offload 100GB of attachments and avatars from your server to R2 for $1.35 per month.
I've also built a bi-direction data migration tool that works with XenForo's CLI cmd.php system (you can not only move data to R2, but you can also move it from R2 if you decide it's not for you down the road).
Anyway, what I'm looking for people who want to give R2 a go for offloading files from your server (if you have less than 10GB of avatars/attachments, there would be no cost). I have it running on three of my live sites right now, but there's some fundamental changes to how XenForo's abstracted file system works, so looking for people who are willing to test it for me (it's also good to run it on different Cloudflare accounts that might be configured differently to see if any issues pop up). It wouldn't be a "here install this" thing, I'd want to coordinate a time that we are both available just in case anything unforeseen pops up so I can assist in realtime if needed. Basically I want to make sure this is fully tested before rolling it out for everyone.
For those that don't know what I'm talking about, R2 is a system that allows you to store objects (files in the case of XenForo) in the cloud, rather than your server and move that data closer to your users by leveraging Cloudflare's 275+ data centers around the world:
Cloudflare R2 | Zero Egress Fee Distributed Object Storage | Cloudflare
Cloudflare R2 is an S3-compatible, zero egress-fee, globally distributed object storage. Move data freely and build the multi-cloud architecture you desire.
I'd be willing to test this plugin with you. I currently have 80gb of images stored on Digital Ocean's Spaces and use [OzzModz] Cloudflare Image Resizing Add-On which does a good job or serving optimized images. I was curious if a better solution would be to use the new "Cloudflare Images" to upload and store on CF and they will serv an optimized image based on the request. This would eliminate the need for S3/Spaces and a Plugin to optimize the image. Any thoughts appreciated.
Cloudflare Image Optimization · Cloudflare Image Optimization docs
Cloudflare Images provides an end-to-end solution to build and maintain your image infrastructure from one API.
developers.cloudflare.com
Mike
digitalpoint
Well-known member
Well, I don't think moving the original images from Digital Ocean's Spaces to R2 will change anything with how you are using Cloudflare Images. R2 is a different product and there's not a way (currently) that I know of where Cloudflare Images fetches directly from your R2 bucket. So in that regard, you really would just be storing where the original images reside.I'd be willing to test this plugin with you. I currently have 80gb of images stored on Digital Ocean's Spaces and use [OzzModz] Cloudflare Image Resizing Add-On which does a good job or serving optimized images. I was curious if a better solution would be to use the new "Cloudflare Images" to upload and store on CF and they will serv an optimized image based on the request. This would eliminate the need for S3/Spaces and a Plugin to optimize the image. Any thoughts appreciated.
Cloudflare Image Optimization · Cloudflare Image Optimization docs
Cloudflare Images provides an end-to-end solution to build and maintain your image infrastructure from one API.developers.cloudflare.com
Mike
That being said, R2 is less expensive than Spaces (although not by a crazy amount). A few benefits you would get though is I've rewritten how XenForo's filesystem mount system works a little bit:
- Fetching an R2 object takes one API call vs. two if you use a drop-in Flysystem adapter like S3 (info on that over here).
- I've also reworked how mounts work. While you can still do it as an "all or nothing" system if you want, it didn't make much sense to me to be storing everything in
internal_data, so I've made it so you can send just certain folders within there to different filesystems (by default, the add-on will just sendinternal_data/attachmentsto R2, and the rest of the folders in there stay on whatever filesystem you want (local or I guess in your case S3). Just doesn't make sense to me to have things like DKIM key and file check results, oembed cache, sitemaps, etc. stored in the cloud. So by default the UI gives you two options (all of the data folder, which is intended to be public) and just the attachments folder in internal_data. You can however do other sub-folders or internal_data entirely if you wanted via config.php file changes.
Anyway, long story short is you still would need whatever you are using for the Cloudflare Image service side of things (normally you just do it with special URLs, so you would still need whatever add-on you use that injects those image URLs for you).f
MRaburn
Active member
Thanks, I may give this a try if I can get all my images moved over.
In reference to CF Images, this is a new service I am seeing as opposed to the before Images where they just optimize and store in the cloud, this is a service where you upload images direct via API, etc. and then they get stored and optimized that way, removing the need for R2, S3, Spaces etc for Images.
"Cloudflare Images lets you set up an image pipeline in minutes. Build a scalable image pipeline to store, resize, optimize and deliver images in a fast and secure manner."
Cloudflare Images | Store and Deliver Images at Scale
Cost-effectively store, resize, optimize, and deliver images with predictable pricing — without the need for intermediary storage.
digitalpoint
Well-known member
Ya, if you have an addon for Cloudflare Images (where the images are stored there, not just resized), you should be able to just keep using that addon regardless if you use R2 for other things like attachments.
However.... off the top of my head, I'd think that would work well for things like avatars (anything in the data directory since that is intended to be public) but not so much for things in internal_data (which is where images uploaded in posts/conversations are going to reside). Image attachments in internal_data normally need to adhere to XenForo group permissions (so XenForo is handling the request and then choosing to serve or not serve the content based on the user's permissions). So if XenForo is needing to do backend API calls to get the image so it can serve it, you aren't really saving a whole lot because it's the server making the underlying request for the image, not the end user (it's not going to be optimized for the end user specifically). At which point, you might as well keep it simple (and cheaper) and use S3/R2 at least for internal_data since those images are never accessed by users directly.
However.... off the top of my head, I'd think that would work well for things like avatars (anything in the data directory since that is intended to be public) but not so much for things in internal_data (which is where images uploaded in posts/conversations are going to reside). Image attachments in internal_data normally need to adhere to XenForo group permissions (so XenForo is handling the request and then choosing to serve or not serve the content based on the user's permissions). So if XenForo is needing to do backend API calls to get the image so it can serve it, you aren't really saving a whole lot because it's the server making the underlying request for the image, not the end user (it's not going to be optimized for the end user specifically). At which point, you might as well keep it simple (and cheaper) and use S3/R2 at least for internal_data since those images are never accessed by users directly.
digitalpoint
Well-known member
digitalpoint updated [DigitalPoint] Cloudflare with a new update entry:
Big update, including support for R2
Read the rest of this update entry...
Big update, including support for R2
Read the rest of this update entry...
digitalpoint
Well-known member
So here's some info about R2 (I'm sure people are going to have questions).
R2 is similar to AWS S3... allowing you to store files/objects in the cloud. In the case of XenForo, it's a fantastic way to offload attachment and avatars from your web server. Cloudflare gives you 10GB for free along with 1,000,000 class A (effectively write) operations per month as well as 10,000,000 class B (basically read) operations per month. There are no egress (bandwidth costs). Cached hits don't count towards class B operations. So basically you can use it for free if you have less than 10GB of data to store and users are uploading less than a million new avatars/attachments per month. Beyond that, it's incredibly cheap ($0.015 per GB per month after the first free 10GB). As an example, if you needed to store 50GB worth of avatars and attachments, the cost would be $0.60 per month.
About XenForo's filesystem...
I've also made some underlying changes to XenForo's abstracted file system to allow individual folders to reside on a specific file system. For example, by default this add-on only moves the attachments folder within internal_data to a different file system (the other folders in internal_data aren't particularly well-suited for being stored in the cloud) I've also made some changes so that if someone used a completely different filesystem (via config.php file changes), it can be more efficient if you choose. For example, XenForo uses Flysystem's default assert mode which means you need to make an extra API call to see if a file exists before you actually read it (not efficient, especially when dealing with cloud-based storage). I've built the filesystem changes in a way that other developers can support their own adapters if they want:
R2 is similar to AWS S3... allowing you to store files/objects in the cloud. In the case of XenForo, it's a fantastic way to offload attachment and avatars from your web server. Cloudflare gives you 10GB for free along with 1,000,000 class A (effectively write) operations per month as well as 10,000,000 class B (basically read) operations per month. There are no egress (bandwidth costs). Cached hits don't count towards class B operations. So basically you can use it for free if you have less than 10GB of data to store and users are uploading less than a million new avatars/attachments per month. Beyond that, it's incredibly cheap ($0.015 per GB per month after the first free 10GB). As an example, if you needed to store 50GB worth of avatars and attachments, the cost would be $0.60 per month.
About XenForo's filesystem...
I've also made some underlying changes to XenForo's abstracted file system to allow individual folders to reside on a specific file system. For example, by default this add-on only moves the attachments folder within internal_data to a different file system (the other folders in internal_data aren't particularly well-suited for being stored in the cloud) I've also made some changes so that if someone used a completely different filesystem (via config.php file changes), it can be more efficient if you choose. For example, XenForo uses Flysystem's default assert mode which means you need to make an extra API call to see if a file exists before you actually read it (not efficient, especially when dealing with cloud-based storage). I've built the filesystem changes in a way that other developers can support their own adapters if they want:
- Ability to disable assert mode on a per adapter basis (faster/more efficient)
- Ability to assign specific folders within the abstracted file system to different adapters (for example, maybe you want to store attachments in the cloud, but not file_check or DKIM keys).
- Ability to configure filesystems without touching config.php (have a UI for it rather than expect users to muck with config.php file).
- I've done it in a way so it's backwardly compatible (you can still do config.php filesystem changes like normal if you want).
Chromaniac
Well-known member
so... i am already using r2 using the xenforo addon. i guess i would need to move back content from internal_data back to my server because this addon is only targeting the attachment folder which is good. rest i guess i would have to move manually from existing bucket to two buckets this addon would create? and of course clear out the lines in config file. hmm.
digitalpoint
Well-known member
Ya, so for those wanting to move things around because of a situation like that, this is how things are organized with this add-on (and why)... you can do more via config.php changes like other filesystems, but I'll just talk about what is done via the admin UI.
There can be two buckets... one for the
data folder entirely (this is intended by XenForo to be public, so you end up accessing the files here via a public subdomain. The bucket is basically the root of the data folder. The files/objects in the bucket will have paths intact like so:
For the internal_data/attachments bucket,
internal_data is still considered the root (it's done this way in case someone wanted to store more than just attachments in there). So the structure of the files/objects with paths intact look like so:
Now... if you want to store all of
internal_data, you still can via config.php edits like other file systems, but eh... not something I would personally do, but you can if you want.If you need to move data around to get a public bucket and a private bucket with those path structures, you can do so and then just flip on R2 in the admin and tell it which bucket it is on your Cloudflare account (it will simply start using that bucket and just assume everything is there.. so you don't need the add-on itself to move anything).
If you have a ton of data, rclone is a good tool. If you don't have a ton (say under 10GB), the CLI tool I built that is included allows you to move data between whatever adapter you have setup in your config.php file and one you have setup in the UI (in this case, that's just going to be my R2 adapter). In theory you could use it to move between an S3 compatible adapter (probably what you are using now) and my R2 adapter. However, it does probably make more sense to move everything off the S3 adapter into your local filesystem and then move from there to my R2 adapter since most of the directories in internal_data don't get sent to my R2 adapter by default.
As a side note, my R2 adapter is going to be faster than an S3 one that XenForo uses because I've made some changes to allow it to disable Flysystem's "assert mode". Which means you are currently doing 2 API calls to read a file, and mine will just need one.
Hopefully that makes some sort of sense...
Chromaniac
Well-known member
I guess Cloudflare just hates me. I did create a brand new api key and rechecked all permissions. This error appeared when I tried enabling the data bucket
Chromaniac
Well-known member
ok so weird behavior.
i got the buckets enabled. i copied attachments data from existing r2 bucket to the bucket generated by the addon. attachments are not visible in threads
i got the buckets enabled. i copied attachments data from existing r2 bucket to the bucket generated by the addon. attachments are not visible in threads
digitalpoint
Well-known member
The file names are set by XenForo, not the add-on. If you look in your internal_data/attachments folder, you will see the same naming structure.
digitalpoint
Well-known member
If you are talking about attachment previews, those are in your data folder (not internal_data). Basically data/attachments are the publicly accessible previews, internal_data/attachments are the private underlying attachment itself (private as in only XenForo is intended to access them and pass them through to the user).ok so weird behavior.
i got the buckets enabled. i copied attachments data from existing r2 bucket to the bucket generated by the addon. attachments are not visible in threads
Chromaniac
Well-known member
digitalpoint
Well-known member
Are you getting any underlying error in your Server Error Log in XenForo admin? Hard to debug on this side without knowing the error, not being able to see your buckets or config or anything else.
