Cloudflare + Image Attachments

[themago]

Are image attachments not cached by cloudflare? I have a forum with 1-1.5TB of image attachments, and recently noticed the servers port speed is maxing out constantly (1.5gbps) and causing the forum to load quite slowly....

Then I realized (I think) that without certain page rules set in cloudflare, image attachments are not set to be cached on xenforo when cloudflare is enabled...

 

[Sam9]

The issue is attachment files are served within Xenforo, which allows for things like permission checks and counting views/downloads. Because of this, Cloudflare won't cache attachments by default (you'll see the cf-cache-status: dynamic header) on these requests.

If you can live without permission checks and view counts for attachments, then you can add a page rule for Cloudflare to cache the /attachments/ folder. Even better would be offloading your attachments to another server/service (eg using S3 compatible storage - Wasabi+Cloudflare would be cheaper if your bandwidth needs aren't too high). If that isn't an option, even just disabling the permissions and views check for attachments and allow attachments to be served directly by the HTTPd (not sure if there is an existing addon that'd make this possible). This would mean every attachment request isn't having to go through Xenforo/PHP/SQL queries, etc.

Note that Cloudflare's total cache size isn't unlimited, so your entire 1.5TB won't be cached at all times by Cloudflare. Depending on the amount of traffic you see, Cloudflare could still hit your server with a few 100Mbps traffic (eg from our experience, ~1.5TB attachments and ~1PB/m traffic sees spikes of ~400Mbps from Cloudflare).

 

[themago]

The issue is attachment files are served within Xenforo, which allows for things like permission checks and counting views/downloads. Because of this, Cloudflare won't cache attachments by default (you'll see the cf-cache-status: dynamic header) on these requests.

If you can live without permission checks and view counts for attachments, then you can add a page rule for Cloudflare to cache the /attachments/ folder. Even better would be offloading your attachments to another server/service (eg using S3 compatible storage - Wasabi+Cloudflare would be cheaper if your bandwidth needs aren't too high). If that isn't an option, even just disabling the permissions and views check for attachments and allow attachments to be served directly by the HTTPd (not sure if there is an existing addon that'd make this possible). This would mean every attachment request isn't having to go through Xenforo/PHP/SQL queries, etc.

Note that Cloudflare's total cache size isn't unlimited, so your entire 1.5TB won't be cached at all times by Cloudflare. Depending on the amount of traffic you see, Cloudflare could still hit your server with a few 100Mbps traffic (eg from our experience, ~1.5TB attachments and ~1PB/m traffic sees spikes of ~400Mbps from Cloudflare).

As far as I know wasabi can't be used on a high traffic website as it's mainly for "storage" and not for constant access requests, isn't that the case?

 

[Sam9]

As far as I know wasabi can't be used on a high traffic website as it's mainly for "storage" and not for constant access requests, isn't that the case?

Right, Wasabi only allows transfer up to the amount stored, or there abouts. Eg, if you have 5TB stored you can only transfer ~5TB/month. They're not overly strict with this, but if your bandwidth needs are too high then they'll move you to a $/GB plan.

Another option is Backblaze B2, which you can use with Cloudflare and have 0 rated bandwidth, due to the bandwidth alliance.

 

[themago]

Right, Wasabi only allows transfer up to the amount stored, or there abouts. Eg, if you have 5TB stored you can only transfer ~5TB/month. They're not overly strict with this, but if your bandwidth needs are too high then they'll move you to a $/GB plan.

Another option is Backblaze B2, which you can use with Cloudflare and have 0 rated bandwidth, due to the bandwidth alliance.

Thanks for the suggestion, on a forum with 200,000-250,000 uniques/day though, I think backblaze's fee per 'transaction' would still end up being really high, right?

 

[Sam9]

Thanks for the suggestion, on a forum with 200,000-250,000 uniques/day though, I think backblaze's fee per 'transaction' would still end up being really high, right?

You'd be charged for uncached transactions that Cloudflare makes to the B2 bucket, which would be considered class B transactions @ $0.004 per 10,000, with the first 2,500 per day being free. So even if every single request went uncached, you'd only be paying pennies for transactions.

 

[themago]

Right, Wasabi only allows transfer up to the amount stored, or there abouts. Eg, if you have 5TB stored you can only transfer ~5TB/month. They're not overly strict with this, but if your bandwidth needs are too high then they'll move you to a $/GB plan.

Another option is Backblaze B2, which you can use with Cloudflare and have 0 rated bandwidth, due to the bandwidth alliance.

If pictures and videos are used in attachments, wouldn't implementing cloudflare get my account suspended since videos should not be cached with cloudflare? (I've already had this happen before) or would a custom page rule be needed?

 

[Sam9]

If pictures and videos are used in attachments, wouldn't implementing cloudflare get my account suspended since videos should not be cached with cloudflare? (I've already had this happen before) or would a custom page rule be needed?

Cloudflare suspends or limits if you're caching an excessive percentage of non html traffic, however the exact percentages are unknown. So for example, running a popular filehost using Cloudflare will result in suspension or bandwidth limiting. However, just hosting a really popular website which is just image and video heavy is unlikely to do so, we're using ~1PB/m with ~4% html requests, and no attachments over 10MB. The only way to find out if your use case is acceptable for Cloudflare would be to try.

If you run into problems with Cloudflare, you could use BunnyCDN or FDC who are relatively cheap as CDNs go, or you could always setup your own servers as a DIY CDN instead. It all depends on your budget.

 

[Jake B.]

The issue you're running into is that XenForo explicitly adds a Cache-Control: private, no-cache, max-age=0 header to attachment responses. Even if you use the first party add-on for s3 compatible storage the full size attachment will be served through PHP by XenForo after downloading it to a temporary file, and won't be cached by cloudflare. What you'll want is a way to force XenForo to return a different Cache-control header with attachments to allow CloudFlare, or any other CDN, to cache the image. Whether something exists publicly for that or not, I've got no clue unfortunately.

 

[themago]

The issue you're running into is that XenForo explicitly adds a Cache-Control: private, no-cache, max-age=0 header to attachment responses. Even if you use the first party add-on for s3 compatible storage the full size attachment will be served through PHP by XenForo after downloading it to a temporary file, and won't be cached by cloudflare. What you'll want is a way to force XenForo to return a different Cache-control header with attachments to allow CloudFlare, or any other CDN, to cache the image. Whether something exists publicly for that or not, I've got no clue unfortunately.

There's an addon by ticktack I was able to find actually, https://xenforo.com/community/resources/image-attachment-cache-control.7964/

I set Enable image attachment cache control / private / 60 days

 

[themago]

Cloudflare suspends or limits if you're caching an excessive percentage of non html traffic, however the exact percentages are unknown. So for example, running a popular filehost using Cloudflare will result in suspension or bandwidth limiting. However, just hosting a really popular website which is just image and video heavy is unlikely to do so, we're using ~1PB/m with ~4% html requests, and no attachments over 10MB. The only way to find out if your use case is acceptable for Cloudflare would be to try.

If you run into problems with Cloudflare, you could use BunnyCDN or FDC who are relatively cheap as CDNs go, or you could always setup your own servers as a DIY CDN instead. It all depends on your budget.

Cloudflare will temporarily suspend your account if you are caching any kind of videos at all, I've had it happen multiple times already on various websites... so unless I'm able to differentiate videos from images in backblaze, I'm not sure if this will work for me

 

[Sam9]

Cloudflare will temporarily suspend your account if you are caching any kind of videos at all, I've had it happen multiple times already on various websites... so unless I'm able to differentiate videos from images in backblaze, I'm not sure if this will work for me

It depends on the ratio of html to non html traffic you have, regardless of it being videos or images. If you have a lot of video files, especially large videos, consuming a lot of bandwidth, then yes that worsen the ratio and you'll likely run into problems.

The issue you're running into is that XenForo explicitly adds a Cache-Control: private, no-cache, max-age=0 header to attachment responses. Even if you use the first party add-on for s3 compatible storage the full size attachment will be served through PHP by XenForo after downloading it to a temporary file, and won't be cached by cloudflare. What you'll want is a way to force XenForo to return a different Cache-control header with attachments to allow CloudFlare, or any other CDN, to cache the image. Whether something exists publicly for that or not, I've got no clue unfortunately.

Setting a page rule as I mentioned will ignore the cache-control header from the origin server, you need to make sure cache level is set to everything and the edge TTL rule too.

I wasn't aware that the externalDataUrl config value was limited to attachment thumbnails and not full attachments, I'm not sure if there's another config value where you can change the attachment path to a URL endpoint? We use the attachment store addon because we'd been using it since XF1, but I'd thought it was now possible to achieve similar results without an addon.

 

[Jake B.]

Setting a page rule as I mentioned will ignore the cache-control header from the origin server, you need to make sure cache level is set to everything and the edge TTL rule too.

Issue there is that when you force /attachments/* to cache with a page rule you may run into error pages from permission checks being cached. Even if you have attachments visible to guests you could still have a request made by a banned user that results in an error page being cached (assuming there is no way to prevent caching if it's not a 200 status code, which I'm not aware of).

I wasn't aware that the externalDataUrl config value was limited to attachment thumbnails and not full attachments, I'm not sure if there's another config value where you can change the attachment path to a URL endpoint? We use the attachment store addon because we'd been using it since XF1, but I'd thought it was now possible to achieve similar results without an addon.

Not that I'm aware of, no. There is a suggestion for this floating around I believe. I'm definitely of the opinion that the S3 storage with XenForo isn't nearly as useful as I had hoped it would be since it actually results in a performance degradation serving them since it can't be cached, and XenForo has to download the image, and then serve it rather than serving it from a local file, or from an NFS share. Only real benefit to use it is that you don't have to worry about running out of disk space, the price will just increase based on usage, and it's fairly cheap compared to some other options.

If the only change XF were to make is an option to allow you to bypass permission checks on full attachments, and requiring the image hash or some other unique string to be present in the URL so somebody can't just scan IDs to find attachments from conversations or private messages, and allowing you to set a reasonable cache-control header it'd be much more powerful than what is currently possible. You'd get all the added functionality Cloudflare has as well, like the ability to automatically convert and serve optimized and webp images and

 

[Sam9]

Not that I'm aware of, no. There is a suggestion for this floating around I believe. I'm definitely of the opinion that the S3 storage with XenForo isn't nearly as useful as I had hoped it would be since it actually results in a performance degradation serving them since it can't be cached, and XenForo has to download the image, and then serve it rather than serving it from a local file, or from an NFS share. Only real benefit to use it is that you don't have to worry about running out of disk space, the price will just increase based on usage, and it's fairly cheap compared to some other options.

If the only change XF were to make is an option to allow you to bypass permission checks on full attachments, and requiring the image hash or some other unique string to be present in the URL so somebody can't just scan IDs to find attachments from conversations or private messages, and allowing you to set a reasonable cache-control header it'd be much more powerful than what is currently possible. You'd get all the added functionality Cloudflare has as well, like the ability to automatically convert and serve optimized and webp images and

Wow, yeah, I wasn't aware Xenforo's bundled method was so limited, unlimited storage is nice but it could be so much more.

The attachment store addon pretty much does exactly what you said, disables permission checks, and allows a custom URL/path/filename scheme to help prevent scanning. Although it's more designed at storing the attachments on an external S3/FTP service than serving the files local, it does look possible to do that too.