Cloudflare optimizations for XenForo

[digitalpoint]

Seems so.

 

[duderuud]

Force a challenge to bots that don't adhere to robots.txt based on their user agent.

Hi @digitalpoint , can you explain how to do this exactly?

 

[digitalpoint]

Hi @digitalpoint , can you explain how to do this exactly?

You can make a WAF rule that presents a managed challenge based on whatever user agents you want.

 

[duderuud]

Oh yeah okay, so you need to create a rule manually when you encounter a bad bot.

 

[digitalpoint]

Oh yeah okay, so you need to create a rule manually when you encounter a bad bot.

Ya... the bots you are dealing with aren't necessarily the same bots that other sites deal with.



I like to keep the stats separate, so I have a rule for referrer spam bots (based on the URLs they are requesting) and a different rule for spiders ignoring robots.txt (based on user agent). Since the action is the same, you could merge those into a single rule, but like I said... I like to keep stats separate.

 

[80sDude]

Are these settings that you recommend still the best way to go? I downloaded your recent update so I’m just curious. Btw thank you for your hard work.

 

[digitalpoint]

Are these settings that you recommend still the best way to go? I downloaded your recent update so I’m just curious. Btw thank you for your hard work.

Mostly, ya.

 

[Wildcat Media]

Page Rules:

• For a URL match of *YourXenForoInstallation.com/css.php?css=*, give it a long Browser Cache TTL, set Cache Level to "Cache Everything", give it a long Edge Cache TTL and enable Origin Cache Control. This will allow XenForo's CSS to be cached in user's browser and the network edge.
• You can do something similar for XenForo's proxied images, but if you do it, you lose the counter XenForo keeps for how many times proxied images are viewed by users. So if you care about that, don't do it.

There is probably a technical reason for not having done this, but couldn't JS also be cached?

<script src="/js/vendor/jquery/jquery-3.5.1.min.js?_v=953c9c67"></script>
<script src="/js/vendor/vendor-compiled.js?_v=953c9c67"></script>
<script src="/js/xf/core-compiled.js?_v=953c9c67"></script>
<script src="/js/xf/notice.min.js?_v=953c9c67"></script>

In other words, using a CF expression like (http.request.uri contains "/js/xf/") or (http.request.uri contains "/js/vendor/") under Cache Rules. Or even just (http.request.uri contains "/js/").

 

[digitalpoint]

JavaScript is cached by default and will adhere to your web server's Cache-Control directives, so you don't need to setup anything for JavaScript normally (you may need to if you have an incorrectly configured web server instructing Cloudflare to specifically not cache JavaScript for some reason).

The CSS in XenForo does need a specific rule though because it has a .php extension (which Cloudflare will not cache by default even if the web server tells it to).

See: https://developers.cloudflare.com/cache/concepts/default-cache-behavior/

Default cached file extensions

Cloudflare only caches based on file extension and not by MIME type. The Cloudflare CDN does not cache HTML by default.

 

[Wildcat Media]

JavaScript is cached by default and will adhere to your web server's Cache-Control directives, so you don't need to setup anything for JavaScript normally

That makes sense. Thanks!

 

[Wildcat Media]

I'm configuring Cloudflare on a much larger site, now that I've been able to get the nameservers switched over to CF. It's been a busy few days making sure everything is set up. (I've had to do it manually as I don't yet have XF2 on that site yet--still finishing up the work for an eventual upgrade from 1.5.)

So once I got the forum's attachments and CSS cached, I've noticed a definite upward trend in what is being cached over the past four hours.

Is there a typical range of cached vs. non-cached requests once everything is set up properly? For the past hour we've had ~71% of requests cached. I don't think there is much more we can eek out of it. I'll know more once we've let it run for a week or so.

 

[Gizmoandsooz]

Cheers for this.

 

[digitalpoint]

All sites are different. Depends on traffic and what sort of content you have. Best thing is just look at your web logs and see if you serving stuff that could be cached.

 

[Kleazy]

All sites are different. Depends on traffic and what sort of content you have. Best thing is just look at your web logs and see if you serving stuff that could be cached.

What I do is look at CF dashboard and filter search based on hit/miss and look at most requests and bandwidth files, file types, path etc to find to biggest non cache offenders and then troubleshoot.
Can it be cached? Should it be? Why is it not?
Then apply .htaccess rules or create a CF cache rule.

Test, monitor, rinse and repeat.

 

[AndrewSimm]

I'm configuring Cloudflare on a much larger site, now that I've been able to get the nameservers switched over to CF. It's been a busy few days making sure everything is set up. (I've had to do it manually as I don't yet have XF2 on that site yet--still finishing up the work for an eventual upgrade from 1.5.)

So once I got the forum's attachments and CSS cached, I've noticed a definite upward trend in what is being cached over the past four hours.

Is there a typical range of cached vs. non-cached requests once everything is set up properly? For the past hour we've had ~71% of requests cached. I don't think there is much more we can eek out of it. I'll know more once we've let it run for a week or so.

I am running around 90% cached for the month. Not sure if that is good or bad in terms of CF users but it's a lot better than it was before.

 

[Wildcat Media]

What I do is look at CF dashboard and filter search based on hit/miss and look at most requests and bandwidth files, file types, path etc to find to biggest non cache offenders and then troubleshoot.

Are you on the Pro plan? (Or can't I find these stats?)

 

[mjda]

Are you on the Pro plan? (Or can't I find these stats?)

You'll find these in the admin control panel, if you have the App for Cloudflare® add-on installed and the "Show Cloudflare statistics" setting enabled.

 

[Wildcat Media]

You'll find these in the admin control panel, if you have the App for Cloudflare® add-on installed and the "Show Cloudflare statistics" setting enabled.

I see the graphs, but I'm not finding the details behind them, like in the post I quoted.

 

[mjda]

I see the graphs, but I'm not finding the details behind them, like in the post I quoted.

Doh! I automatically assumed you were replying to the post right above yours. I actually don't see those filters either. I'm assuming they're part of the "Cache Analytics" that are advertised as a perk to upgrading to Pro.

 

[Wildcat Media]

Doh! I automatically assumed you were replying to the post right above yours. I actually don't see those filters either. I'm assuming they're part of the "Cache Analytics" that are advertised as a perk to upgrading to Pro.

No worries! I agree, it seems to be a Pro or higher account level that would provide those stats.

In a few months I might subscribe to Pro for a month, just to get access to those stats so I can make adjustments. As I've just set up Cloudflare on this site I'm working on, I'd like to let things settle in for a few months.