What caching configurations do you all use?

[SFFNetwork]

We're currently exploring how we can leverage caching to improve performance on our forum, and reduce server load. That said, I have limited experience with caching and no experience configuring the same for XenForo.

Given that, I was curious if some folks would share their thoughts on what collection of caches/providers they've gone with, and why. Hopefully this can be helpful to us and others as we try to figure out what implementation might suit our forum best. )At the moment we are considering APC + memcached.)

Thanks!

 

[briansol]

Redis with Xon's addon

If i ever need anything more than that, it's going to cloudflare guest caching

 

[eva2000]

Several layers

• Cloudflare CDN Cache via custom Cloudflare Worker guest full HTML page caching + CF bypass cache on cookie page rules
• Custom PWA service worker pre-caching and navigation caching
• Centmin Mod Nginx with PHP-FPM fastcgi_cache guest full HTML page micro caching
• Xon Redis cache addon for Redis server caching
• Zend Opcache for PHP opcode caching

did I say I love caching

 

[Alpha1]

Cloudflare CDN Cache via custom Cloudflare Worker guest full HTML page caching + CF bypass cache on cookie page rules

Could you please expand on this one?

 

[JoyFreak]

Redis with Xon's addon

If i ever need anything more than that, it's going to cloudflare guest caching

I was made to believe that Xon’s redis add-on only makes a difference on bigger sites.

 

[eva2000]

Could you please expand on this one?

I whipped up a custom Cloudflare Worker that tells Cloudflare on a per forum dynamically generated HTML page url and extension basis what to cache at Cloudflare CDN edge server and for how long for guest visitors which is cache bypassed for logged users.

I also dynamically vary cache TTL based on my site's off peak and peak visitor times so peak hour has lower cache TTL than off peak.

So I can have a group of sticky closed forum threads that rarely update set with 30 day cache TTL expiry while frequent posted/replied to threads have shorter cache TTL like 2 to 5 mins at peak hour and 15 to 30 mins at off peak times. I can even set cache TTL on a per hour time of day basis.

Or that I know my own avatar won't be updated frequently so I set cache TTL much higher than other members' avatar images which has a shorter cache TTL

Think of it as Varnish cache or Xenforo full page cache just all the caching is done on Cloudflare CDN edge servers and with finer grain cache logic control.

Cloudflare Workers can also via CF-priority header tune the HTTP/2 priorities of individual assets to tell web browsers what to load and when in relation to other assets. Though Cloudflare's HTTP/2 prioritization configuration is already one of the best https://blog.cloudflare.com/better-http-2-prioritization-for-a-faster-web/ - it can sometimes be tuned better for specific pages.

If a Worker adds a “cf-priority” header to the response, Cloudflare edge servers will use the specified priority and concurrency for that response. The format of the header is <priority>/<concurrency> so something like response.headers.set('cf-priority', “30/0”); would set the priority to 30 with a concurrency of 0 for the given response. Similarly, “30/1” would set concurrency to 1 and “30/n” would set concurrency to n.

With this level of flexibility a site can tweak resource prioritization to meet their needs. Boosting the priority of some critical async scripts for example or increasing the priority of hero images before the browser has identified that they are in the viewport

You can see all this in action as a guest visitor on my forums at https://community.centminmod.com/ or on my Wordpress blog at https://blog.centminmod.com/

 

[eva2000]

I was made to believe that Xon’s redis add-on only makes a difference on bigger sites.

Benefits are indeed relative to your forums. But that is for performance in terms of response time latency. If you goal is to offload some requests from your backend to Redis in general regardless of response time latency, then Xon's redis addon will help i.e. in situations where you suddenly get a huge spike in legit/attack traffic like patten requests. Though in such cases utilising Cloudflare CDN level cache to offload from your server would be a better option

 

[ddrager]

I whipped up a custom Cloudflare Worker that tells Cloudflare on a per forum dynamically generated HTML page url and extension basis what to cache at Cloudflare CDN edge server and for how long for guest visitors which is cache bypassed for logged users.

I also dynamically vary cache TTL based on my site's off peak and peak visitor times so peak hour has lower cache TTL than off peak.

So I can have a group of sticky closed forum threads that rarely update set with 30 day cache TTL expiry while frequent posted/replied to threads have shorter cache TTL like 2 to 5 mins at peak hour and 15 to 30 mins at off peak times. I can even set cache TTL on a per hour time of day basis.

Or that I know my own avatar won't be updated frequently so I set cache TTL much higher than other members' avatar images which has a shorter cache TTL

Think of it as Varnish cache or Xenforo full page cache just all the caching is done on Cloudflare CDN edge servers and with finer grain cache logic control.

Cloudflare Workers can also via CF-priority header tune the HTTP/2 priorities of individual assets to tell web browsers what to load and when in relation to other assets. Though Cloudflare's HTTP/2 prioritization configuration is already one of the best https://blog.cloudflare.com/better-http-2-prioritization-for-a-faster-web/ - it can sometimes be tuned better for specific pages.



You can see all this in action as a guest visitor on my forums at https://community.centminmod.com/ or on my Wordpress blog at https://blog.centminmod.com/

Hello again! We worked together years ago on vB things. Our forum just moved to XenForo

I am probably looking to get Cloudflare caching set up like you said. I would prefer to handle it with page rules, but this only supports invalidating cache by cookie. I think in order to set up good caching, will need to use workers as you have said. Doing this we can evaluate http headers and can probably add cache keys for thread id, node id, and more - so we can expire them when posts are updated.

 

[eva2000]

Hello again! We worked together years ago on vB things. Our forum just moved to XenForo

I am probably looking to get Cloudflare caching set up like you said. I would prefer to handle it with page rules, but this only supports invalidating cache by cookie. I think in order to set up good caching, will need to use workers as you have said. Doing this we can evaluate http headers and can probably add cache keys for thread id, node id, and more - so we can expire them when posts are updated.

Ah that username looks familiar so did an email history search, hey David It's been like 8 years! Yes, I saw that your forums finally moved off heavily modified vBulletin - that must of been one enormous task! I logged into your forum with my user account and had a look around and nice and refreshing change. Even played of your new Xenforo mobile app too

For Cloudflare caching, custom cache keys would be a Cloudflare Enterprise plan feature though https://support.cloudflare.com/hc/en-us/articles/115004290387-Creating-Cache-Keys

Though with Cloudflare Enterprise plan, you can also do cache prefetching to warm up all 200+ Cloudflare datacenter's caches for a higher cache hit rate https://support.cloudflare.com/hc/en-us/articles/206776707-Does-Cloudflare-Do-Prefetching- and Enterprise plans have Argo Tiered Caching enabled by default even when Argo itself is not enabled

FYI, for past 2 years, I've been an official Cloudflare community MVP (similar to Microsoft MVP) so get play with more Cloudflare existing and upcoming product/features. So I can be a bit biased but been a Cloudflare customer for 9+ years

 

[Xon]

Xon Redis cache addon for Redis server caching

I've got a new redis caching related add-on . It can save a hilarious amount of MySQL server bandwidth for sites with large numbers of forums depending on site usage patterns

 

[eva2000]

I've got a new redis caching related add-on . It can save a hilarious amount of MySQL server bandwidth for sites with large numbers of forums depending on site usage patterns

oooh exciting - maybe a redis with zstd compression based caching ?

I'm ready for such

php --ri redis

redis

Redis Support => enabled
Redis Version => 5.3.2
Redis Sentinel Version => 0.1
Available serializers => php, json
Available compression => zstd, lz4

 

[Xon]

oooh exciting - maybe a redis with zstd compression based caching ?

I'm ready for such

zstd support was added back in Redis Cache v2.3.0 in 2018-09-25

Enabling it for new keys is very simple, also support negative and positive zstd compression levels

$config['cache']['config']['compression_lib'] = 'zstd';

Existing keys will be decoded with the old version, so you can drop in replace new compression methods. Just don't remove the extension!

 

[djbaxter]

I've got a new redis caching related add-on . It can save a hilarious amount of MySQL server bandwidth for sites with large numbers of forums depending on site usage patterns

Do you have an ETA for this?

 

[m1ne]

LSCache
Zend Opcache
Redis
CloudFlare Workers + full page HTML caching for guests

The results are... good.

 

[eva2000]

zstd support was added back in Redis Cache v2.3.0 in 2018-09-25

DOH that was ages ago too. So this is at zstd PHP extension level and not redis zstd ?

php --ri zstd

zstd

Zstd support => enabled
Extension Version => 0.9.0
Interface Version => 1.4.5

CloudFlare Workers + full page HTML caching for guests

Very nice

 

[Xon]

DOH that was ages ago too. So this is at zstd PHP extension level and not redis zstd ?

Yeah, my Redis cache add-on supports using various compressors (ie zstd php extensio)n to compress data before it sends it to redis.

Note; css via css.php is always gzip compressed. This allows it to just output the gzip'ed data if the client has support for it without needing to decompress it locally.

 

[eva2000]

Note; css via css.php is always gzip compressed. This allows it to just output the gzip'ed data if the client has support for it without needing to decompress it locally.

Thanks for the info !

Yeah, my Redis cache add-on supports using various compressors (ie zstd php extensio)n to compress data before it sends it to redis.

Makes sense as not all redis servers or redis extensions might have zstd support.

 

[ddrager]

I've got a new redis caching related add-on . It can save a hilarious amount of MySQL server bandwidth for sites with large numbers of forums depending on site usage patterns

I can confirm

 

[eva2000]

I can confirm

View attachment 242101

wow - scale - GB or TB ?

 

[ddrager]

wow - scale - GB or TB ?

I'm still trying to get the scale working right on monitoring but I believe that is 100s of mbps. So DB traffic went from 200-300mbps to 50-70mbps