Let’s make XenForo great: make thread pages cacheable

[Stockwalker]

I’m an avid user of xF however I’m disappointed to see that caching is not supported at all. It would actually give quite a big performance boost to statically render pages behind forum/ and threads/. Yes, there is a reason that this is not supported, and not all setups might accommodate this, but as a CloudFlare user this could be easily toggled on.

Cached pages can always be busted whenever there is a new post. So that’s not an excuse to not have this, neither the fact that “dynamic content” needs to be generated first. Come on, it’s 2022.

So my question is, why is this feature not supported out of the box? Are there any plans to do this ? Fact is that statically pre rendered pages are all the rage nowadays and xF could benefit a lot from having this feature.

 

[tristan9]

Like I said, would have to do a couple minor tweaks/URL exclusions with their cache mechanism, but it's nothing that couldn't be worked out very easily at the network level...

Last I checked it didn't seem that straightforward where exactly the CSRF was required or not, but yes that is what I had tried. I'll try again; maybe I'd originally messed something up.

But yes the reasons you mention are exactly why I'm interested in general.

 

[tristan9]

Yup so after checking again, it is not that straightforward.

XF wants the CSRF token updated quite regularly (couple of minutes at most, by the looks of it), so a user can end up with a stale CSRF cookie fairly easily if they go from cached page to cached page for a little bit.

At that point random guest features fail:

• member hover tooltips (it's an ajax GET that doesn't handle the CSRF token having expired)
• doing a search (it's a direct POST)

This will happen until the guest somehow hits an uncached page/endpoint that sets a new CSRF cookie. Something like trying to login, the login/keep-alive ajax call, etc...

So as outlined, the problem that CSRF tokens are not refreshed on-demand remains, as far as I can tell.

 

[imno007]

I know that the top paying web design firm in the world couldn't afford me. A "country pumpkin local web design shop" would need to sell their souls and multiple generations of their families to get me to do web design (actually probably not even then).

Pretty sure he was joking. You might have got that, but it didn't seem like it, judging from your response

 

[Arantor]

Pretty sure he was joking.

Pretty sure he wasn't.

 

[digitalpoint]

Yup so after checking again, it is not that straightforward.

XF wants the CSRF token updated quite regularly (couple of minutes at most, by the looks of it), so a user can end up with a stale CSRF cookie fairly easily if they go from cached page to cached page for a little bit.

At that point random guest features fail:

• member hover tooltips (it's an ajax GET that doesn't handle the CSRF token having expired)
• doing a search (it's a direct POST)

This will happen until the guest somehow hits an uncached page/endpoint that sets a new CSRF cookie. Something like trying to login, the login/keep-alive ajax call, etc...

So as outlined, the problem that CSRF tokens are not refreshed on-demand remains, as far as I can tell.

Yep... there are a few little things here and there that would need to be worked out. Nothing insurmountable though. I just mucked around with it a little bit as a test and I was able to get Cloudflare to cache pages for guests without any issues or XenForo-side changes.

You are correct in that CSRF would need to be worked out for AJAX and POST requests. A couple solutions off the top of my head (or a mix of them would work):

• You could disable CSRF checks for guests (probably not the best thing to do from a security standpoint, but it could be done).
• You could make some of the AJAX requests be normal links rather than AJAX requests (login, registration, member pop-ups, etc.)
• If you wanted to make something where XenForo really worked as intended with AJAX and stuff, you could inject a request to the guest page views that runs behind the scenes (an AJAX request that effectively gets/updates CSRF tokens and then use that token on the cached pages).

The last option is probably what I would do if I were going to do it (a little more work, but user experience would be better). Fast/cached HTML pages in the closest Cloudflare data center to the user with a small HTTP request going back to the server for the CSRF token. The server would basically just need to manage CSRF tokens for the guests is all.

Whole thing could be rolled into a relatively simple XenForo add-on with some config options like how long you want to cache guest pages and stuff. I'm half tempted to roll it into my Cloudflare add-on, although I don't have a need for it myself, so not high on my to-do list. But it would be a fun little thing to do.

Either way, it wouldn't be hard at all.

 

[Slavik]

Pretty sure he was joking. You might have got that, but it didn't seem like it, judging from your response

I can assure you @digitalpoint knew I was being sarcastic

 

[Stockwalker]

He started it. As a customer I posted it to receive advice as I genuinely thought this feature would be useful. What I received instead is being criticised for making a honest typo and a lecture from a sarcastic dude about how I should stick to ancient software and what a “dynamic website” is. Then a whole army of XF fanbois attacked me. Nice community. Yeah I won’t be apologising.

Ironically it’s the same user who initially mocked me who actually gave a response that’s worthy of something so I guess thanks for that

 

[Arantor]

Well, you were the one going round asserting that other people don’t know anything. Me, I’m a relative nobody who has managed sites with 10,000 concurrent users, digitalpoint maintains the main package for CloudFlare for XF, and I’d say that between us we might know a thing or two… but you decided that we’re Luddites. I just laughed and left you to it, digitalpoint responded how he saw fit to what was an insult.

 

[Stockwalker]

Well, you were the one going round asserting that other people don’t know anything. Me, I’m a relative nobody who has managed sites with 10,000 concurrent users, digitalpoint maintains the main package for CloudFlare for XF, and I’d say that between us we might know a thing or two… but you decided that we’re Luddites. I just laughed and left you to it, digitalpoint responded how he saw fit to what was an insult.

Well, its pretty obvious that he was the one who started the spitting, obviously no manners, so yeah give disrespect get disrespect. I posted the question in good faith, it doesn’t matter if this guy is the master of the universe or just a peasant. You got that?

Also, his package DOES NOT include my proposed feature, only cookie cutter ones. He can start being a tough guy by implementing this feature rather than flaming strangers online. Thanks.

 

[Arantor]

Yes, you gave disrespect and got it back.

 

[digitalpoint]

Yup so after checking again, it is not that straightforward.

XF wants the CSRF token updated quite regularly (couple of minutes at most, by the looks of it), so a user can end up with a stale CSRF cookie fairly easily if they go from cached page to cached page for a little bit.

At that point random guest features fail:

• member hover tooltips (it's an ajax GET that doesn't handle the CSRF token having expired)
• doing a search (it's a direct POST)

This will happen until the guest somehow hits an uncached page/endpoint that sets a new CSRF cookie. Something like trying to login, the login/keep-alive ajax call, etc...

So as outlined, the problem that CSRF tokens are not refreshed on-demand remains, as far as I can tell.

Had a bit of free time while dinner cooks and I took a shot at it. I think I have it working. Everything works as normal (AJAX/POST, etc.) and it's being live tested on my iolabs.io site... network edge caching (via Cloudflare) for 1 hour (didn't want to set it too high in case there's something unforeseen). But so far it seems to work.

For logged in (non-guest) users, everything works as normal (everything is dynamic).

 

[rdn]

Had a bit of free time while dinner cooks and I took a shot at it. I think I have it working. Everything works as normal (AJAX/POST, etc.) and it's being live tested on my iolabs.io site... network edge caching (via Cloudflare) for 1 hour (didn't want to set it too high in case there's something unforeseen). But so far it seems to work.

For logged in (non-guest) users, everything works as normal (everything is dynamic).

Is this by design or bug?
I can see that even 403 pages (the page behind login) are cached on CF.

 

[digitalpoint]

By design right now, but there’s a few things still I need to think about.

 

[digitalpoint]

Ended up making a few changes to it this morning, but overall it seems to be working quite well. Just ran a test on GTmetrix from Canada (the server is physically in southwest US), and this is what the results are:

Time to first byte: 38ms
First contentful paint: 316ms
Largest contentful paint: 316ms
Time to interactive: 458ms
Onload time: 459ms

GTmetrix Performance Report

GTmetrix Grade: A (Performance: 100% / Structure: 98%)

gtmetrix.com

The interesting thing about the time to first byte is that if you didn't need to do DNS queries or SSL handshakes (which a guest user clicking around wouldn't need to do), the time to first byte gets down to around 15ms.

 

[tristan9]

Whole thing could be rolled into a relatively simple XenForo add-on with some config options like how long you want to cache guest pages and stuff. I'm half tempted to roll it into my Cloudflare add-on, although I don't have a need for it myself, so not high on my to-do list. But it would be a fun little thing to do.

First, thanks for actually giving it a shot and looking at it for real, then reporting on it.

Also, and maybe it's my misreading of tone in text, but something irks me from your answers. I never said it was impossible (or even fiendishly difficult or anything) for someone with good knowledge of XF's internals. But merely that it really does not work out of the box currently. Also, maybe XF internals knowledge is entirely unnecessary too, but I'd definitely want to dig quite a bit to understand XF's security model before playing around with its cookie management behaviour, at least.

As a result I was supporting what I see as a good suggestion for changing the default behaviour of XF towards supporting this by default. Especially because it would have no drawbacks (that I can think of anyway) for current users, and make it easier for everyone interested.

Anyway, by the looks of it, this is the snippet you added to make it all work fine (by forcing a login keep-alive query), if I am not mistaken?

<script>$(document).on('xf:page-load-start',function(e,i){XF.config.time.now=Math.floor(new Date().getTime()/1000)});$(document).on('xf:page-load-complete',function(e,i){if(XF.Cookie.get('csrf')){XF.KeepAlive.refresh()}});</script>

The interesting thing about the time to first byte is that if you didn't need to do DNS queries or SSL handshakes (which a guest user clicking around wouldn't need to do), the time to first byte gets down to around 15ms.

I mean, that's what the both op and I kept saying. I assume you say so for the benefit of everyone else perhaps. But yes, this is a legitimate day and night difference even in close geographical loaction, and straight up an entire world of difference if you mix in network latency around the globe.

 

[digitalpoint]

Correct, it does not work out of the box. It’s also more than that bit of JavaScript of course (there’s also some PHP changes as well as some Cloudflare config changes). I’m not sure it would be practical for XF to have something like that in their core because it requires the site to use Cloudflare in order to work. It of course could be implemented a different way, but if http requests need to go all the way to the server (and you aren’t caching at the network edge) you lose most of the benefits.

But if they do, great… it could benefit some sites.

 

[K a M a L]

Correct, it does not work out of the box. It’s also more than that bit of JavaScript of course (there’s also some PHP changes as well as some Cloudflare config changes). I’m not sure it would be practical for XF to have something like that in their core because it requires the site to use Cloudflare in order to work. It of course could be implemented a different way, but if http requests need to go all the way to the server (and you aren’t caching at the network edge) you lose most of the benefits.

But if they do, great… it could benefit some sites.

This requires "Bypass Cache on Cookie" which is only available to Business and Enterprise plans?

 

[digitalpoint]

This requires "Bypass Cache on Cookie" which is only available to Business and Enterprise plans?

Nope, works on any (including Free) plan. The Bypass Cache on Cookie method is a very old method (I think they introduced it in 2016). The way I’m doing it doesn’t require any Page Rules.

 

[K a M a L]

Nope, works on any (including Free) plan. The Bypass Cache on Cookie method is a very old method. The way I’m doing it doesn’t require any Page Rules.

That's intersting .. when do you expect to release it ?

 

[digitalpoint]

A few people are using it as a beta version, but no issues have popped up yet. So probably not too far down the road.