XF 2.1 2.1.5a - rebuild master data

[DarkIce2]

Is it necessary to click rebuild master data button after hitting /install URL when upgrading from 2.1.5 to 2.1.5a?

Also is it expected for my forum to still show a username / password prompt when hitting the /install URL after the upgrade has been completed?

 

[Paul B]

FYI, the ticket is still open and I have asked the dev's for more input.

 

[Kirby]

Would be quite interesting to know the cause once this is solved

@Groatz
Did you check if the phrases were missing at compile time or are missing at runtime?
In many cases, phrases are compiled into the templates.
So if phrases were missing at compile time, the templates will contain the variable names instead.

I'd check one of the affected compiled templates to see if it does contain the phrase names instead of the actual text.
1) If it does, this would indicate that the phrases were missing at compile time.
2) If it does not, it would indicate that the affected phrases are dynamically loaded and are somehow missing/do not load properly at runtime

In case 2) I'd check the DB for one of those phrases if it is actually missing.
2a) If it is missing this would indicated that it somehow got deleted
2b) If it is not missing it would indicated that somehow loading of the phrase does fail

In case 2a) I'd check MySQL binary logs to see when the phrase did get deleted or renamed and try to correlate events.
In case 2b) I'd check the code path for loading the phrase to see what is going wrong.

 

[Groatz]

Would be quite interesting to know the cause once this is solved

Agreed.

Did you check if the phrases were missing at compile time or are missing at runtime?
In many cases, phrases are compiled into the templates.
So if phrases were missing at compile time, the templates will contain the variable names instead.

The phrases are present, but displayed as the phrase name. That is, if the phrases title was "fu_bar" which should map to the obvious, instead it would literally display as "fu_bar". That's pretty common if we install an addon. Only comes good after master data rebuild. But there are other instances where we perhaps add something new ourselves and it will be the same, again until master data rebuild. So, that's really both templates and phrases affected both independently and collectively.

I'd check one of the affected compiled templates to see if it does contain the phrase names instead of the actual text.
1) If it does, this would indicate that the phrases were missing at compile time.
2) If it does not, it would indicate that the affected phrases are dynamically loaded and are somehow missing/do not load properly at runtime

Thinking about what you're saying I can say it's both compile time and runtime use cases where it is failing.

In case 2) I'd check the DB for one of those phrases if it is actually missing.
2a) If it is missing this would indicated that it somehow got deleted
2b) If it is not missing it would indicated that somehow loading of the phrase does fail

The phrases are present in the DB, not missing. Yes, it's the load (static or dynamic) that is failing.

In case 2a) I'd check MySQL binary logs to see when the phrase did get deleted or renamed and try to correlate events.
In case 2b) I'd check the code path for loading the phrase to see what is going wrong.

Will investigate further. Many thanks for the guided walkthrough. :-D

PS we had about 10k simultaneous users on tonight (not anywhere near our upper limits, but still lively) and the site didn't blink performance-wise. But we did do a master data rebuild a little earlier to ensure it was delivering clean content.

Regards, Groatz

 

[Kirby]

That's pretty common if we install an addon. Only comes good after master data rebuild. But there are other instances where we perhaps add something new ourselves and it will be the same, again until master data rebuild.

This definitly isn't normal. I don't know how your cluster is engineered/set up but this behaviour seems to indicate to me that there is an issue with session stickiness and propagating/syncing changes to all nodes.
Normally, if you change smth. (eg. update an Add-on, install a new one, change a phrase or template, etc.) those changes should ideally propagate in pretty much realtime to all nodes.
If this can't be guaranteed it would at least be necessary to ensure RYW consistency, eg. stick the active session to the node the has the latest data.

Thinking about what you're saying I can say it's both compile time and runtime use cases where it is failing.

Are you just thinking so or did you actually verify that this is the case?
When troubleshooting complex issues like this, guessing often doesn't help much - you need hard facts.

The phrases are present in the DB, not missing. Yes, it's the load (static or dynamic) that is failing.

If you have such a dynamic phrase that is failing on all nodes, did you actually try to track down where it is failing?
Eg. did you check that the queries are correct?
Did you manually execute them and got the phrase?
Did you step into the code (with XDebug, etc.)?

 

[Groatz]

This definitly isn't normal. I don't know how your cluster is engineered/set up but this behaviour seems to indicate to me that there is an issue with session stickiness and propagating/syncing changes to all nodes.
Normally, if you change smth. (eg. update an Add-on, install a new one, change a phrase or template, etc.) those changes should ideally propagate in pretty much realtime to all nodes.
If this can't be guaranteed it would at least be necessary to ensure RYW consistency, eg. stick the active session to the node the has the latest data.

As I've already said, syncing is not the issue. All nodes are kept in sync. The faults appear both on the master admin node and uniformly through the cluster.

Are you just thinking so or did you actually verify that this is the case?

Verified.

When troubleshooting complex issues like this, guessing often doesn't help much - you need hard facts.

That would be why your comments are the more useful so far.

If you have such a dynamic phrase that is failing on all nodes, did you actually try to track down where it is failing?
Eg. did you check that the queries are correct?
Did you manually execute them and got the phrase?
Did you step into the code (with XDebug, etc.)?

We are presently drawing the line at debugging the core XenForo (ie verifying code execution, queries, stepping through with a debugger). The workaround will get us by for now. The ticket with XenForo is still open and they deserve first shot at it.

Regards, Groatz

 

[Kirby]

Hmm, I would have started debugging this before opening a ticket to make sure that there acually is an issue with XenForo and not just our code/setup, but that might just be my personal preference.

It's been another day since the last update, any news so far?

 

[Groatz]

Hmm, I would have started debugging this before opening a ticket to make sure that there acually is an issue with XenForo and not just our code/setup, but that might just be my personal preference.

Given the only variable was the XF 2.1.5a upgrade, and nothing like this has ever happened before, I don't see why we would start there. Nobody else appears to be claiming they have this problem so there is no altruism driver either. We'll certainly look at that path if the XF devs are unable to offer anything.

It's been another day since the last update, any news so far?

Haven't heard any more from the XF folk as yet. If/when it's resolved (or even progress) I'll certainly provide an update.

Regards, Groatz

 

[Paul B]

I did respond to your ticket yesterday at midday.

I discussed it again with the developers and they have looked over the code and confirmed that there are no changes between 2.1.4 and 2.1.5 which would account for this.

The php files which were changed between the two releases have no relation with the code cache.

Furthermore, there are other customers who are running large installations with multiple servers who are not experiencing this issue - which suggests that fundamentally the XF code is working.

 

[Groatz]

I did respond to your ticket yesterday at midday.

Thanks!

I discussed it again with the developers and they have looked over the code and confirmed that there are no changes between 2.1.4 and 2.1.5 which would account for this.

The php files which were changed between the two releases have no relation with the code cache.

I see code cache has been thrown around repeatedly in this thread but I don't necessarily know if that is the actual problem area or not.

Furthermore, there are other customers who are running large installations with multiple servers who are not experiencing this issue - which suggests that fundamentally the XF code is working.

We'll continue with our workaround for the present and should we resolve the root cause will advise.

Regards, Groatz