XF 2.1 2.1.5a - rebuild master data

[DarkIce2]

Is it necessary to click rebuild master data button after hitting /install URL when upgrading from 2.1.5 to 2.1.5a?

Also is it expected for my forum to still show a username / password prompt when hitting the /install URL after the upgrade has been completed?

 

[Groatz]

Ever since the 2.1.5a we have needed to rebuild master data pretty much every day anyway. I don't recall about the username / password prompt, sorry. I can see how on the one hand it needs valid admin permissions so maybe you caught it as it needed to refresh credentials.

Regards, Groatz

 

[Jeremy P]

Is it necessary to click rebuild master data button after hitting /install URL when upgrading from 2.1.5 to 2.1.5a?

It shouldn't be, I believe the only changes were in PHP.

Ever since the 2.1.5a we have needed to rebuild master data pretty much every day anyway.

You should almost never need to rebuild master data, especially not on a regular basis.

 

[Groatz]

You should almost never need to rebuild master data, especially not on a regular basis.

Yeah, it would be brilliant if we didn't need to. But it's either that or put up with broken phrases and templates and unusable site.

Regards, Groatz

 

[Ozzy47]

Then there is something wrong with your installation or host.

 

[Groatz]

Then there is something wrong with your installation or host.

Yep, sadly true, but hopefully the next release of XF fixes whatever has broken in 2.1.5a. We're just going to run a nightly cron master data rebuild in the meantime.

Cheers, Groatz

 

[Ozzy47]

I honestly find it hard to believe it is because of a XF update. You would think if it was, many more people would be experiencing the same issue.

 

[Groatz]

I honestly find it hard to believe it is because of a XF update. You would think if it was, many more people would be experiencing the same issue.

Agreed. Hopefully more people will come forward so that something might be done about it.

Regards, Groatz

 

[Jeremy P]

I might be wrong, but I don't think it's something others are experiencing. I've been running 2.1.5a since release and I've never run into this issue or needed to rebuild the master data. I'm not sure what steps you've taken to troubleshoot, but I'd wager it's specific to something about your installation or server configuration, and not really a bug in XF itself.

 

[sbj]

It sounds like something went wrong with uploading the files, as it is happening since he switched. Try to upload the files again, be sure to overwrite them and restart your server again to make sure caching isn't a problem somehow.

 

[Paul B]

hopefully the next release of XF fixes whatever has broken in 2.1.5a

There is nothing to fix as far as we are aware, so most likely the issue you are experiencing will continue in subsequent releases.

I would suggest posting a thread in the troubleshooting forum with all of the details.

 

[Groatz]

I might be wrong, but I don't think it's something others are experiencing. I've been running 2.1.5a since release and I've never run into this issue or needed to rebuild the master data. I'm not sure what steps you've taken to troubleshoot, but I'd wager it's specific to something about your installation or server configuration, and not really a bug in XF itself.

Before XF 2.1.5a everything works As Intended. After XF 2.1.5a changes to addons (install/upgrade) or changes to things like Notices lead to problems relating to templates and phrases. Templates: out of sync, or not found. Phrases: bare (eg displaying phrase name rather than content). Reinstalling addons, reinstalling files, bouncing servers, resyncing servers - no effect. After trying every other conceivable approach and knowledge we have amassed over many years of using XenForo we rebuild master data - problems disappear immediately and then gradually reappear over the following days as we administer/moderate the site normally.

It sounds like something went wrong with uploading the files, as it is happening since he switched. Try to upload the files again, be sure to overwrite them and restart your server again to make sure caching isn't a problem somehow.

We have arguably one of the largest XenForo installations with a user base in the hundreds of thousands (up to 35k simultaneous visitors at times), many tens of millions of posts, 40+ autoscaled web servers, multiple load balancers, large RDS, CDN, ElasticSearch cluster, and other supporting infrastructure like remote monitoring systems) with dedicated sysops and custom devops staff in a controlled environment of dev, staging, and production systems. We use Nginx and PHP (php-fpm) 7.2 (about to move to 7.3 after it passes pre-prod stability testing). We have other live sites that we use for final pre-production Go/NoGo before upgrading our primary site, and these sites don't exhibit the issue although they are identically configured but much smaller. I say this just to set the context that yes we have a lot of familiarity with XF and common troubleshooting steps. I'm sure we have a lot more to learn but we are generally aware when there is a fault.

But yes, uploading files and restarting the (all) servers has been tried. Amongst many other things.

Sure it may just be our unique setup that broke at the exact moment when we upgraded our primary site to XF 2.1.5a after not having seen anything like this in the last decade or so back to XF 1 and vBulletin before that.

Regards, Groatz

 

[sbj]

If your setup is that large indeed, then it means you have potentially thousands of variations of problems waiting to happen. So I would bet against your setup vs. XF 2.1.5a being faulty.
All I can say is that I don't want to be in your position to find the needle in the haystack. Good luck.

 

[Slavik]

I would hazard a guess this is going to be an issue with the front end caching going wrong somewhere, if its not manifesting in the dev setups where the caches likely never get multiple refreshes or similar.

 

[Jeremy P]

Sure it may just be our unique setup that broke at the exact moment when we upgraded our primary site to XF 2.1.5a after not having seen anything like this in the last decade or so back to XF 1 and vBulletin before that.

It could be something which manifested as a result of the upgrade but was not caused by a defect in the update itself. When it occurs, do your database tables contain the ostensibly missing records? I'm not sure how you're managing your code_cache, but issues there could result in missing or outdated templates as well.

Maybe it's not worth speculating given the complexity of your setup and the competence of the team around it, but it seems like it would at least be worth debugging to identify what exactly is going wrong so that you can narrow in on a potential cause. It doesn't sound like an issue that's been reported by anyone else, so I wouldn't hold out much hope for a fix otherwise.

 

[Groatz]

The symptom almost certainly stems from the code_cache, but we manage the synchronisation of that reasonably well. Our system is well-engineered and constantly being refined with detailed monitoring and zero tolerance to errors. It rarely just "breaks" for no reason, quite the opposite.

It would seem that the corruption is being injected upstream of the code_cache, and by rebuilding master data the corruption is (temporarily) alleviated. We now going to rebuild the master data every night. We run hot 23x7 so there is about an hour when relatively few users are impacted by the rebuild.

It will have to do for the time being. Note that my original reply was simply a response to the question by @DarkIce2 about master data where, in our circumstance, it is necessary. Your own mileage may vary.

Regards, Groatz

 

[Ozzy47]

Before XF 2.1.5a everything works As Intended. After XF 2.1.5a changes to addons (install/upgrade) or changes to things like Notices lead to problems relating to templates and phrases. Templates: out of sync, or not found. Phrases: bare (eg displaying phrase name rather than content). Reinstalling addons, reinstalling files, bouncing servers, resyncing servers - no effect. After trying every other conceivable approach and knowledge we have amassed over many years of using XenForo we rebuild master data - problems disappear immediately and then gradually reappear over the following days as we administer/moderate the site normally.



We have arguably one of the largest XenForo installations with a user base in the hundreds of thousands (up to 35k simultaneous visitors at times), many tens of millions of posts, 40+ autoscaled web servers, multiple load balancers, large RDS, CDN, ElasticSearch cluster, and other supporting infrastructure like remote monitoring systems) with dedicated sysops and custom devops staff in a controlled environment of dev, staging, and production systems. We use Nginx and PHP (php-fpm) 7.2 (about to move to 7.3 after it passes pre-prod stability testing). We have other live sites that we use for final pre-production Go/NoGo before upgrading our primary site, and these sites don't exhibit the issue although they are identically configured but much smaller. I say this just to set the context that yes we have a lot of familiarity with XF and common troubleshooting steps. I'm sure we have a lot more to learn but we are generally aware when there is a fault.

But yes, uploading files and restarting the (all) servers has been tried. Amongst many other things.

Sure it may just be our unique setup that broke at the exact moment when we upgraded our primary site to XF 2.1.5a after not having seen anything like this in the last decade or so back to XF 1 and vBulletin before that.

Regards, Groatz

Well given that, It is entirely possible that due to the size of the site it has exposed something that most sites won't come across. If feasible I would start a ticket and provide access to the XF staff so they can investigate to find out if you have triggered some sort of bug.

 

[Slavik]

Well given that, It is entirely possible that due to the size of the site it has exposed something that most sites won't come across. If feasible I would start a ticket and provide access to the XF staff so they can investigate to find out if you have triggered some sort of bug.

XenForo doesnt really care if its being run on 1 server or 100, as long as the config respects a few basic fundamentals. This issue is simply likely to be a faulty hosting node in play somewhere. I manage multiple sites for customers on multi server setups and none of them have experienced this issue.

 

[Kirby]

We are also running XF on multiple nodes (though not 40) and havn't seen such an issue.
Normally, such things happen when nodes get out of sync somehow.

Have you tried to track down which node(s) are devlivering those broken templates?
Did you check that code_cache is consistent across all nodes?

 

[Groatz]

We are also running XF on multiple nodes (though not 40) and havn't seen such an issue.
Normally, such things happen when nodes get out of sync somehow.

Have you tried to track down which node(s) are devlivering those broken templates?
Did you check that code_cache is consistent across all nodes?

Issue is consistent across all nodes, including the admin server (which is back from the main rotation, only admins use it). Yes, we sync every couple of minutes and this is verified. Which again leads me to believe that the problem is happening at the XF level rather than the mechanics of keeping in sync. @Brogan has advised us that the XF devs do not believe any changes in XF 2.1.5a would cause this. So it is still not empirically definitive whether the upgrade was causal or just coincidental to this issue.

We're going to build a new AMI (yes we're on AWS) for the cluster to reduce one variable (ie freshen up Nginx, PHP-FPM, and the initial server sync). We can also consider completing the PHP 7.3 upgrade around the same time. That will rule out some potential problems perhaps. We've never had anything like this before that we couldn't solve. :-/

Regards, Groatz