XF 2.1 2.1.5a - rebuild master data
- Thread starter DarkIce2
- Start date
Groatz
Member
Ever since the 2.1.5a we have needed to rebuild master data pretty much every day anyway. I don't recall about the username / password prompt, sorry. I can see how on the one hand it needs valid admin permissions so maybe you caught it as it needed to refresh credentials.
Regards, Groatz
Regards, Groatz
It shouldn't be, I believe the only changes were in PHP.
You should almost never need to rebuild master data, especially not on a regular basis.
Groatz
Member
Yeah, it would be brilliant if we didn't need to. But it's either that or put up with broken phrases and templates and unusable site.
Regards, Groatz
Groatz
Member
Yep, sadly true, but hopefully the next release of XF fixes whatever has broken in 2.1.5a. We're just going to run a nightly cron master data rebuild in the meantime.
Cheers, Groatz
Groatz
Member
Agreed. Hopefully more people will come forward so that something might be done about it.
Regards, Groatz
I might be wrong, but I don't think it's something others are experiencing. I've been running 2.1.5a since release and I've never run into this issue or needed to rebuild the master data. I'm not sure what steps you've taken to troubleshoot, but I'd wager it's specific to something about your installation or server configuration, and not really a bug in XF itself.
Groatz
Member
Before XF 2.1.5a everything works As Intended. After XF 2.1.5a changes to addons (install/upgrade) or changes to things like Notices lead to problems relating to templates and phrases. Templates: out of sync, or not found. Phrases: bare (eg displaying phrase name rather than content). Reinstalling addons, reinstalling files, bouncing servers, resyncing servers - no effect. After trying every other conceivable approach and knowledge we have amassed over many years of using XenForo we rebuild master data - problems disappear immediately and then gradually reappear over the following days as we administer/moderate the site normally.
We have arguably one of the largest XenForo installations with a user base in the hundreds of thousands (up to 35k simultaneous visitors at times), many tens of millions of posts, 40+ autoscaled web servers, multiple load balancers, large RDS, CDN, ElasticSearch cluster, and other supporting infrastructure like remote monitoring systems) with dedicated sysops and custom devops staff in a controlled environment of dev, staging, and production systems. We use Nginx and PHP (php-fpm) 7.2 (about to move to 7.3 after it passes pre-prod stability testing). We have other live sites that we use for final pre-production Go/NoGo before upgrading our primary site, and these sites don't exhibit the issue although they are identically configured but much smaller. I say this just to set the context that yes we have a lot of familiarity with XF and common troubleshooting steps. I'm sure we have a lot more to learn but we are generally aware when there is a fault.
But yes, uploading files and restarting the (all) servers has been tried. Amongst many other things.
Sure it may just be our unique setup that broke at the exact moment when we upgraded our primary site to XF 2.1.5a after not having seen anything like this in the last decade or so back to XF 1 and vBulletin before that.
Regards, Groatz
sbj
Well-known member
If your setup is that large indeed, then it means you have potentially thousands of variations of problems waiting to happen. So I would bet against your setup vs. XF 2.1.5a being faulty.
All I can say is that I don't want to be in your position to find the needle in the haystack. Good luck.
All I can say is that I don't want to be in your position to find the needle in the haystack. Good luck.
It could be something which manifested as a result of the upgrade but was not caused by a defect in the update itself. When it occurs, do your database tables contain the ostensibly missing records? I'm not sure how you're managing your
code_cache, but issues there could result in missing or outdated templates as well.Maybe it's not worth speculating given the complexity of your setup and the competence of the team around it, but it seems like it would at least be worth debugging to identify what exactly is going wrong so that you can narrow in on a potential cause. It doesn't sound like an issue that's been reported by anyone else, so I wouldn't hold out much hope for a fix otherwise.
Last edited:
Groatz
Member
The symptom almost certainly stems from the
It would seem that the corruption is being injected upstream of the
It will have to do for the time being. Note that my original reply was simply a response to the question by @DarkIce2 about master data where, in our circumstance, it is necessary. Your own mileage may vary.
Regards, Groatz
code_cache, but we manage the synchronisation of that reasonably well. Our system is well-engineered and constantly being refined with detailed monitoring and zero tolerance to errors. It rarely just "breaks" for no reason, quite the opposite.It would seem that the corruption is being injected upstream of the
code_cache, and by rebuilding master data the corruption is (temporarily) alleviated. We now going to rebuild the master data every night. We run hot 23x7 so there is about an hour when relatively few users are impacted by the rebuild.It will have to do for the time being. Note that my original reply was simply a response to the question by @DarkIce2 about master data where, in our circumstance, it is necessary. Your own mileage may vary.
Regards, Groatz
Ozzy47
Well-known member
Well given that, It is entirely possible that due to the size of the site it has exposed something that most sites won't come across. If feasible I would start a ticket and provide access to the XF staff so they can investigate to find out if you have triggered some sort of bug.
XenForo doesnt really care if its being run on 1 server or 100, as long as the config respects a few basic fundamentals. This issue is simply likely to be a faulty hosting node in play somewhere. I manage multiple sites for customers on multi server setups and none of them have experienced this issue.
We are also running XF on multiple nodes (though not 40) and havn't seen such an issue.
Normally, such things happen when nodes get out of sync somehow.
Have you tried to track down which node(s) are devlivering those broken templates?
Did you check that
Normally, such things happen when nodes get out of sync somehow.
Have you tried to track down which node(s) are devlivering those broken templates?
Did you check that
code_cache is consistent across all nodes?
Groatz
Member
Issue is consistent across all nodes, including the admin server (which is back from the main rotation, only admins use it). Yes, we sync every couple of minutes and this is verified. Which again leads me to believe that the problem is happening at the XF level rather than the mechanics of keeping in sync. @Brogan has advised us that the XF devs do not believe any changes in XF 2.1.5a would cause this. So it is still not empirically definitive whether the upgrade was causal or just coincidental to this issue.
We're going to build a new AMI (yes we're on AWS) for the cluster to reduce one variable (ie freshen up Nginx, PHP-FPM, and the initial server sync). We can also consider completing the PHP 7.3 upgrade around the same time. That will rule out some potential problems perhaps. We've never had anything like this before that we couldn't solve. :-/
Regards, Groatz
Similar threads
