XF 1.2 Forum Down

[The Sandman]

Admin Extra suddenly went down early this morning. The AdminCP is still accessible. There are about 30 pages of server errors, not identical but all "Undefined Index: Permissions" errors. Disabling the listeners in config.php didn't fix anything.

This was the first error message thrown when the site went down -

Error Info

ErrorException: Undefined index: permissions - library/XenForo/AlertHandler/DiscussionMessage/Post.php:37
Generated By: s.molinari, Today at 5:22 AM

Stack Trace

#0 /home/adminx/public_html/library/XenForo/AlertHandler/DiscussionMessage/Post.php(37): XenForo_Application::handlePhpError(Array, Array, Array)
#1 /home/adminx/public_html/library/XenForo/Model/Alert.php(267): XenForo_AlertHandler_DiscussionMessage_Post->canViewAlert(Array, Array)
#2 /home/adminx/public_html/library/XenForo/Model/Alert.php(78): XenForo_Model_Alert->_getViewableAlerts(8473, 'fetchPopupItems')
#3 /home/adminx/public_html/library/XenForo/ControllerPublic/Account.php(73): XenForo_Model_Alert->getAlertsForUser()
#4 /home/adminx/public_html/library/XenForo/FrontController.php(337): XenForo_ControllerPublic_Account->actionAlertsPopup(Object(XenForo_RouteMatch))
#5 /home/adminx/public_html/library/XenForo/FrontController.php(134): XenForo_FrontController->dispatch()
#6 /home/adminx/public_html/index.php(13): XenForo_FrontController->run()
#7 {main}

Request State

array(3) {
  ["url"] => string(210) "http://adminextra.com/account/alerts-popup?&_xfRequestUri=%2Fthreads%2Fgoogle-plus-clone-script.6915%2F&_xfNoRedirect=1&_xfToken=8473%2C1387102942%2C1b551c50b51273b2a9d14b9957e6ac7cae9aa296&_xfResponseType=json"
  ["_GET"] => array(4) {
    ["_xfRequestUri"] => string(39) "/threads/google-plus-clone-script.6915/"
    ["_xfNoRedirect"] => string(1) "1"
    ["_xfToken"] => string(56) "8473,1387102942,1b551c50b51273b2a9d14b9957e6ac7cae9aa296"
    ["_xfResponseType"] => string(4) "json"
  }
  ["_POST"] => array(0) {
  }
}

I have been getting some random errors from the Notable Members add-ons recently but none that seemed to have any major affect.

Any ideas on where to start looking for the problem?

 

[Mike]

Last time this came up, it was coming up in cases where it was impossible, which points to opcode cache corruption. I'd try restarting the web server (if possible).

Failing that, try rebuilding the user cache.

 

[The Sandman]

Thanks Mike. I did restart Apache a short while ago which didn't help. I'll rebuild the user cache right now...

 

[The Sandman]

Rebuilding the user cache didn't help. A strange thing though - I watched as the script ran and the counter incremented by a couple of hundred with each iteration until about 3,500, then suddenly jumped to about 8,500 and then continued to go up by a couple of hundred for a few more cycles before finishing.

 

[Mike]

The output there is user IDs so that's not unexpected.

This does look like a situation that appears to be impossible -- the value should always exist. Can you temporarily disable your opcode cache and restart your webserver (or FastCGI server, I should point out)? If it sorts it, there could be an issue there, though I'm not sure why it'd appear suddenly.

If you still have problems, please submit FTP and ACP access details via the ticket.

 

[The Sandman]

The output there is user IDs so that's not unexpected.

This does look like a situation that appears to be impossible -- the value should always exist. Can you temporarily disable your opcode cache and restart your webserver (or FastCGI server, I should point out)? If it sorts it, there could be an issue there, though I'm not sure why it'd appear suddenly.

If you still have problems, please submit FTP and ACP access details via the ticket.

Commenting out the section of the config.php file with the cache instructions and restarting apache didn't help. Is that what you meant by disabling the opcode cache? And my other forum using the cache (with different prefix) hasn't had any problems during this time.

I've already submitted a ticket and will work with you guys that way.

 

[The Sandman]

Thanks Mike and Brogan - all is well now!

 

[TPerry]

Thanks Mike and Brogan - all is well now!

Was this something specific to your setup or something that might effect others? Or are you uncomfortable with providing what the solution was?

 

[Adam Howard]

Thanks Mike and Brogan - all is well now!

So the final solution to this was... ?

 

[x4rl]

Thanks Mike and Brogan - all is well now!

Just took 45 seconds to load a page from your site.

 

[Paul B]

It was a combination of (possible) things: opcode cache, server restart, user cache rebuild and corrupt browser session.

The site was never actually down for me and others but it was down for the OP.
I had a poke around in the ACP but couldn't see any obvious issues and I wasn't actually able to trigger the error.

 

[The Sandman]

Was this something specific to your setup or something that might effect others? Or are you uncomfortable with providing what the solution was?

So the final solution to this was... ?

It was a combination of (possible) things: opcode cache, server restart, user cache rebuild and corrupt browser session.

The site was never actually down for me and others but it was down for the OP.
I had a poke around in the ACP but couldn't see any obvious issues and I wasn't actually able to trigger the error.

As Brogan said, it was a combination of restarting Apache and a usercache rebuild that seemed to do the trick. I didn't know that my browser session was corrupted so I didn't immediately realize when the issue was fixed despite refreshing the browser with CTL F5 - it was only after Brogan told me the site was up and running that I closed and reopened the browser and was able to view the forums. I have no idea what triggered the error - it had something to do with the opcode cache (XCache) but TAZ also runs on the same server using XCache with a different prefix and was unaffected during this time.