Trials and Tribulations of a Service Provider.
We have one of the best service providers in the business. WestNic has offered exemplary service and a secure computing platform to Nerd Vittles and PBX in a Flash for many years. We consume enormous computing resources for what we pay. But the last couple weeks have been painful. First, we were on vacation when WestNic made the transition (again) to PHP 5.3. These things usually happen in the middle of the night, and this was no exception. Unfortunately, we still were running a very old, highly customized (but very secure) version of WordPress. When morning came, Nerd Vittles died. We immediately knew why because we already had experienced PHP 5.3 a few months earlier, and WestNic graciously rolled it back… just for us. Unfortunately (for us), they didn’t tell us the new drop dead date. And, yes, we should have been updating WordPress. But it’s kinda like going to the dentist. You never quite get around to it until you have to. Well, now we had to. This involved backing up and restoring Nerd Vittles to another server still running the older version of PHP. So far, so good. It took about three hours to do the three WordPress updates, but all went well. Then we moved the site back to its home, and nothing worked again. Unfortunately, this hit on a weekend, and the weekend guys claimed it was a WordPress problem. It wasn’t this time, but it took until Monday morning to get the new php.ini file sorted out to accomodate PHP 5.3. Whew!
Then came the real fun. About 25% of the threads on the PBX in a Flash Forum could not be displayed. All you got was a blank screen when you clicked on a thread. As is customary with these types of issues, the XenForo developers blamed the provider. And the provider blamed XenForo. The provider uses mod_security to protect its web sites. But the provider assured us that nothing had changed. Well, nothing in mod_security anyway. After days and days of testing and back and forth, it turned out that the provider had added a new security mechanism, suhosin, which its developer touts as the “Guardian Angel” for PHP. That may be true for providers, but not so much for folks that actually depend upon their sites working. Welcome to a new can of worms!
Having been on both sides of this fence, we can readily appreciate the dilemma of the service providers. They don’t want their servers hacked. Denying access to all users would accomplish that goal but would reduce the number of paying customers pretty dramatically. So we all try to reach that happy medium trading off a little security for a bit more access. In this case, it turned out to be a couple of suhosin settings that monitor the length of URLs. We discovered that only after running literally hundreds of tests. Since XenForo’s forum software makes extensive use of lengthy URLs to maintain compatibility with older vBulletin posts, this caused a problem. HTML requests with URLs exceeding a certain length are simply thrown in the bit bucket by suhosin. The biggest hint was sitting in the service provider’s Apache log, but we had no access to that information, and they never looked until two and a half days after we first opened a trouble ticket. No errors appeared in our logs, and users got nothing but blank pages where the subject of a post on the forum exceeded 50 characters. Fortunately, that was enough of a hint to finally resolve the problem. The unfortunate part of this story is that, without 25 years of personal IT experience plus over 100 IT gurus that visit our sites regularly, it’s doubtful this ever would have gotten resolved other than by begging the provider to turn off mod_security and suhosin for our sites, something we were unwilling to do. If something similar ever happens to you, the command you need to know is php -v. This will tell you what’s running with PHP on your host. Our provider had implied that suhosin had not yet been activated. php -v suggested just the opposite. So did their error log once they looked. The other place to start searching for configuration information is /usr/local/lib/php.ini. This will tell you how your provider has PHP configured and whether your local php.ini file is even activated. Our provider suggested more than once that our local php.ini file had been misconfigured. We’d never touched it and, in our case, the server’s php.ini file indicated that it was never activated regardless of what its contents may have contained.
We’re glad everything is fixed. We all learned more than we ever wanted to know about suhosin. Still wishing there had been a little better communications with our provider. It would have made resolution a lot easier and quicker for all concerned. It’s especially difficult to resolve thorny issues like this using service tickets with response times of half a day per message. Did we mention there is virtually no documentation on suhosin and what each of its several dozen settings actually do. Our apologies to everyone that was impacted by the service disruptions. We’re glad it’s behind us.
