XF 1.1 Error 500 and database errors

[Ingenious]

My forum members are reporting that around 10am each morning the forum can fall over, some are saying there is a 500 error message, some are saying it's an unexpected database error.

I've logged into the AdminCP and the server log shows no error messages at all (even for last week when the forum did fall over while I was on-line at a different time of day and saw a database error message) so I am not sure what exactly the server log is actually logging

I have raised this with my web host, I am on a shared/cloud service, and I remember before my conversion I might get say 3 - 4 error emails a week from vB, all the same, the database (which is shared) being unavailable or busy, I don't mind that since I am on both a shared and a cheap package. But when several members start actually posting about errors and it's during the day, that's a bit more serious. Also the vB errors were spread at random times, I can see no cluster at 10am.

The host have said they can find no record of any problems and asked if XenForo has any other logs or debug information which would record in more detail what the problem is? Where would you suggest I go from here please?

My add-ons are Kotomi script which is only run for an external page (a member map), Sitemap for Xenforo which runs at 3am, and Tapatalk (reports of 10am errors pre-date installing this).

 

[Paul B]

Is there a routine server backup being performed at that time?

Either that or another domain on the shared server is running an intensive script at that time.

 

[Ingenious]

That's what I thought too initially, but I think I have exhausted my options with the host until I can go back with something more concrete than what I have at the moment - which is no documented error logged. Hence why I need to know how I can get some more detailed information when the forum falls over. As I said above, when I saw a database error myself in the forum, it didn't say anything except it was an unexpected error and was not logged anywhere.

 

[Paul B]

500 errors are notoriously difficult to debug and usually server related.

Really it's up to your host to monitor what is happening on the server at that time of day to try and determine the cause.

 

[Ingenious]

OK, putting aside 500 errors then and looking at database connection errors. Sorry to bring vB up again, but when that could not connect to the database I got an email which told me the error. Thus not only did I get a record of when the forum fell over due a database error, but what the error message was. Is there anything similar in XenForo to log this information somewhere? If I can record when the forum falls over due to a database issue (too busy, too many connections, gone AWOL) then I can establish whether there is a pattern.

 

[Dean]

I went nuts debugging a 500 error, until I put the forum in debug mode, then I got some useful errors. Maybe?
http://xenforo.com/community/threads/frequently-asked-questions.5183/#post-248490

 

[Ingenious]

Thanks Dean, appreciated, I didn't know I could enable debug just for my IP, so I'll try that.

 

[Ingenious]

Is there a routine server backup being performed at that time?

Spot on Brogan!!

I went into debug mode and waited for my forum to do its daily fall over, then cut and paste the error in a ticket to Mike here. He said the error was indicating a problem with the host or database and was not a bug. Armed with that I went back to my host. They weren't that helpful to be honest but we tracked this down to a database backup being performed daily at the same time. For some reason they thought 10am was a good time to do this and said they couldn't change it on the shared/cluster hosting since it is an automated system.

The odd thing is, when I log in via SSH and do a mysqldump, at that time the forum just becomes unresponsive while it dumps, I don't get error messages nor do I get a few mins downtime like I do with their daily back up. The difference IMO is my dump is plain text, theirs is gzipped, wonder if that extra time lag is causing the issue? I got the dreaded response "Other customers have not complained" which a we all know, translated, means "We don't believe you and don't want to help you, GO AWAY"

Anyway to cut a long story short, I now do my own daily backup via cron in the wee small hours. Not at 10am in the morning.

Thanks Brogan for the advice and being right, and to Mike for the ticket support.

 

[Digital Doctor]

Anyway to cut a long story short, I now do my own daily backup via cron in the wee small hours. Not at 10am in the morning

10am is a weird time for backups. Maybe they are 4 timezones West of you and they are doing it at 6am !

 

[Ingenious]

I'd like to follow up on this one for the benefit of anyone else out there suffering from unexplained but regular forum fall-overs.

Changing the 10am backup from a mysqldump+gzip to just a plain text dump, and moving this to the small hours of the morning, fixed the 10am fall over. So it was assumed the extra strain on the service created by this was causing the database to lockup or time out.

But sadly my site has continued to fall to bits every morning but at 8.30am now, and this does not correspond with any backups on my site or any crons running. The suspicion was that another user on this shared hosting was hammering the database at this time, maybe with a back up, but the web hosts did not think this could be the case since it is a cloud/clustered service and designed to cope with spikes in traffic.

After much experimentation including 24/7 monitoring of my Xenforo site, my vB site (which runs the redirects) and my Wordpress site, plus selected other random websites not owned by me but hosted on the same service, I found that only the Xenforo forum was falling over and none of the others, even those connecting to the same database.

This is the latest response from my webhost in this matter, I hope this is some use to others if this comes up in the future and I will continue to monitor the site:

Now fixed I hope. The cause isn't a bug as such, but it is behaviour similar to a bug which was on the MySQL list for some time:

http://bugs.mysql.com/bug.php?id=42041

And is also on the MySQL website, and not classed as a bug, just 'behaviour':

http://dev.mysql.com/doc/refman/5.1/en/statement-repreparation.html

The cause is that Xenforo uses prepared statements, which can only remain valid as long as the MySQL server can retain its cache of the tables. If it doesn't, it needs to reprepare the statement. When the system runs a backup, the relevant caches could be flushed by the database being read in to be backed up if it has a lot of tables.

I've altered a couple of settings on the server which should resolve the issue. I've also switched the site to PHP 5.3 which is newer and faster than 5.2.

If it happens again tomorrow I'll migrate the database to one of our new Percona MySQL servers.

I'd welcome comments from anyone sufficiently knowledgeable about Xenforo whether those statements above are correct - sadly it's all well beyond me. My downtime is between 2 and 5 minutes each day.

Also there is a reason why I was running PHP5.3 - it's because on my particular host, GeoIP which I use to filter registrations (I am a UK only forum for legal reasons, because of the subject of my site) is only available on PHP5.2 -otherwise I would of course be running 5.3!

 

[Jake Bunce]

That agrees with information about this error that I found previously:

http://xenforo.com/community/threads/error-on-install.17901/#post-236565

 

[Ingenious]

Thanks Jake. The suggestions on config changes are useful to know.