Bit of tech help? - Database tables keep crashing

[AdamD]

No idea why this is suddenly an issue, but earlier today a few database tables came up as marked as crashed

So I shut down Mysql, ran a repair/optimise on all and everything seemed ok

An hour later, some more tables came up with marked as crashed

I checked the logs and saw lots of errors like this:

130318 16:33:56 [ERROR] /usr/sbin/mysqld: Incorrect key file for table './x/jos_session.MYI'; try to repair it
and
130318 16:34:06 [ERROR] /usr/sbin/mysqld: Table './x/jos_session' is marked as crashed and should be repaired

I checked the tmp folder, it had some 899mb free, so it wasn't that, but repairing, the tables just didn't seem to "hold"

I've now changed the mysql tmp folder to another location, to see if it helps, but any tips?

 

[Slavik]

No idea why this is suddenly an issue, but earlier today a few database tables came up as marked as crashed

So I shut down Mysql, ran a repair/optimise on all and everything seemed ok

An hour later, some more tables came up with marked as crashed

I checked the logs and saw lots of errors like this:

130318 16:33:56 [ERROR] /usr/sbin/mysqld: Incorrect key file for table './x/jos_session.MYI'; try to repair it
and
130318 16:34:06 [ERROR] /usr/sbin/mysqld: Table './x/jos_session' is marked as crashed and should be repaired

I checked the tmp folder, it had some 899mb free, so it wasn't that, but repairing, the tables just didn't seem to "hold"

I've now changed the mysql tmp folder to another location, to see if it helps, but any tips?

As youve already worked out, the usual cause is /tmp/ filling up..

maybe install tmpwatch?

 

[AdamD]

Yea it's odd, the tmp directory wasn't full, not anywhere near.

It has the correct permissions and there are tmp files in there, so I'm a bit at a loss

I see mysql writing to the new tmp folder ok, at least.

 

[Biker]

Had a similar issue a couple of years ago where tables were showing up as corrupted when everything had been running just fine for over 2 years. I'd repair the tables only to have more show up as corrupted shortly afterwards. After a couple of hours of fighting the issue, the entire server took a dump due to the motherboard going Tango Uniform.

 

[Slavik]

Had a similar issue a couple of years ago where tables were showing up as corrupted when everything had been running just fine for over 2 years. I'd repair the tables only to have more show up as corrupted shortly afterwards. After a couple of hours of fighting the issue, the entire server took a dump due to the motherboard going Tango Uniform.

Maybe time for Adam to make a backup... just in case

 

[Biker]

Maybe time for Adam to make a backup... just in case

Might be wise. Fortunately, the server does a full backup of all databases and user accounts every night and then shoots them over to the FTP server. Lost about 12 hours worth of data and posts, but I can live with that.

 

[AdamD]

Hmm well this is a VPS with Raid 10, but, I suppose that wouldn't matter much, if the motherboard was dying lol

 

[Mike Edge]

Has been a lot of issues with filling /tmp lately For us to the point of moving our sql temp directory to a folder in /home to have the space. Odd thing though for everyone with this issue is, there hasn't been a xF update in a long time to be a reflection on a recent change.

 

[AdamD]

Hmm, still getting problems with /tmp

Noticed another site's admin panel kept giving me the nginx 500 internal server error, but other pages were working ok, so checking the log, I see:

[Wed Mar 20 13:06:10 2013] [warn] [client 66.249.73.91] mod_fcgid: stderr: PHP Warning:  Unknown: open(/tmp/sess_aef044dfe3defd98e6931b4df9c19e8d, O_RDWR) failed: No space left on device (28) in Unknown on line 0
[Wed Mar 20 13:06:10 2013] [warn] [client 66.249.73.91] mod_fcgid: stderr: PHP Warning:  Unknown: Failed to write session data (files). Please verify that the current setting of session.save_path is correct (/tmp) in Unknown on line 0

So again, I check the tmp folder, it's practically empty
I really can't understand this heh, why is the server having issues with /tmp?
Tmp files ARE being written to it, but it seems to report that the folder is full, when it's not.
The session.save_path is set correctly, too.

 

[SilverCircle]

Hmm, still getting problems with /tmp

Noticed another site's admin panel kept giving me the nginx 500 internal server error, but other pages were working ok, so checking the log, I see:

[Wed Mar 20 13:06:10 2013] [warn] [client 66.249.73.91] mod_fcgid: stderr: PHP Warning:  Unknown: open(/tmp/sess_aef044dfe3defd98e6931b4df9c19e8d, O_RDWR) failed: No space left on device (28) in Unknown on line 0
[Wed Mar 20 13:06:10 2013] [warn] [client 66.249.73.91] mod_fcgid: stderr: PHP Warning:  Unknown: Failed to write session data (files). Please verify that the current setting of session.save_path is correct (/tmp) in Unknown on line 0

So again, I check the tmp folder, it's practically empty
I really can't understand this heh, why is the server having issues with /tmp?
Tmp files ARE being written to it, but it seems to report that the folder is full, when it's not.
The session.save_path is set correctly, too.

I would first do a file system check and see what it says. This could very well be a corrupted file system - rare thing, but it still can happen due to software or even hardware issues. No RAID or other hardware can protect against software issues or other problems causing corrupted data to be physically written to the file system.

Where is you /tmp? Maybe the partition is really full? Remember that most Unix systems reserve a certain amount of disk space to the root user, so any application running as non-root can report a full file system when there is still some space left on the device (typically 5-10% are reserved).

Or maybe you've run out of inodes (df -i will show you inode usage on the filesystem), but that's unlikely unless you have some stuff that needs an insanely high number of files or directories.

 

[AdamD]

Yea I found out what's causing it

I completely forgot about the Inode limit that VPS' have, seems nginx wasn't clearing out it's temporary files in the tmp folder, so even though it was empty, it had reached it's inode limit.

At least, I hope that's it!