Mariadb crashing, tables marked as crashed

[nathanw]

Hi all,

Using Xenforo 2.3
Debian 12
PHP 8.2.28
MariaDB 10.11.11

Mariadb is crashing every few days.
Journalctl -u mariadb shows recurring issues:

Mar 27 19:02:11 <hostname> mariadbd[3937034]: 2025-03-27 19:02:11 0 [Note] Starting MariaDB 10.11.11-MariaDB-0+deb12u1 source revision e69f8cae1a15e15b9e4f5e0f8497e1f17bdc81a4 server_uid Ay+OhJdDG3olVaBW>
Mar 27 19:02:11 <hostname> mariadbd[3937034]: 2025-03-27 19:02:11 0 [Note] InnoDB: Compressed tables use zlib 1.2.13
Mar 27 19:02:11 <hostname> mariadbd[3937034]: 2025-03-27 19:02:11 0 [Note] InnoDB: Number of transaction pools: 1
Mar 27 19:02:11 <hostname> mariadbd[3937034]: 2025-03-27 19:02:11 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
Mar 27 19:02:11 <hostname> mariadbd[3937034]: 2025-03-27 19:02:11 0 [Note] InnoDB: Using liburing
Mar 27 19:02:11 <hostname> mariadbd[3937034]: 2025-03-27 19:02:11 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
Mar 27 19:02:11 <hostname> mariadbd[3937034]: 2025-03-27 19:02:11 0 [Note] InnoDB: Initialized memory pressure event listener
Mar 27 19:02:11 <hostname> mariadbd[3937034]: 2025-03-27 19:02:11 0 [Note] InnoDB: Completed initialization of buffer pool
Mar 27 19:02:11 <hostname> mariadbd[3937034]: 2025-03-27 19:02:11 0 [Note] InnoDB: File system buffers for log disabled (block size=512 bytes)
Mar 27 19:02:11 <hostname> mariadbd[3937034]: 2025-03-27 19:02:11 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=11946954518
Mar 27 19:02:11 <hostname> mariadbd[3937034]: 2025-03-27 19:02:11 0 [Note] InnoDB: End of log at LSN=11953411090
Mar 27 19:02:12 <hostname> mariadbd[3937034]: 2025-03-27 19:02:12 0 [Note] InnoDB: To recover: 585 pages
Mar 27 19:02:12 <hostname> mariadbd[3937034]: 2025-03-27 19:02:12 0 [Note] InnoDB: 128 rollback segments are active.
Mar 27 19:02:12 <hostname> mariadbd[3937034]: 2025-03-27 19:02:12 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
Mar 27 19:02:12 <hostname> mariadbd[3937034]: 2025-03-27 19:02:12 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
Mar 27 19:02:12 <hostname> mariadbd[3937034]: 2025-03-27 19:02:12 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
Mar 27 19:02:12 <hostname> mariadbd[3937034]: 2025-03-27 19:02:12 0 [Note] InnoDB: log sequence number 11953411090; transaction id 16087972
Mar 27 19:02:12 <hostname> mariadbd[3937034]: 2025-03-27 19:02:12 0 [Note] Plugin 'FEEDBACK' is disabled.
Mar 27 19:02:12 <hostname> mariadbd[3937034]: 2025-03-27 19:02:12 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
Mar 27 19:02:12 <hostname> mariadbd[3937034]: 2025-03-27 19:02:12 0 [Warning] You need to use --log-bin to make --expire-logs-days or --binlog-expire-logs-seconds work.
Mar 27 19:02:12 <hostname> mariadbd[3937034]: 2025-03-27 19:02:12 0 [Note] Server socket created on IP: '127.0.0.1'.
Mar 27 19:02:12 <hostname> mariadbd[3937034]: 2025-03-27 19:02:12 0 [Note] /usr/sbin/mariadbd: ready for connections.
Mar 27 19:02:12 <hostname> mariadbd[3937034]: Version: '10.11.11-MariaDB-0+deb12u1'  socket: '/run/mysqld/mysqld.sock'  port: 3306  Debian 12
Mar 27 19:02:12 <hostname> systemd[1]: Started mariadb.service - MariaDB 10.11.11 database server.
Mar 27 19:02:12 <hostname> /etc/mysql/debian-start[3937080]: Upgrading MySQL tables if necessary.
Mar 27 19:02:12 <hostname> mariadbd[3937034]: 2025-03-27 19:02:12 0 [Note] InnoDB: Buffer pool(s) load completed at 250327 19:02:12
Mar 27 19:02:12 <hostname> /etc/mysql/debian-start[3937091]: Checking for insecure root accounts.
Mar 27 19:02:12 <hostname> /etc/mysql/debian-start[3937095]: Triggering myisam-recover for all MyISAM tables and aria-recover for all Aria tables
Mar 27 19:02:12 <hostname> mariadbd[3937034]: 2025-03-27 19:02:12 32 [ERROR] mariadbd: Table './xenforo/xf_search_index' is marked as crashed and should be repaired
Mar 27 19:02:12 <hostname> mariadbd[3937034]: 2025-03-27 19:02:12 32 [Warning] Checking table:   './xenforo/xf_search_index'
Mar 27 19:02:13 <hostname> mariadbd[3937034]: 2025-03-27 19:02:13 40 [ERROR] mariadbd: Table './xenforo/xf_session' is marked as crashed and should be repaired
Mar 27 19:02:13 <hostname> mariadbd[3937034]: 2025-03-27 19:02:13 40 [Warning] Checking table:   './xenforo/xf_session'
Mar 27 19:02:13 <hostname> mariadbd[3937034]: 2025-03-27 19:02:13 40 [ERROR] mariadbd: Table './xenforo/xf_session_admin' is marked as crashed and should be repaired
Mar 27 19:02:13 <hostname> mariadbd[3937034]: 2025-03-27 19:02:13 40 [Warning] Checking table:   './xenforo/xf_session_admin'
Mar 28 08:45:16 <hostname> mariadbd[3937034]: 2025-03-28  8:45:16 0 [Note] InnoDB: Memory pressure event freed 6321 pages
Mar 28 08:45:37 <hostname> mariadbd[3937034]: 2025-03-28  8:45:37 31329 [Warning] Aborted connection 31329 to db: 'xenforo' user: 'xenforo' host: 'localhost' (Got an error reading communication packets)
Mar 28 08:45:41 <hostname> mariadbd[3937034]: 2025-03-28  8:45:41 31337 [Warning] Aborted connection 31337 to db: 'xenforo' user: 'xenforo' host: 'localhost' (Got an error reading communication packets)
Mar 28 08:45:53 <hostname> mariadbd[3937034]: 2025-03-28  8:45:53 31344 [Warning] Aborted connection 31344 to db: 'xenforo' user: 'xenforo' host: 'localhost' (Got an error reading communication packets)
Mar 28 08:46:15 <hostname> mariadbd[3937034]: 2025-03-28  8:46:15 0 [Note] InnoDB: Memory pressure event freed 413 pages
Mar 28 08:46:15 <hostname> mariadbd[3937034]: 2025-03-28  8:46:15 31349 [Warning] Aborted connection 31349 to db: 'xenforo' user: 'xenforo' host: 'localhost' (Got an error reading communication packets)
Mar 28 08:46:59 <hostname> systemd[1]: mariadb.service: A process of this unit has been killed by the OOM killer.
Mar 28 08:46:59 <hostname> systemd[1]: mariadb.service: Main process exited, code=killed, status=9/KILL
Mar 28 08:46:59 <hostname> systemd[1]: mariadb.service: Failed with result 'oom-kill'.
Mar 28 08:46:59 <hostname> systemd[1]: mariadb.service: Consumed 3min 33.099s CPU time.

After I restart MariaDB, Xenforo works again until something crashes the tables.

I have tried repairing all tables listed as crashed, but phpmyadmin shows "The storage engine for the table doesn't support repair".

Does anyone have any tips on how I can troubleshoot and fix this issue?

 

[Mike Fara]

Mar 28 08:46:59 <hostname> systemd[1]: mariadb.service: A process of this unit has been killed by the OOM killer.

Check your memory allocation in my.cnf this is what is crashing you (out of memory).

ALTER TABLE xf_search_index FORCE;
ALTER TABLE xf_session FORCE;
ALTER TABLE xf_session_admin FORCE;

sudo mv /var/lib/mysql/ib_logfile0 /var/lib/mysql/ib_logfile0.bak
sudo mv /var/lib/mysql/ib_logfile1 /var/lib/mysql/ib_logfile1.bak


But only when the root cause is fixed, or it will keep happening.

 

[nathanw]

@Mike Fara Thanks for the advise.
I've created 16gb of swap where I previously had none and will see if that does the trick.

For my understanding, what is the purpose of the alter table commands? I ran them without the 'force' paramater as phpmyadmin didn't accept it.