MariaDB has stopped on a cPanel server and won't start again. Every site shows "Error
establishing a database connection", chkservd keeps restarting mysql and failing, and
the error log ends with an InnoDB assertion or Plugin 'InnoDB' registration as a STORAGE ENGINE failed. This is the recovery sequence: find the cause, start InnoDB in forced
recovery only if you have to, dump the data, and rebuild the data directory cleanly.
innodb_force_recovery does not repair anything. It lets you start the server in a
degraded state so you can read data out. Corrupt pages stay corrupt.
Stop chkservd fighting you
chkservd restarts MariaDB every few minutes. Each failed start writes to the redo log and error log while you are trying to read them. Turn off monitoring for the duration:
whmapi1 configureservice service=mysql enabled=1 monitored=0
systemctl stop mariadb
Turn it back on at the end. If you forget, the next crash goes unnoticed.
Read the error log before touching anything
The log is the .err file in the data directory, named after the hostname:
ls -t /var/lib/mysql/*.err | head -1
tail -100 "$(ls -t /var/lib/mysql/*.err | head -1)"
Match what you see to a cause:
| Log line | Cause | First action |
|---|---|---|
No space left on device, errno 28 | Full / or /var | Free space, then start normally |
Can't create/write to file '/tmp/...' | Full or read-only /tmp | Fix /tmp, start normally |
Cannot allocate memory for the buffer pool | innodb_buffer_pool_size too big for RAM | Lower it in /etc/my.cnf |
Unable to lock ./ibdata1 error: 11 | Another mysqld still running | pkill -9 mariadbd, start again |
Database page corruption on disk, [FATAL] InnoDB: ... corrupted | Real page corruption | Forced recovery |
Assertion failure in file ... trx0purge / log0recv | Undo or redo log damage | Forced recovery |
Most "corruption" on shared servers is a full disk. Clear it first with our
full disk guide. After that, a plain
/scripts/restartsrv_mysql usually lets crash recovery complete without help.
Copy the data directory first
Before any forced start, take a cold copy. Levels 4 and above can leave the files worse than they were, and this copy is the only way back:
df -h /var/lib/mysql /home
rsync -a /var/lib/mysql/ /home/mysql-crash-copy-$(date +%F)/
Start in forced recovery
Add the setting under [mysqld] in /etc/my.cnf. Start at 1:
[mysqld]
innodb_force_recovery = 1
systemctl start mariadb
tail -f "$(ls -t /var/lib/mysql/*.err | head -1)"
If it doesn't start, raise the value by one and try again. Never jump straight to 6.
| Level | Skips | Risk |
|---|---|---|
| 1 | Crashes on corrupt pages | Low; you lose only the corrupt pages |
| 2 | Background purge thread | Low; undo logs grow while running |
| 3 | Rollback of incomplete transactions | Moderate; uncommitted rows may appear |
| 4 | Further rollback (MariaDB 10.6.5+) | Secondary indexes may be corrupted |
| 5 | Undo log scan at startup | Inconsistent reads; treat dumps as suspect |
| 6 | Redo log roll-forward | Last resort; can cause further corruption |
At level 4 and above, treat the server as read-only. Don't let sites write to it.
Dump everything you can
Once MariaDB is up, block web traffic so nothing writes, then dump each database separately. One bad table then fails one dump, not all of them:
mkdir -p /home/sqldumps
for db in $(mysql -Nse "SHOW DATABASES" | grep -Ev '^(information_schema|performance_schema|sys)$'); do
mysqldump --single-transaction --routines --events "$db" > "/home/sqldumps/$db.sql" \
|| echo "FAILED: $db" >> /home/sqldumps/failed.txt
done
cat /home/sqldumps/failed.txt
For a database in failed.txt, dump it table by table to find the corrupt table. You can
often recover the rows on either side of the bad page with
SELECT ... ORDER BY id DESC. Anything that won't dump comes back from backup. A recent
JetBackup snapshot or an
R1Soft single-database restore is quicker
than hand-extracting rows.
Rebuild a clean data directory
Remove innodb_force_recovery from /etc/my.cnf, move the damaged data directory aside,
and initialise a fresh one:
systemctl stop mariadb
mv /var/lib/mysql /var/lib/mysql.broken
mkdir /var/lib/mysql && chown mysql:mysql /var/lib/mysql
mariadb-install-db --user=mysql --datadir=/var/lib/mysql
systemctl start mariadb
Re-import mysql.sql first, which restores users and grants, then every other dump. Then
let cPanel reconcile its view of database ownership and turn monitoring back on:
mysql mysql < /home/sqldumps/mysql.sql && mysql -e "FLUSH PRIVILEGES"
for f in /home/sqldumps/*.sql; do db=$(basename "$f" .sql); [ "$db" = mysql ] && continue; mysql -e "CREATE DATABASE IF NOT EXISTS \`$db\`"; mysql "$db" < "$f"; done
/usr/local/cpanel/bin/setupdbmap
/scripts/restartsrv_mysql
whmapi1 configureservice service=mysql enabled=1 monitored=1
Keep /var/lib/mysql.broken and the cold copy until customers confirm their sites. Then
check whether the root cause was RAM pressure. An oversized buffer pool that gets
OOM-killed mid-write is a common trigger, and our
MariaDB tuning guide sizes it against PHP and
the web server.
What innodb_force_recovery level should I use?+
Can I leave innodb_force_recovery enabled on a live cPanel server?+
Where is the MySQL error log on cPanel?+
Why does MariaDB keep crashing after a full disk on cPanel?+
Do I need to re-map databases to cPanel users after restoring MySQL?+
Next steps
- Fix a full disk on a cPanel server: the most common cause of InnoDB crashes.
- Upgrade MariaDB or MySQL in WHM: move off an end-of-life version once the server is stable.
- Backup strategy for cPanel hosts: so the next crash is a restore, not a recovery.
Rebuilding onto new hardware? Move your cPanel license to the new IP, or see cPanel license pricing for a new node.