If you’re managing a server with cPanel on AlmaLinux and using MySQL 8, you might encounter a specific bug that can cause MySQL to fail due to daily cPanel updates. Even after downgrading MySQL, you may find that the InnoDB database system’s ibdata file is corrupted, preventing MySQL from starting.
The Issue
The root cause of this problem is an upstream bug, as documented in cPanel support articles. The issue still hot in the Cpanel database forum. Here’s how to address and fix this issue:
Step-by-Step Fix
1. Downgrade MySQL
First, you need to downgrade MySQL to a version before the issue occurred. Use the following command:
yum downgrade mysql-community-*
2. Version Lock MySQL
To prevent the update from reoccurring, you should lock the MySQL version:
dnf install python3-dnf-plugin-versionlock yum versionlock mysql-community-*
Unfortunately, these steps alone might not fully resolve the issue, as was the case with our server. Despite downgrading MySQL, our server did not recover, necessitating further support from cPanel, CloudLinux, and third-party experts.
Recovery Process
It took nearly a full day to fix the issue. Previously, I’ve encountered similar issues with InnoDB and resolved them by switching the database engine to MyISAM. However, this situation was different.
Fortunately, our server was prepared with daily MySQL backups. The main challenge was that MySQL was down, so we couldn’t restore the database immediately. Here’s the detailed recovery process:
1. Rename the MySQL Directory
Since /var/lib/mysql was unusable, I renamed it to avoid conflicts:
mv /var/lib/mysql /var/lib/mysql-old
Then creating new mysql folder inside /var/lib and give proper permission.
2. Restart MySQL
After renaming the directory, restarting MySQL brought the database back online:
systemctl restart mysqld
3. Restore the MySQL Database
With MySQL running again, I restored the mysql.sql database:
mysql mysql < /path/to/backup/mysql.sql
4. Monitoring and Final Restoration
After monitoring for stability, I confidently restored the remaining databases using script to restore. This script facilitated the complete recovery of all databases.
5. Fixing CloudLinux Integration
Using CloudLinux, it was crucial to run the following command to prevent the “error establishing database connection” message:
cagefs -r
Running this command resolved the issue, ensuring the databases were accessible and the server was fully operational.
Conclusion
Handling MySQL issues after cPanel updates can be challenging, especially with InnoDB corruption. By following the outlined steps and ensuring regular backups, you can minimize downtime and restore services effectively. Always keep your system updated with the latest support insights and solutions to maintain optimal server performance.
