inertz.org

Howto repair corrupted Virtuozzo container

Posted on by admin

Sometimes some broken Virtuozzo container cannot start due to container corruption like example below;

[email protected] [~]# vzctl start 9068405
Mount image: /vz/private/9068405/root.hdd

/dev/ploop16823p1: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
(i.e., without -a or -p options)
Failed to mount image /vz/private/9068405/root.hdd: Error in e2fsck (fsutils.c:282): e2fsck failed (exit code 4)

Failed to mount image: Error in e2fsck (fsutils.c:282): e2fsck failed (exit code 4)
[152]
Unmount image: /vz/private/9068405/root.hdd

Data corruption can occur from a variety of reasons. Usually this can be fixed by doing file system check to the disk image.

First we need to know the ploop ID. This can be done by mounting the ploop image.

[email protected] [~]# ploop mount /vz/private/9068405/root.hdd/DiskDescriptor.xml
Opening delta /pstorage/pcs1-cluster/private/9068405/root.hdd/root.hds
Opening delta /pstorage/pcs1-cluster/private/9068405/root.hdd/root.hds
Adding delta dev=/dev/ploop16823 img=/pstorage/pcs1-cluster/private/9068405/root.hdd/root.hds (rw)

Once we know the ploop ID, we need to know ploop partition device.

[email protected] [~]# fdisk -l /dev/ploop16823

WARNING: GPT (GUID Partition Table) detected on '/dev/ploop16823'! The util fdisk doesn't support GPT. Use GNU Parted.

Disk /dev/ploop16823: 107.4 GB, 107374182400 bytes
255 heads, 63 sectors/track, 13054 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Device Boot Start End Blocks Id System
/dev/ploop16823p1 1 13055 104857599+ ee GPT

Doing file system check to the partition.

[email protected] [~]# fsck /dev/ploop16823p1 -Cy
fsck from util-linux-ng 2.17.2
e2fsck 1.41.12 (17-May-2010)
/dev/ploop16823p1: clean, 807261/6553600 files, 16353823/26213888 blocks

Umount back the ploop.

[email protected] [~]# ploop umount -d /dev/ploop16823p1
Unmounting device /dev/ploop16823
Opening delta /pstorage/pcs1-cluster/private/9068405/root.hdd/root.hds

Start back the container.

[email protected] [~]# vzctl --verbose start 9068405
Mount image: /vz/private/9068405/root.hdd
Container is mounted
Starting the Container ...
running: /usr/sbin/cpufeatures --quiet sync
running: /usr/sbin/vzpkg info -q centos-6-x86_64 osrelease
Warning: VSwap_slm compatiblity mode all
RAM: 2097152 Swap: 0 ovr: 1.00
UB_SWAPPAGES, {0, 0}
UB_LOCKEDPAGES, {2097152, 2097152}
UB_PRIVVMPAGES, {9223372036854775807, 9223372036854775807}
UB_NUMPROC, {500, 500}
UB_PHYSPAGES, {2097152, 2097152}
UB_VMGUARPAGES, {2097152, 2097152}
UB_OOMGUARPAGES, {2097152, 2097152}
CPU limit: 100%
Running the command: /etc/sysconfig/vz-scripts/vz-start
Setting permissions 20002 dev 0x7d00
Adding offline management to Container(1): 4643
Adding IP addresses: 192.168.0.93/255.255.255.0 192.168.0.94/255.255.255.0
Running the command: /etc/sysconfig/vz-scripts/vz-net_add
Run the script /etc/sysconfig/vz-scripts/dists/scripts//redhat-add_ip.sh
Hostname of the Container set: main.test.com.my
Run the script /etc/sysconfig/vz-scripts/dists/scripts//redhat-set_hostname.sh
Run the script /etc/sysconfig/vz-scripts/dists/scripts//set_dns.sh
File resolv.conf was modified
Run the script /etc/sysconfig/vz-scripts/dists/scripts//set_console.sh
Starting the Container ...

Fix broken virtuozzo container

Posted on by admin

On this guide, we will learn how to fix broken virtuozzo container. The server is actually a cloud storage server. The purpose of the cloud storage server function like almost high availability. When the hardware node down, the vps container inside the node will automatically migrate to available active hardware node. In this setup we have 5 hardware node or we also called it Parallels Cloud Storage(PCS).

Sometimes during this event of hardware failure or server down, some of the container will be broken due to sometimes there is no much time to recover from the downtime.

The broken container will be tag as ‘B’ if we run command ‘shaman stat’ example;

8826061 on 101.30.110.101 0
8833527 on 101.30.110.103 0
B 8916305 on Unknown 0
8959776 on 101.30.110.103 0

For this case the container ID ‘8916305’ is broken.

We need to go to ‘/pstorage/pcs1-cluster/.shaman/broken’ to see the actual location of the PCS server.

[email protected] [/pstorage/pcs1-cluster/.shaman/broken]# cat ct-8916305
RELOCATE_PRIO=0
PATH=private/8916305
NODE=101.30.110.104
TIME=1572058900
INFO=Resource relocation failed
[email protected] [/pstorage/pcs1-cluster/.shaman/broken]#

In this case we now know the actual container located at PCS4. So we can just delete the file name ‘ct-8916305’

[email protected] [/pstorage/pcs1-cluster/.shaman/broken]# rm ct-8916305

rm: remove regular file `ct-8916305'? y

After that we need to check the configuration file of the container by login to the PCS where the container located. Make sure all setting are correct.

[email protected] [/pstorage/pcs1-cluster/.shaman/broken]# cat /vz/private/8916305/ve.conf
DESCRIPTION="Test=20User=20Test=20User"
HOSTNAME="svr1.test.com.my"
IP_ADDRESS="192.168.26.8/255.255.255.0 192.168.26.9/255.255.255.0"
NAMESERVER="8.8.8.8 8.8.4.4"
NETFILTER="full"
NETIF=""
OFFLINE_MANAGEMENT="yes"
OFFLINE_SERVICE="vzpp"
ONBOOT="yes"
ORIGIN_SAMPLE="4vCPU-Supreme-Plan"
OSTEMPLATE=".centos-7-x86_64"
SEARCHDOMAIN="google.com"
SLMMODE="all"
TEMPLATES=""
UUID="0820b9ee-af22-4f0a-bb67-18578b1ce678"
CPULIMIT_MHZ="8192"
DISKSPACE="104857600:104857600"
SLMMEMORYLIMIT="8589934592:8589934592"
VE_ROOT="/vz/root/$VEID"
VE_PRIVATE="/vz/private/$VEID"
DISTRIBUTION="centos"
TECHNOLOGIES="x86_64 nptl"
VEID="8916305"
CONFIG_CUSTOMIZED="yes"
RATEBOUND="yes"
NUMPROC="500:500"
NAME="Test User Test User"

Register the container to the hardware node;

[email protected] [/pstorage/pcs1-cluster/.shaman/broken]# vzctl register /vz/private/8916305 8916305
Container was successfully registered
Adding IP addresses to the pool: 192.168.26.8 192.168.26.9
Deleting offline management
Adding offline management to Container(1): 4643
[email protected] [/pstorage/pcs1-cluster/.shaman/broken]#

Resume the container;

[email protected] [/pstorage/pcs1-cluster/.shaman/broken]# vzctl start 8916305
Mount image: /vz/private/8916305/root.hdd
Container is mounted
Starting the Container ...
Os release: 3.10.0-042stab139.1
CPU limit: 100%
Setting permissions 20002 dev 0x7d00
Adding offline management to Container(1): 4643
Adding IP addresses: 192.168.26.8/255.255.255.0 192.168.26.9/255.255.255.0
Hostname of the Container set: svr1.test.com.my
File resolv.conf was modified
Starting the Container ...

Restore database from *.frm for corrupted database

Posted on by admin

Recently on of my client have problem with the database corruption in Cpanel. The website trowing error related with database. We check our daily MySQL backup and found there is no data in the client MySQL database. To make thing worst our weekly and monthly backup also doesn’t have the database. According to client the last access to the website is in March.

Our daily MySQL backup does not contain data in the SQL query. This indicate data corruption.

We try to repair it using below command, but none helps.

mysqlcheck --auto-repair --optimize databasename
find /var/lib/mysql/databasename -name "*.MYI" -exec myisamchk -r -f {} \;
find /var/lib/mysql/databasename -name "*.ibd" -exec innochecksum {} \;
mysqlcheck --repair --use-frm databasename

After researching in the Internet we found that we can reconstruct the database from *.frm files using MySQL utilities called mysqlfrm

First you need to install mysql-utilities.

yum install mysql-utilities

Then cd to /var/lib/mysql and using below command to dump the database.

mysqlfrm --diagnostic databasename > /root/databasename.sql

The sql will give you tables but not the data but it is sufficient.

Backup /var/lib/mysql/databasename to somewhere or just rename it to something.

Restore the database.

mysql databasename < /root/databasename.sql

Restore the permission because the ownership is root.

chown -R mysql:mysql /var/lib/mysql/databasename

Stop the MySQL

Copy the data from databasename folder that you just rename or resync using command. Example;

rsync -arvh --progress /root/databasename/ /var/lib/mysql/databasename/

Start the MySQL

Supposed you have a working database and website now.