Last night, I was updating some packages on a few Centos 6.2 VMs that I have in my home server when I found out that some commands hung (such as
rpm), and that I couldn't open new login shells on any of the VMs.
I tried restarting one, just to see the boot process hang while starting bind/named. I restarted the VM again in
single user mode, and disabled that service using chkconfig. Rebooted the VM, and it hung up on dhcpd, and later, after doing the same process, hung up on postfix.
At this point, I noticed a pattern: I obviously had broken a core library. Some of the packages that I had cherry picked for the update, were the standard C++ runtime libraries, and some other core components.
After searching the web for a while and fearing that I would need to reinstall all the VMs (luckily I have a golden VM template and I use puppet, so this wouldn't have been terrible, just very annoying), I found the release notes for
Centos 6.3, which stated
If you update CentOS 6.2 to 6.3 with certain packages only, the system
may hang upon reboot. For example, this happens if you apply only the
security updates from CentOS 6.3. To fix this issue, boot the system
with a "enforcing=0" kernel option and update selinux policy to
3.7.19-155.el6_3 (or later) available in the 6.3 updates repository.
Phew, I had a solution. I restarted all the VMs with the boot option
enforcing=0, and updated the following two package
- selinux-policy
- selinux-policy-targeted
Some of the VMs had an inconsistent rpm database (2 VMs with unfinished transactions and 1 with duplicate packages). I was able to sort this out by running
yum-complete-transactions and cleaning any of the unfinished transactions.
After that, all of the VMs started properly without any issues. Unfortunately, the one that failed to start up is the one that has all the core network services (dns, dhcpd, ntp, etc). And further it failed with the message
Kernel panic - not syncing - Attempted to kill init!. I particularly don't like KP. But I assumed that it was related to selinux issue.
I rebooted the VM with
enforcing=0 and it did boot up, which pointed to an error on the selinux database. I run
semodule -B to rebuild the selinux policy and I got an odd error saying that it couldn't find some files. The fix was quite simple
- Removed the policy packages with yum remove selinux-policy selinuc-policy-targeted
- Reinstalled the packages with yum install selinux-policy selinuc-policy-targeted
- And to be sure I run semodule -B again
This time I didn't get any errors and the VM restarted without a KP.
So last night I've learned:
- Don't do any patching/upgrading half an hour before your target time to go to sleep.
- Respect selinux
- Always upgrade first selinux policy files, before upgrading any other OS library package
- Do a snapshot of the VMs before any big patching (schoolboy error)
