So what happened this time? OMG what a complete and utter debarcle....
1. Firstly the upgrade kernel didn't appear to install correctly. Rebooting brought me straight back to the existing f19 installation. A few readings of the known issues (such as a separate /var) and a quick tweak here and there, then fedup was re-run. After the server was rebooted it never came back. Now as this server is in a data centre nowhere near me, this is quite problematic. Thank goodness for web interfaces which allow me to trigger remote ctl-alt-del's or hardware resets. Many, many hardware resets were done to no avail. Booting into the rescue mode at least allowed me to make some progress, and tweak a few more things. Still, after a reboot, it went away and never came back. A few more resets and a few more rescue mode sessions later, i find the rpmdb had been completely corrupted and needed rebuilding. No problem, that's easy to fix
Code: Select all
# rpm --rebuilddb
1a. During one of these many reboot cycles I managed to get the server up and booted, but then found that none of the firewalls would load?
Code: Select all
ERROR: Your kernel/iptables do not include state match support. No version of Shorewall will run on this system
2. Time for the remote console. Bit of a pain in the arse, because you have to request it, wait for someone in the datacentre to connect it, and then you have 2hrs before they nick it back and stick onto somebody else's server. This is no problem. I can do this. Fire it up. Java, oh great, we all know how well Java, Remote Consoles and Linux play well together, not well. Lets try it anyway. I can see the video \o/ woo hoo. Panic on boot
Code: Select all
Invalid MD raid superblock
3. So while we have the remote console still (and we are fast approaching the 2hr limit) lets reinstall the proper f20 kernel and reboot. Nope, it's not playing games. It fails to mount half the filesystems and drops me to an emergency shell. These filesystems are on Logical Volumes (LV) which aren't active. Okay, again a simple fix.
Code: Select all
# lvchange -ay <lvname>
Code: Select all
# cp /etc/lvm/lvm.conf.rpmnew /etc/lvm/lvm.conf
Code: Select all
# dracut -f
4. Time to see what state the system is in
Code: Select all
# yum check
All I can say is it's a good job I do this sort of thing as a day job. Most normal users would have reinstalled a *long* time ago....
Stuart