Submitted by Piete Brooks on Fri, 05/09/2014 - 07:44
A misconfiguration in the networking kit caused an outage yesterday afternoon. This caused many Linux Xen VMs to loose access to their root filesystems, so they mounted them read-only. After rebooting some of them, one of the Xen Servers lost the ability to reboot VMs, so itself had to be rebooted, causing all its VMs to crash. Many of the filesystems were deemed "unfixable" by fsck, so fsck had to be manually run before the systems rebooted.
Please let sys-admin know of any remaining problems.
