Emergency Reboot HV3 (Resolved)
08/31/2012 @ 10:36 PM CST
We are issuing an emergency reboot on the HV3 hypervisor to address a network issue that arose late this evening. We are seeing a halt in operation on the internal management network of HV3, causing many functions to be offline and inaccessible. We hope this reboot will resolve the issues. If we continue to see problems, we will need to prepare for further downtime to address a possible hardware issue. We will update this post when we know more.
08/31/2012 @ 10:46 PM CST
At this time the server has rebooted and VPS are coming back online. Estimated downtime for each VM on the server was less than 10 minutes. So far we have been able to confirm that it doesn't appear to be a hardware issue related to the server's NIC. We are still investigating the root cause of the network issue that occurred, which caused certain management functions to stop working. We do know that a mistake on a NOC tech's part led to the hypervisor's primary connection to the switch to be disconnected momentarily, causing VM's to go offline, as he was attempting to help us troubleshoot the original network issue.