March 19, 2019 at 1:51 PM

VPS services were interrupted today at 11:30AM CDT (GMT-5) due to switching hardware failure. Internet connectivity for all VPSs was interrupted due to the hardware failure. Our initial reaction was to immediately replace the failed switch with a replacement we have on hand, however, there were technically difficulties preventing us from performing such replacement quickly.  We have instead re-routed connections through a different switch temporarily - note there will be no discernable difference from the VPS or the Internet, but physically the routing is slightly different than previously to allow VPSs to properly reach the Internet right now.

Once physical re-routing was completed, service was restored fully at 3:20PM CDT (GMT-5). Prior to this we did get a few hypervisors reconnected which were returned to service about an hour beforehand, but about 65% of VPSs were down for the duration.

We are consulting with the datacenter and our team to ensure the faulty switch is replaced safely and without additional downtime. We're also working on a better plan for future switching failure, and we are doing additional work on site to ensure everything can be transitioned back without more downtime and with better recourse options in the future.

We are very sorry for the severe outage today on our VPS hosting services. We do everything we can to prepare for hardware failure, but sometimes we aren't as well prepared as we might think, and this was one of those cases. We do our best to learn from our mistakes, and we hope this has not shaken your trust in our ability to deliver quality web hosting services going forward.