We believe there may have been another hardware failure from the same batch of switches that caused the previous outage. Currently impacted nodes are HV11 and SHV1 and we expect to have a quick resolution for connectivity. Internal VPS administration tasks may fail for a time until we restore internal service to the hypervisors. Additional updates will be posted here as we are able.

Update #1: We have identified another failed switch from the same batch as the previous failure. We are working to restore outbound connectivity for the three affected hypervisors and currently have an estimated timeframe of about an hour for resolution.

Update #2: HV11 VPS service has been restored, SHV1 to follow shortly.

Update #3: SHV1 VPS service has been restored.

SHV1 and HV11 VPS service has been restored at this time. We are still working to get other internal connections restored. Until this part of the solution is completed, internal VPS operations (reboots, backup restores) will fail. We are working to reroute this traffic but also to restore the faulty hardware as soon as possible.

Update #4: Hardware has been swapped on site as of 3/26/2019 and all VPS features are now fully restored.

VPS services were interrupted today at 11:30AM CDT (GMT-5) due to switching hardware failure. Internet connectivity for all VPSs was interrupted due to the hardware failure. Our initial reaction was to immediately replace the failed switch with a replacement we have on hand, however, there were technically difficulties preventing us from performing such replacement quickly.  We have instead re-routed connections through a different switch temporarily - note there will be no discernable difference from the VPS or the Internet, but physically the routing is slightly different than previously to allow VPSs to properly reach the Internet right now.

Once physical re-routing was completed, service was restored fully at 3:20PM CDT (GMT-5). Prior to this we did get a few hypervisors reconnected which were returned to service about an hour beforehand, but about 65% of VPSs were down for the duration.

We are consulting with the datacenter and our team to ensure the faulty switch is replaced safely and without additional downtime. We're also working on a better plan for future switching failure, and we are doing additional work on site to ensure everything can be transitioned back without more downtime and with better recourse options in the future.

We are very sorry for the severe outage today on our VPS hosting services. We do everything we can to prepare for hardware failure, but sometimes we aren't as well prepared as we might think, and this was one of those cases. We do our best to learn from our mistakes, and we hope this has not shaken your trust in our ability to deliver quality web hosting services going forward.

We have scheduled network maintenance to be conducted between 4:00PM CDT (GMT-5) and 4:15PM CDT (GMT-5) on Friday (4/20/2018) to improve our network topology and infrastructure. We do expect a short outage of 2-5 minutes during the maintenance window.

The maintenance detailed above applies to the following services:

• Unlimited Hosting
• Performance Hosting
• Reseller Hosting
• VPS Hosting

Dedicated servers and backup services will still be accessible during the maintenance window.

Update as of 4/19/18 at 1:46PM: An issue has been identified which requires we push back the scheduled timeframe. We do not yet have details on the adjusted maintenance window but we will post an update here as soon as we know more.

This maintenance has been rescheduled for Friday 4/20/2018 between 4:00PM CDT (GMT-5) and 4:15PM CDT (GMT-5).

This maintenance has been completed successfully as of 4:37PM CDT 4/20/2018.

At 2PM CDT (GMT-5) a power outage occurred at our upstream provider, leaving all services offline while the issue was remedied. Following power restoration, we still saw several servers offline and had to investigate the cause of this problem. One of our power distribution units (PDUs) had failed. We had a spare on site for exactly this type of occurrence, and it was installed as quickly as possible to bring all still-offline services back online at approximately 5PM CDT (GMT-5).

Although this issue was out of our control, we are pleased our emergency response went well with a full PDU failure being resolved in such a short period of time given the extreme situation. It is impossible to avoid hardware failure completely, but we are continually working to ensure we are prepared for such problems when they do arise as they did today.

We would like to thank everyone for their patience and understanding as we worked through the outage today. If you are still seeing any service problems, please let us know and we will check into any remaining issues right away.

We are currently investigating an issue affecting the OnApp control panel and two hypervisors hosting VPS customers. We have techs on site investigating the cause and working to get this issue resolved as soon as possible. Updates will be posted here as soon as we have more information.

[ posted at 8:16PM CDT 4/8/2017 ]

The issue has been identified as a partial PDU failure in the rack servicing our Chicago-based VPS services. This took down two hypervisors and the OnApp control panel server for roughly 60 minutes today (4/8/2017). We have identified the failing PDU ports and will have a replacement installed shortly; service is already restored and VPS services should be back to normal operation at this time.

[ update posted at 9:02PM CDT 4/8/2017 ]