Around  12:00AM (CDT) we identified a network connectivity issue which was affecting traffic to all of our US-based hosting services. After getting in touch with our upstream provider within a few minutes, they were aware of an issue with two routers on their network and were working to restore connectivity. Connectivity was restored around 1:30AM (CDT). We are now awaiting a detailed report from the datacenter regarding the issue which we will also post here on the Service Announcements blog as soon as possible. We would like to thank everyone for their patience and understanding; please know this is not normal (neither for us nor our upstream provider) and we do expect provisions will be put in place to prevent similar issues from occurring again in the future.

We have identified an issue on the Neptune server which requires an urgent maintenance reboot overnight tonight. We will perform the reboot between 11:00PM and 12:00AM on 6/30/2020 (tonight). The expected downtime should be less than 15 minutes, and we expect the maintenance to remedy speed issues with email attachment sending operations and backup operations/management.

We apologize for a short unscheduled outage today on the Neptune server. We have issued an unscheduled reboot on Neptune today around 4:54PM CDT. We are still investigating the cause of the issue, but we believe unusually high memory usage to be the cause at this time. We will be actively monitoring for a recurrence of the situation which brought about the issue, and we do not currently expect any further downtime for the Neptune server. Service should be up again as of 5:00PM CDT (total downtime around 10 minutes).

Update (5:17PM): Customers hosted on IP address 162.249.125.46 may be offline still as a result of the same issue above. We have determined the memory spike was caused by a DDoS attack, which continues at this time. We are working on detaching the attacked domain from the server and using other mitigation tactics to get customers on 162.249.125.46 back online as quickly as possible. Thank you for your patience.

Update (6:33PM): Users on 162.249.125.46 should see their sites back online at this time. Please note services may feel sluggish on making connection attempts due to the ongoing DDoS attack. Services are up and running (speedily), but the network service is taking a bit longer to respond because it is bearing the brunt of the attack right now in the form of thousands of active connection attempts. We have configured the server temporarily to be able to handle this style of attack, although the downside is there are minor hiccups in the connection we are noticing as a result. We will revert to normal settings once the attack has subsided, and we thank you for your patience.

We believe there may have been another hardware failure from the same batch of switches that caused the previous outage. Currently impacted nodes are HV11 and SHV1 and we expect to have a quick resolution for connectivity. Internal VPS administration tasks may fail for a time until we restore internal service to the hypervisors. Additional updates will be posted here as we are able.

Update #1: We have identified another failed switch from the same batch as the previous failure. We are working to restore outbound connectivity for the three affected hypervisors and currently have an estimated timeframe of about an hour for resolution.

Update #2: HV11 VPS service has been restored, SHV1 to follow shortly.

Update #3: SHV1 VPS service has been restored.

SHV1 and HV11 VPS service has been restored at this time. We are still working to get other internal connections restored. Until this part of the solution is completed, internal VPS operations (reboots, backup restores) will fail. We are working to reroute this traffic but also to restore the faulty hardware as soon as possible.

Update #4: Hardware has been swapped on site as of 3/26/2019 and all VPS features are now fully restored.

VPS services were interrupted today at 11:30AM CDT (GMT-5) due to switching hardware failure. Internet connectivity for all VPSs was interrupted due to the hardware failure. Our initial reaction was to immediately replace the failed switch with a replacement we have on hand, however, there were technically difficulties preventing us from performing such replacement quickly.  We have instead re-routed connections through a different switch temporarily - note there will be no discernable difference from the VPS or the Internet, but physically the routing is slightly different than previously to allow VPSs to properly reach the Internet right now.

Once physical re-routing was completed, service was restored fully at 3:20PM CDT (GMT-5). Prior to this we did get a few hypervisors reconnected which were returned to service about an hour beforehand, but about 65% of VPSs were down for the duration.

We are consulting with the datacenter and our team to ensure the faulty switch is replaced safely and without additional downtime. We're also working on a better plan for future switching failure, and we are doing additional work on site to ensure everything can be transitioned back without more downtime and with better recourse options in the future.

We are very sorry for the severe outage today on our VPS hosting services. We do everything we can to prepare for hardware failure, but sometimes we aren't as well prepared as we might think, and this was one of those cases. We do our best to learn from our mistakes, and we hope this has not shaken your trust in our ability to deliver quality web hosting services going forward.