Category: "Chicago DC"

Unscheduled Reboot for Neptune Shared Hosting Server (11/11/2019)

November 11, 2019 at 5:02 PM

We apologize for a short unscheduled outage today on the Neptune server. We have issued an unscheduled reboot on Neptune today around 4:54PM CDT. We are still investigating the cause of the issue, but we believe unusually high memory usage to be the cause at this time. We will be actively monitoring for a recurrence of the situation which brought about the issue, and we do not currently expect any further downtime for the Neptune server. Service should be up again as of 5:00PM CDT (total downtime around 10 minutes).

Update (5:17PM): Customers hosted on IP address 162.249.125.46 may be offline still as a result of the same issue above. We have determined the memory spike was caused by a DDoS attack, which continues at this time. We are working on detaching the attacked domain from the server and using other mitigation tactics to get customers on 162.249.125.46 back online as quickly as possible. Thank you for your patience.

Update (6:33PM): Users on 162.249.125.46 should see their sites back online at this time. Please note services may feel sluggish on making connection attempts due to the ongoing DDoS attack. Services are up and running (speedily), but the network service is taking a bit longer to respond because it is bearing the brunt of the attack right now in the form of thousands of active connection attempts. We have configured the server temporarily to be able to handle this style of attack, although the downside is there are minor hiccups in the connection we are noticing as a result. We will revert to normal settings once the attack has subsided, and we thank you for your patience.

Unscheduled VPS Service Outage Affecting HV11 and SHV1 (Resolved - 3/22/2019)

March 22, 2019 at 11:28 AM

We believe there may have been another hardware failure from the same batch of switches that caused the previous outage. Currently impacted nodes are HV11 and SHV1 and we expect to have a quick resolution for connectivity. Internal VPS administration tasks may fail for a time until we restore internal service to the hypervisors. Additional updates will be posted here as we are able.

Update #1: We have identified another failed switch from the same batch as the previous failure. We are working to restore outbound connectivity for the three affected hypervisors and currently have an estimated timeframe of about an hour for resolution.

Update #2: HV11 VPS service has been restored, SHV1 to follow shortly.

Update #3: SHV1 VPS service has been restored.

SHV1 and HV11 VPS service has been restored at this time. We are still working to get other internal connections restored. Until this part of the solution is completed, internal VPS operations (reboots, backup restores) will fail. We are working to reroute this traffic but also to restore the faulty hardware as soon as possible.

Update #4: Hardware has been swapped on site as of 3/26/2019 and all VPS features are now fully restored.

Unscheduled VPS Service Outage (3/19/2019)

March 19, 2019 at 1:51 PM

VPS services were interrupted today at 11:30AM CDT (GMT-5) due to switching hardware failure. Internet connectivity for all VPSs was interrupted due to the hardware failure. Our initial reaction was to immediately replace the failed switch with a replacement we have on hand, however, there were technically difficulties preventing us from performing such replacement quickly.  We have instead re-routed connections through a different switch temporarily - note there will be no discernable difference from the VPS or the Internet, but physically the routing is slightly different than previously to allow VPSs to properly reach the Internet right now.

Once physical re-routing was completed, service was restored fully at 3:20PM CDT (GMT-5). Prior to this we did get a few hypervisors reconnected which were returned to service about an hour beforehand, but about 65% of VPSs were down for the duration.

We are consulting with the datacenter and our team to ensure the faulty switch is replaced safely and without additional downtime. We're also working on a better plan for future switching failure, and we are doing additional work on site to ensure everything can be transitioned back without more downtime and with better recourse options in the future.

We are very sorry for the severe outage today on our VPS hosting services. We do everything we can to prepare for hardware failure, but sometimes we aren't as well prepared as we might think, and this was one of those cases. We do our best to learn from our mistakes, and we hope this has not shaken your trust in our ability to deliver quality web hosting services going forward.

Scheduled Network Maintenance (4/20/2018) [Complete]

April 17, 2018 at 1:26 PM

We have scheduled network maintenance to be conducted between 4:00PM CDT (GMT-5) and 4:15PM CDT (GMT-5) on Friday (4/20/2018) to improve our network topology and infrastructure. We do expect a short outage of 2-5 minutes during the maintenance window.

The maintenance detailed above applies to the following services:

  • Unlimited Hosting
  • Performance Hosting
  • Reseller Hosting
  • VPS Hosting

Dedicated servers and backup services will still be accessible during the maintenance window.

Update as of 4/19/18 at 1:46PM: An issue has been identified which requires we push back the scheduled timeframe. We do not yet have details on the adjusted maintenance window but we will post an update here as soon as we know more.

This maintenance has been rescheduled for Friday 4/20/2018 between 4:00PM CDT (GMT-5) and 4:15PM CDT (GMT-5).

This maintenance has been completed successfully as of 4:37PM CDT 4/20/2018.

Service Outage (10/4/2017)

October 4, 2017 at 5:17 PM

At 2PM CDT (GMT-5) a power outage occurred at our upstream provider, leaving all services offline while the issue was remedied. Following power restoration, we still saw several servers offline and had to investigate the cause of this problem. One of our power distribution units (PDUs) had failed. We had a spare on site for exactly this type of occurrence, and it was installed as quickly as possible to bring all still-offline services back online at approximately 5PM CDT (GMT-5).

Although this issue was out of our control, we are pleased our emergency response went well with a full PDU failure being resolved in such a short period of time given the extreme situation. It is impossible to avoid hardware failure completely, but we are continually working to ensure we are prepared for such problems when they do arise as they did today.

We would like to thank everyone for their patience and understanding as we worked through the outage today. If you are still seeing any service problems, please let us know and we will check into any remaining issues right away.