Category: "Chicago DC"

Titan DDoS (Resolved)

September 18, 2012 at 2:27 PM

Titan is currently being targeted by a distributed denial of service (DDoS) attack, which is causing intermittent service interruptions to users on this server.

We are currently working to resolve this issue as quickly as possible.

09/18/2012 @ 03:46 PM CST

At this time, service should start returning to normal. Thank you for your patience while we worked to mitigate this incident.

Emergency Reboot HV3 (Resolved)

August 31, 2012 at 10:41 PM

08/31/2012 @ 10:36 PM CST

We are issuing an emergency reboot on the HV3 hypervisor to address a network issue that arose late this evening. We are seeing a halt in operation on the internal management network of HV3, causing many functions to be offline and inaccessible. We hope this reboot will resolve the issues. If we continue to see problems, we will need to prepare for further downtime to address a possible hardware issue. We will update this post when we know more.

08/31/2012 @ 10:46 PM CST

At this time the server has rebooted and VPS are coming back online. Estimated downtime for each VM on the server was less than 10 minutes. So far we have been able to confirm that it doesn't appear to be a hardware issue related to the server's NIC. We are still investigating the root cause of the network issue that occurred, which caused certain management functions to stop working. We do know that a mistake on a NOC tech's part led to the hypervisor's primary connection to the switch to be disconnected momentarily, causing VM's to go offline, as he was attempting to help us troubleshoot the original network issue.

Goliath Offline (Resolved)

August 29, 2012 at 9:55 AM

08/29/2012 @ 09:50 AM CST

We are actively working to resolve an issue with Goliath that is causing the server to be offline.

This post will be updated with more information as soon as it becomes available.

08/29/2012 @ 10:15 AM CST

We are seeing a kernel panic on Goliath. We are resolving the kernel issue at this time and we expect the server to be back online within 20 minutes.

08/29/2012 @ 11:15 AM CST

We are still working to resolve the kernel panic issue on the Goliath server. Websites will still appear offline as we work as quickly as possible to restore service. Our apologies for the inconvenience and you have our thanks for your continued patience.

08/19/2012 @ 12:20 PM CST

The root cause of the issue has been identified and corrected at this time. At this time, service has been restored.

More Details

We apologize for the significant downtime this morning. We have just resolved the issue and the system is coming back online at this time. You can expect your website to be accessible immediately.

We identified the issue initially as a kernel panic. Unfortunately, due to some problems with our debugging tools, our recovery time was delayed significantly versus typical similar situations. Once we were able to begin debugging the issue, it was identified as a filesystem configuration error or malfunction. Ultimately it was determined to be a filesystem configuration error on the primary (root) filesystem and it has now been resolved.

We would like to extend our deepest apologies for the downtime today. This downtime could not have been avoided; however, it could have been reduced significantly had our debug & recovery tools been more thoroughly tested prior to this issue. We are making corrections to our internal processes & recovery methods to eliminate extended downtime issues such as what occurred today.

We will offer SLA credits to all customers on the Goliath server upon request, up to 1 month service credit, per our SLA.

I would like to personally thank you for your continued patronage to our business and for your patience and understanding throughout the service outage today.

Regards,

Matt Eli

Primary Nameserver Down (Resolved)

August 1, 2012 at 2:23 PM

The primary partition of our primary nameserver system has encountered a catastrophic failure this afternoon. We are getting a new set of drives in place and reimaging them at this time. The primary nameserver should be back online within the next few hours. In the meantime, we will keep you posted here with further information. Service speed or availability should not be impacted by this problem, however if you do see any trouble, try clearing your DNS Cache to remedy the issue.

This affects the following nameserver IPs:

65.254.36.170 (ns1.theservercompany.com)
65.254.36.172 (ns1.geekstorage.com)

This should not cause any downtime for your site(s)!

Xen HV3 Scheduled Maintenance (Resolved)

July 16, 2012 at 1:05 PM

We are performing an upgrade to the Xen HV3 hardware configuration, as the first in a line of multiple hypervisor upgrades which will provide RAM upgrades for the currently available VPS plans (if you are not on HV3, be patient, upgrades are coming to all hypervisors soon :)). Downtime is expected to be around 20-30 minutes, and occur on Thursday morning 7/19/2012 between 12:01AM and 4:00AM CDT.

07/19/2012

The server upgrade has been completed at this time.