The primary partition of our primary nameserver system has encountered a catastrophic failure this afternoon. We are getting a new set of drives in place and reimaging them at this time. The primary nameserver should be back online within the next few hours. In the meantime, we will keep you posted here with further information. Service speed or availability should not be impacted by this problem, however if you do see any trouble, try clearing your DNS Cache to remedy the issue.

This affects the following nameserver IPs:

65.254.36.170 (ns1.theservercompany.com)
65.254.36.172 (ns1.geekstorage.com)

This should not cause any downtime for your site(s)!

All shared & reseller hosting servers will be down for scheduled maintenance on 11/29/2010 for roughly 10-15 minutes per server as we issue reboots to implement new software. The downtime will occur between 10:00PM and 11:59PM CST (GMT-6), and will last roughly 10-15 minutes for each individual server.

This maintenance is to enable CloudLinux on all of our shared & reseller hosting machines.

This update will improve service speeds significantly across all shared & reseller hosting services, and implement improved resource tracking & management per account. Users of the "x3" cPanel theme will also find new statistics in their cPanel display, corresponding to their CPU Resource Usage and Concurrent Connections.

If you have any questions regarding this maintenance, please e-mail us at [email protected].

Update @ 2200 hrs: Scheduled maintenance has begun on-time. Estimated downtime for each server is approximately 10-15 minutes.

Update @ 2240 hrs: Scheduled maintenance has been completed and all Shared Hosting and Reseller Hosting servers are back online, running CloudLinux!

We are currently experiencing an outage at our Los Angeles location, which has rendered services unavailable for the below servers. We will post updates as we receive them from the datacenter, and would like to thank everyone for their patience and understanding.

Metis
Iris
Omega
Alpha
Aries
Node1
Node2
Node4

7:50PM Update: It appears there has been a power outage at the Los Angeles datacenter. The power has been restored however as of yet we are not seeing our servers coming back online. We are awaiting further updates from the datacenter at this time.

7:56PM Update: These servers are coming back online at this time:

Metis
Iris
Omega
Node1
Node2
Node4

Note for VPS servers: Since this was an unclean shutdown, your VPS will be coming back online within ~20 minutes from the actual Node coming online, after which the node will perform quota maintenance within several hours, causing your VPS to go offline again for 10-30 minutes, and then come back online finally after the quota maintenance is completed. We apologize for this inconvenience.

8:51PM Update: The server listed below is back online at this time. We are working on finishing a FSCK on the Aries server, and the Omega server is pending further investigation regarding a disk issue. We hope to have further positive updates for you shortly.

Alpha

8:58PM Update: The server listed below is back online at this time. We are still waiting on further updates regarding the Omega server.

Aries

9:58PM Update: We are still working on getting the issues on the Omega server resolved.

11:40PM Update: We are still waiting on an update from the datacenter regarding the hardware issue that seems to have come up on the Omega server after the power outage earlier today. As of yet we still have not receive a response from the datacenter and we do not have an ETA to provide. It is likely the datacenter is backlogged with many other similar requests due to the power outage, so it is difficult to guess how long it may take to get this issue resolved. We promise to keep the blog as up to date as possible with new information as we receive it.

12:54AM Update: The datacenter has still not been able to get to our issue to bring the Omega server back online. Unfortunately there is still no ETA for a resolution.

3:48AM Update: The datacenter is currently looking in to the hardware issue on Omega. If the issue is what we expect, the server should be back online within the next hour. Another update to follow as soon as we receive word back from the datacenter.

4:58AM Update: Unfortunately we have not yet received an update from the datacenter regarding the root cause of the issue. We are still waiting to hear back from their end.

5:42AM Update: The datacenter has let us know that they will get back to the issues on Omega as soon as they can, there are still many pending issues on other servers that are slowing down the process of investigating the troubles on the Omega server.

7:42AM Update: We unfortunately do not have any significant updates to provide at this time regarding the Omega server. We can confirm there is a hardware issue on the Omega server, sadly this means that the server will be at the end of the datacenter support queue, since there are far more servers still offline that can be fixed more quickly that they need to attend to. We still do not have any sort of time frame as to when the Omega server will be back online, as we still do not know what exactly needs to be replaced and thus how much more work is required to get the server back online once the hardware is replaced. Optimistically, once the datacenter gets the faulty hardware replaced, we hope to have the server online very quickly afterwards.

12:54AM Update: Great news, Omega is back online! After a hard drive replacement, the primary raid array is back to normal operating status. Do note that hard drive functions may be slowed for some time as the RAID array is rebuilt on to the new hard drive, this can last up to a few days. We would like to thank everyone for their patience and understanding throughout this process.

On May 15th, 2010, between the hours of 4 AM to 6 AM CST, the Los Angeles DC network will be undergoing emergency maintenance to upgrade multiple network switches. We have been informed by the DC to expect two to three outages during the maintenance window, lasting approximately 15-20 minutes each.

This emergency maintenance is aimed at resolving the network issues that have been occurring over the last couple of weeks.

If you have any questions regarding this, please contact us at [email protected].

UPDATE:

This maintenance has been completed.

At approximately 12:30 PM CST the data center in Los Angeles, CA experienced a network and/or power failure. We are aware of this issue and will do everything we can to help resolve this issue as quickly as possible. At the current time we do not have an ETA for service restoration. We will update you as soon as we know more.

If you have any questions, please e-mail us at [email protected].

UPDATE:

Service has been restored at approximately 01:05 PM CST. We are not aware of the cause of today's outage, but we will find out and let you know.

UPDATE:

We've heard back from the data center regarding the cause of the downtime today:

"Todays network issue was related to a faulty switch card which took down one of the vlan routers. Unfortunately the errors in the logs looked to be related to network changes made yesterday and delayed diagnosing the real issue. Once network engineers reversed all changes made yesterday and the network continued having issues the decision was made to swap the card. Once the card initialized and came online traffic began flowing normally again.

In an effort to prevent further issues, we have setup a planed maintenance window for tomorrow morning (Saturday May 14th) between 2AM PST to 4AM PST. During this window there will be two to three service disruptions of 15-20 minutes each as multiple network switches are upgraded."