Chicago Outage (4/20/2011)

April 20, 2011 at 12:47 PM

A network issue has caused about 45 minutes of downtime at the Chicago datacenter, this outage affected the following servers:

Apollo
Node3
Node5
Node6
Zeus

As well as our primary server hosting GeekStorage.com, causing an outage in support channels, as well. Services are restored at this time to all servers, and we are continuing to investigate the cause of the issue. There was a problem with the switch hosting all of these servers that has caused the outage, we are still investigating why & how this issue occurred so suddenly without warning, and we have plans to replace the switch during our planned migration in several days (check the post previous to this one for more details on the migration).

Chicago DC - Server Relocation Scheduled

April 9, 2011 at 8:12 AM

We have finalized migration date for all servers at the Chicago DC to the new datacenter location a couple blocks away. The migration will affect the following servers:

  • Apollo
  • Zeus
  • Node3
  • Node5
  • Node6

The downtime will last approximately 3 hours per server, after which service will be restored on the same IPs. No reconfiguration or DNS modifications are necessary. The maintenance is scheduled to begin & end on Monday, April 25th, 2011, between 12:00AM and 6:00AM CST. GeekStorage technicians will be on-site to perform the migration and ensure downtime is kept to a minimum.

If you have any questions or concerns, please contact our support team at [email protected].

Chicago DC - Server Relocation Maintenance Notice

March 27, 2011 at 9:15 PM

This is notice of an upcoming maintenance window for ALL servers located in our 350 E. Cermak Rd. data center in Chicago, IL.

During the period between April 19th – April 26th we will be moving servers located in our cabinet at 350 E. Cermak Rd. to our cabinet at 725 S. Wells St.

We are still in the planning stages of this relocation and will provide a tentative date and time that each server will be moved in the near future. This is a physical relocation of the server hardware from one data center to another. As such, this move will cause downtime for each server when it is moved. The move will be performed in phases and downtime will be kept to an absolute minimum.

In the weeks leading up to this maintenance window, we will be preparing the network and organizing this relocation to be as quick as possible. This move will NOT require any change of IP space. The typical relocation scenario for each server will include a graceful shutdown, physical move, cabinet installation and then server boot.

In the near future, we will send all of our customers a detailed e-mail with the scheduled date, time and anticipated downtime for your server’s move during this maintenance window. We do not anticipate this downtime to be long. The two data centers are 2.3 miles from one another.

We will have personnel onsite at the data center to oversee the hardware relocation.

Why move?

Our data center provider recently built a brand new data center at 725 S. Wells St. Our current data center location at 350 E. Cermak Rd. is near full on capacity and any future equipment would have to be placed in the new data center. This move allows us room to expand and keep our equipment in the same data center, allowing us many options for connectivity between our hardware.

The network is the same at both data center locations.

Have questions?

If you have any questions regarding this upcoming maintenance window, please contact us at [email protected].

If you have service located on a server in our Chicago data center, please keep an eye on your inbox for additional details.

Aries - Unplanned Outage 03/27/2011 (Resolved)

March 27, 2011 at 8:24 PM

03/27/2011 @ 1957 hours CST:

We are aware of an issue with the Aries server, which is currently web inaccessible, and we are working to investigate and resolve this issue as quickly as possible. Thank you for your patience.

03/27/2011 @ 2025 hours CST:

Aries is now back online. Upon reboot of the server Aries underwent a file systems check, which caused approximately 20 minutes of additional downtime.

03/27/2011 @ 2033 hours CST:

Upon investigation of the incident, it was discovered that a user on the server had been compromised due to their having installed an outdated web script. The attacker used his limited access to this user's account to operate an outbound DDoS attack, which caused the server to overload. This activity has been halted by us and this user's account has been suspended pending additional maintenance. This is a good reminder to always ensure that your website's scripts are kept up-to-date! It should be noted that the attacker did not gain any access to the server which would affect users other than the specific user targeted due to his script vulnerability.

If you have any questions, please contact us at [email protected].

Node4 - Unplanned Outage (01/17/2011) (RESOLVED)

January 17, 2011 at 2:36 AM

01/17/2010 @ 0152 hours:

We are aware of an issue with Node4 and are currently in the process of rebooting the server. VPS Nodes can take a while to come back online, as every VPS will need to perform a file systems check (FSCK) before starting up.

01/17/2010 @ 0252 hours:

VPS are starting to come back online after the reboot.