Category: "Chicago DC"

Aries - Unplanned Outage 03/27/2011 (Resolved)

March 27, 2011 at 8:24 PM

03/27/2011 @ 1957 hours CST:

We are aware of an issue with the Aries server, which is currently web inaccessible, and we are working to investigate and resolve this issue as quickly as possible. Thank you for your patience.

03/27/2011 @ 2025 hours CST:

Aries is now back online. Upon reboot of the server Aries underwent a file systems check, which caused approximately 20 minutes of additional downtime.

03/27/2011 @ 2033 hours CST:

Upon investigation of the incident, it was discovered that a user on the server had been compromised due to their having installed an outdated web script. The attacker used his limited access to this user's account to operate an outbound DDoS attack, which caused the server to overload. This activity has been halted by us and this user's account has been suspended pending additional maintenance. This is a good reminder to always ensure that your website's scripts are kept up-to-date! It should be noted that the attacker did not gain any access to the server which would affect users other than the specific user targeted due to his script vulnerability.

If you have any questions, please contact us at [email protected].

Node4 - Unplanned Outage (01/17/2011) (RESOLVED)

January 17, 2011 at 2:36 AM

01/17/2010 @ 0152 hours:

We are aware of an issue with Node4 and are currently in the process of rebooting the server. VPS Nodes can take a while to come back online, as every VPS will need to perform a file systems check (FSCK) before starting up.

01/17/2010 @ 0252 hours:

VPS are starting to come back online after the reboot.

All Servers - Planned Maintenance (11/29/2010)

November 28, 2010 at 2:09 PM

All shared & reseller hosting servers will be down for scheduled maintenance on 11/29/2010 for roughly 10-15 minutes per server as we issue reboots to implement new software. The downtime will occur between 10:00PM and 11:59PM CST (GMT-6), and will last roughly 10-15 minutes for each individual server.

This maintenance is to enable CloudLinux on all of our shared & reseller hosting machines.

This update will improve service speeds significantly across all shared & reseller hosting services, and implement improved resource tracking & management per account. Users of the "x3" cPanel theme will also find new statistics in their cPanel display, corresponding to their CPU Resource Usage and Concurrent Connections.

If you have any questions regarding this maintenance, please e-mail us at [email protected].

Update @ 2200 hrs: Scheduled maintenance has begun on-time. Estimated downtime for each server is approximately 10-15 minutes.

Update @ 2240 hrs: Scheduled maintenance has been completed and all Shared Hosting and Reseller Hosting servers are back online, running CloudLinux!

Chicago - Downtime (09/12/2010)

September 12, 2010 at 12:02 PM

During major network upgrades at the Chicago datacenter very early this morning (2AM CDT) we began to experience heavy packet loss on all Chicago servers. These servers include:

Apollo
Node3
Node5
Node6
Zeus

After around 7AM CDT the heavy packet loss turned in to a total inability to access the network. The datacenter is aware of this problem, and working with their hardware vendor to get the issue resolved as soon as possible.

This outage has stemmed from a scheduled multiple core router upgrade that was expected to not impact service availability for more than a few minutes at a time. Unfortunately, some unforeseen complications have resulted in many issues this morning. A link to a news feed from the datacenter can be found below, all updates will be posted as they are available at the link below, and also more information regarding the cause for the outage is available here.

https://support.steadfast.net/index.php?_m=news&_a=viewnews&newsid=285

2:57PM CDT Update: We have receive an ETA of 2 hours for resolution of the network issues from the datacenter at this time.

5:45PM CDT Update: Service at the Chicago location should now be back online, there may still be a little bit of packet loss as the datacenter is still working on the network hardware. We have gone ahead and scheduled a hard drive replacement on the Apollo (reseller) server for 10:30PM, this maintenance will last roughly 30 minutes and should improve speeds significantly once the new hard drive has been mirrored in the RAID array.