September 12, 2010 at 12:02 PM

During major network upgrades at the Chicago datacenter very early this morning (2AM CDT) we began to experience heavy packet loss on all Chicago servers. These servers include:

Apollo
Node3
Node5
Node6
Zeus

After around 7AM CDT the heavy packet loss turned in to a total inability to access the network. The datacenter is aware of this problem, and working with their hardware vendor to get the issue resolved as soon as possible.

This outage has stemmed from a scheduled multiple core router upgrade that was expected to not impact service availability for more than a few minutes at a time. Unfortunately, some unforeseen complications have resulted in many issues this morning. A link to a news feed from the datacenter can be found below, all updates will be posted as they are available at the link below, and also more information regarding the cause for the outage is available here.

https://support.steadfast.net/index.php?_m=news&_a=viewnews&newsid=285

2:57PM CDT Update: We have receive an ETA of 2 hours for resolution of the network issues from the datacenter at this time.

5:45PM CDT Update: Service at the Chicago location should now be back online, there may still be a little bit of packet loss as the datacenter is still working on the network hardware. We have gone ahead and scheduled a hard drive replacement on the Apollo (reseller) server for 10:30PM, this maintenance will last roughly 30 minutes and should improve speeds significantly once the new hard drive has been mirrored in the RAID array.