We have identified an issue on the Atlas server that requires immediately attention and will initiate downtime shortly. Downtime should be less than 30 minutes and there should be no other adverse effects of this maintenance.
This maintenance has been completed and service accessibility and optimum speeds should be restored on the Atlas server at this time.
The Goliath server has crashed this morning. We are looking in to the cause and working towards a resolution at this time. We will bring Goliath back online as soon as possible, however we do not yet have an ETA for resolution. Further updates will be posted here as they become available.
Update 9:30AM CDT: Goliath is running a Filesystems Check (FSCK) and should be back online within 15-30 minutes.
Update 9:36AM CDT: Goliath is back online at this time and services should again be accessible.
Final Update 9:52AM CDT: We do not have a specific cause of the crash however we can attribute the most likely cause to an I/O error. As such, we are tweaking configurations to help ensure this does not occur again, and enabling additional logging to monitor the server more closely for some time (all RAID arrays are in tact and optimal, and a FSCK was completed successfully, as well). We apologize for the downtime and would like to thank everyone for their patience this morning.
ATTENTION SHARED HOSTING USERS:
If you are experiencing a default cPanel page when browsing to your website hosted with us and you are using third party DNS, you will need to update your hosted IP at your DNS provider per our IPv4 Renumbering notices that we have e-mailed you.
You can find your hosting IP by logging in to cPanel, then click 'Expand Stats' on the left hand column, and then write down the hosted IP displayed there. Input that IP into your third party DNS control panel to update your website.
If you need assistance with this process, please open a support ticket and we'll be happy to help.
Emergency Maintenance Notification
**NOTE: This only applies to dedicated server customers who are located in LAS1. NO SHARED, RESELLER, OR VPS CUSTOMERS ARE LOCATED IN LAS1.***
Start Time: Tuesday, July 22 2014 4:00PM Pacific Time
Estimated Completion Time: Unknown, current estimate 12-18 hours
Expected Service Impact: None
Possible Service Impact: High
Maintenance Information:
LAS1 is currently operating on emergency backup generator power. The electric utility company which provides power to our LAS1 datacenter (NV Energy) has determined that our main service transformer has suffered a critical failure, resulting in a complete loss of utility power to the datacenter. We are actively working with their engine ers to complete a repair or replacement of the unit ASAP.
Our facility is currently operating normally on generator backup power. All redundant power systems have operated flawlessly thus far, and customer power has not been impacted in any way. Our fuel suppliers have been notified and are on standby should we require additional fuel at any time.
Please stand by for updates, as this situation is rapidly evolving.
Update #1: 07/22/2014 @ 10:37 PM Pacific Time
The data center has concluded emergency power maintenance without any impact to power delivered to customer equipment. NV Energy has completed replacement of our main utility transformer with a larger unit, which we hope will preclude any future incidents. We have transitioned off of our backup generators and are now running normally on utility power.
Please let us know if you have any questions regarding this matter.
Start Date: 06/15/2014
Start Time: 09:00 PM Central Time
Estimated End Date: 06/16/2014
Estimated End Time: 02:00 AM Central Time
Service: OnApp Platform Upgrade
Type of Work: Software Upgrade and Testing
Impact of Work: Intermittent service availability of VPS servers during upgrade and testing during the maintenance window
This notification is to inform you that on Sunday, June 15th, 2014, beginning at 09:00 PM Central Time, with an estimated ending on Monday, June 16th, 2014, at 2:00 AM Central Time, the OnApp Platform will have intermittent inaccessibility during a planned software upgrade and necessary testing. All support tickets will be answered as usual. We do ask that you refrain from submitting a ticket regarding your VPS during this maintenance window as any issue(s) you encounter are likely a direct result of this maintenance.
Update #1 - 9:58 PM Central Time, 06/15/2014
The start of the maintenance has been delayed until 10 PM Central Time. We still hope to have the maintenance fully completed by the 2 AM Central Time estimated ending.
Update #2 - 2:17 PM Central Time, 06/16/2014
We are wrapping up some maintenance work on the backend, but otherwise the maintenance has been completed successfully. Most VPS experienced less than 5 minutes of downtime (if any) and there were only a handful that experienced a slightly longer downtime, so we're thankful this maintenance went quite smoothly overall.
We will need to perform about 1 more hour of maintenance on the platform at 11 PM Central Time 06/16/2014. This is a pretty minor maintenance operation and we don't expect more than 5 minutes of downtime (if any) for all VPS on the OnApp platform.