We have restored service to Node5 after a complication with R1Soft CDP Backups and the operating RAID array was encountered. The system is building a new drive in to the RAID array now, which will cause intermittent speed issues, depending on where you files are physically located on the actual hard drives (in fact it is one of the least utilized drives, so speed issues should be minimal). As with all Virtuozzo system crashes, Quotas must be re-calculated. This occurs over the course of the next several hours as each VPS is taken down, recalculated, and then brought back online, which will cause roughly 15-30 minutes of additional downtime for most VPS configurations. This will occur over the next 12-16 hours, and unfortunately we can't say when it will happen for your VPS, but it will happen.

We would like to sincerely apologize for the downtime today on Node5. We do our best to avoid service problems related to new technologies (in this case: R1Soft CDP 3.0), but we have not succeeded in this regard with the new backup software. We are interfacing directly with R1Soft concerning how to reduce the system strain encountered when actually taking backups. Until we can do that, however, backups will be suspended for the operating Virtuozzo nodes in Chicago.

We have been investigating the issue causing Node5 to crash. We will update this post when we have more information and a definitive timeline to service restoration. Thank you for your continued patience and understanding.

We would like to apologize to clients on the Titan and Atlas servers this morning. We scheduled 15 minutes downtime but there have been about 2 hours downtime in total. The R1Soft CDP Backups did not play well with Titan, resulting in several instances where the system had to be rebooted to clear zombie processes that were otherwise causing services to encounter problems with disk space usage. We have halted R1Soft Backups on both Titan & Atlas, as well as Node6, until we are able to interface with R1Soft directly concerning the problems we have seen on these servers.

Users on Goliath should see R1Soft Backups available via cPanel at this time, and Virtuozzo VPS users at the Chicago datacenter on Node3 and Node5 may notice degraded speeds today while the backups continue seeding.

The Titan server requires emergency maintenance to update the local kernel. This is not common since we implement Ksplice for automated reboot-less kernel updates, however with our new backup solution we have encountered a problem regarding the kernel which requires it be updated manually as soon as possible.

Scheduled Timeframe: Between 12PM and 1AM CDT 4/12/11

Expected Downtime: 15 minutes

We are pleased to announce that we will be upgrading our backup facilities across all Chicago Shared, Reseller, and VPS systems to the new R1Soft CDP 3.0 software beginning today. This process will disable backup access for a few days, but once the upgrade & backup seed process is completed, you will find the R1Soft Backups link in your cPanel for easy access to backups.

What does this mean for backups in the meantime? Access to R1Soft CDP backups will not be available again until after your server has been seeded in the new R1Soft CDP 3.0 system. This does mean you will not be able to gain access to the older backups, as well.

Are any changes being made to backup policy & retention? Yes. We are changing from an hourly backup schedule to a daily backup schedule with 7 day retention. This will improve service speeds throughout the course of the day, and keep backup hours to a minimum even on our largest systems.

Will I notice anything else? You may notice slight speed degradation while backups are being seeded for the first time. This process should be negligible in its impact on the currently running systems, however, and we do not expect it to cause any problems with site accessibility.