Category: "Chicago DC"

SpamExperts Mail Server Outage (7/30/2026)

July 30, 2026 at 10:04 AM

We are currently working on resolving an issue on the SpamExperts mail server hosting the primary mail exchanger for our shared and reseller hosting clients. Customers using SpamExperts for inbound filtering, and all customers using outbound mail, will currently see issues with mail service. A drive failure occurred this morning which took down the system, and we are working on recovering the system now. We are operating in three phases:

1) Backup data from failed disk RAID 1 array. (this is what we are working on now)
2) Boot system to return service availability.
3) Replace drive and rebuild RAID 1 array.

Currently we are on step 1, and this will take a bit more time (estimating about 1-2 hours as of this post). Once that is finished, we should be able to boot the system back up until a replacement drive can be installed and the RAID array rebuilt for full data integrity.

We will post additional updates to the blog here in this post. Thank you for your patience this morning as we work through this hardware issue.

UPDATE 11:53AM CDT (GMT-5):

The main SpamExperts server has been brought back online and we are now waiting on SpamExperts technicians to investigate the issue with domain management right now in the SpamExperts interface and API. Access to SpamExperts will not be possible until this is resolved, and mail delivery may still be complicated as a result.

UPDATE 3:31PM CDT (GMT-5):

The main issue remaining with the software has been resolved and mail service should now be returned to normal functionality.

Shared Service Maintenance (7/9/2026)

July 9, 2026 at 11:38 PM

Emergency maintenance is taking place now to address the new GhostLock (CVE-2026-43499) vulnerability across all shared hosting services as systems require a quick reboot. Downtime should be minimal (5-15 minutes); we apologize for the interruption in service.

Datacenter Outage (6/9/2026)

June 10, 2026 at 1:28 AM

Today we experienced a major outage at the Chicago datacenter hosting our shared, reseller, VPS, and dedicated hosting services. Beginning at 1:50PM (all times are in CDT), we noticed a major outage impacting all services in Chicago, and requested additional information from our upstream provider. We received details of a major ongoing DDoS attack indicating techs on-site were attempting to mitigate the DDoS attack at that time. The outage ended around 3:00PM, however the attack picked back up again about an hour later, and continued well into the night as the datacenter team continued working to mitigate the attack. Ultimately, additional bandwidth capacity was added at the datacenter to allow the attack to be better handled by both the network and the network admin team. Services came back online around 9:30PM, though there are still techs monitoring and working on making sure there is no further impact to service availability. We are not 100% certain this is the end of all issues but services have been stable for a few hours now and that is a good sign.

We did encounter several issues with communication during this outage, most notably our Twitter/X account was inaccessible to us during the attack (because our site and email was also down during the outage). We will be working on ensuring this is not an issue should future service interruptions of this nature occur.

We want to thank everyone for their patience and understanding throughout this outage; if you do still see any issues impacting your services please submit a support ticket by emailing [email protected] and we will investigate right away.

Nameserver IP Changes

March 12, 2025 at 3:23 PM

Beginning in April, we will be retiring a number of legacy nameserver IP addresses and switching to a new nameserver IP in their place. This change is important to all customers running private or custom nameservers, and further information is provided below. This change mostly impacts reseller customers and each reseller will receive an email concerning these changes, but any account using private or custom nameservers will need to make an update to their nameserver settings, as well.

Nameserver IP Changes

We will be retiring these nameserver IPs at the end of March:
162.249.124.12
162.249.124.13
162.249.124.14
162.249.125.18
162.249.126.4
104.36.58.3

The new nameserver IP to replace the IPs mentioned above is:
104.36.63.107

NOTE: For resellers, this applies only if you are using our redundant DNS IPs for your custom/private nameservers. You should check your current nameserver IPs at your domain registrar to be sure whether you need to make any changes to your nameservers.

If you are using one of the above listed "retiring" IP addresses for one of your nameservers, be sure to replace the old IP with the new IP listed above. It is imperative that you check on this setting prior to April 1, 2025. Failure to do so could cause DNS instability for your clients.

The current/new settings for our redundant DNS addresses is as follows:
ns1 -> 104.36.63.107
ns2 -> 144.217.82.16

If your nameserver IP settings instead match your private reseller IPs (something like 104.36.62.x), that is acceptable as well and those settings do not require adjustment.

If changes are required, it is also advised to update the DNS entry for the nameserver (ns1 or ns2) in your DNS zone file in the cPanel Zone Editor, as well.

Thank You

If you do have any questions for us, we are always available via email at [email protected] so please feel free to get in touch with us whenever you like. We will be happy to double-check your settings and provide information for completing the above changes at your domain registrar.

Outage at Chicago Datacenter (2/22/2025)

February 22, 2025 at 11:23 AM

We are currently seeing several dedicated server and VPS clients offline as a result of an issue at the Chicago datacenter location. We believe there has been a power distribution unit (PDU) failure and we are working with the datacenter to determine the cause and appropriate course of action. We will continue to post updates here to this announcement as more information becomes available.

Services impacted:
DNS (partial outage)
VPS services on OnApp Platform (18 services impacted)
Dedicated Servers (at least 5 servers impacted)

Update 11:46AM (CDT): Datacenter has confirmed one of the PDUs servicing our equipment has failed. Techs are working on replacing the PDU now.
Update 12:37PM (CDT): The PDU has been replaced and all dedicated server clients should be back online. Partial VPS and DNS service outage will require additional work and updates will be provided via support ticket to each impacted customer.