November 1, 2014 at 5:14 PMOur admin Geeks have been dealing with a number of DDoS attacks targeting the Helios system that began last night. The DDoS attacks have caused significant downtime for users on the primary shared IP of Helios. We do have a solution, and most customers on Helios have received e-mails pertaining to this. Let's look at how we handle a DDoS attack.
DDoS Mitigation
These are the steps we take to mitigate incoming and frequent DDoS attacks:
Step 1: Nullroute.
Our first line of defense against DDoS is to immediately halt incoming traffic to the attacked IP address(es). This is called a "nullroute." Our upstream provider nullroutes the attacked IP as soon as they see an attack and see that it meets specific criteria (namely, if it will impact other users significantly).
A nullroute does two things: it prevents other IPs in our own network from going down due to the incoming attack, and it also prevents routing of the attacked IP so the attacker can't reach the IP. This means they can't attack anything until our IP is back online. We leave the IP offline for a little while, and bring it back up once the attacker has lost interest. Often times this is the end of the attacks, but sometimes they pick back up later.
Step 2: Identify the attacked website.
Incoming DDoS attacks always have a target. The attacker has a grudge with a particular domain on the server, or maybe they just don't like the server. The underlying reason is rarely known; most attackers never reach out and let us know why.
If we can identify the attacked website, we can segregate the associated traffic to a different IP address. This way, we can ultimately just nullroute the IP hosting the targeted site instead of the IP hosting a few hundred websites. This limits service interruption to one account, and keeps other customers online during DDoS attacks. If the DDoS attacks never subside, then the account holder can choose to employ DDoS mitigation services to keep their site online through an attack. Mitigation services can be very costly.
Step 3: Account IP Dispersion.
If the attacked website is unable to be identified, we resort to something we call Account IP Dispersion. You may be familiar with your account IP address. The same IP address that hosts your account may also host anywhere from one to a few hundred websites. We try to bring that number down as far as possible by dispersing accounts among a dozen or more new IPs, and doing this relies on a bit of custom code we built here at GeekStorage. In short, the code scans each website on the attacked IP to determine whether the hosted domains on that account are using our nameservers. If all of the account domains point to our nameservers, then we can change the hosted IP reliably and without downtime. We do this by using both the old and new IPs to host the website while the IP transitions (which can take up to 24 hours).
The first Helios account IP dispersion is complete; no IP is host to more than 30 accounts (excluding the IP holding domains that do not point to our nameservers). Within 24 hours, if attacks continue, we will narrow down which account is being attacked to a subset of 30, and disperse those accounts once again until we determine the attacked website. Fewer accounts will be impacted by the DDoS attacks with each iteration of the account IP dispersion until ultimately only one account sees downtime.
Helios Status
The Helios server is currently 100% online. It has seen four downtime-causing DDoS attacks over the last 24 hours, but our Geeks are hard at work keeping an eye on incoming traffic and preparing for the next step should the attacks continue. We hope to post a quick resolution to this issue within the next 24 hours.
If you are concerned about additional downtime, there are a couple things to keep in mind:
- Migration to a new server is almost as effective as migration to a new IP. Most customers are already on a new IP. The likelihood of additional downtime for any particular account has been reduced by about 85% by the account IP dispersion. There is also a small chance that your site is the target, in which case a migration will not help.
- Enabling CloudFlare via cPanel can improve your site speed and reliability. This holds true whether your account is being affected by DDoS attacks or otherwise.
We hope this sheds some light on the actions we are taking to improve availability on the Helios server despite recent DDoS attacks. If you have any questions for our support team, please don't hesitate to ask. We are available 24/7 via our support desk or at [email protected].
