Online Wars: Surviving DDoS Attacks
Recently more than ever, hackers with different agendas are launching different attacks on many famous websites with an attempt to take them down, control the servers to attack other servers, corrupt or destroy the database or even steal valuable data. Those hackers launch their attacks using immature to sophisticated mechanisms. Their agenda might be political one like an Intelligent agency attacking its rivals’ websites during war, or religious groups attacking their opponents.
In the Arab world, these attacks are becoming more prominent. last September around 900 Sunni religious websites, Arabic news agencies and forum have been under intense attacks which disabled them. Later on, Alarabiya domain name was hacked.
In light of the ever increasing online threats I asked Ahmed Mekkawy the team leader of System Admins in eSpace to write a guest post about his experience in surviving a DDoS attack on a famous Arabic forum. Mekkaway is also one of the admins team of Egypt Gnu/Linux Users Group. He is well known of being fond of Free Software.
A famous arabic forum has been under really strong attacks that led the forum to go down more than once. In the end it was down for over a week. At that time the site owners contacted eSpace to try to secure the site. This task fell back on me as I’m the SysAdmins’ team leader.
After some investigations, I found out that there were multiple attacks, the main one was a Distributed Denial of Service (DDoS) coming from around 10k different IPs. The first thing that came into my mind is
The attack wasn’t continuous all the time, but it was more like an attack waves, and between them there were a relatively a small number of attackers who never stop around the clock.
The forum’s web application was an old one, it acts as an application server and a web server at the same time. It was single threaded so I couldn’t get use of the multi-core server processing power, my bottleneck at that time was the processor usage. So I decided to go for performance enhancement. So I added varnish as a reverse-proxy and nginx to act as a frontend, so the HTTP request is managed by nginx first, then passed to varnish to server it if it’s cached, if not in the cache, then it’s brought from our backend.
Lots of tweaks were made to both varnish and nginx. For those who don’t know those two software, varnish is the best reverse proxy I found that acts at fabulous speed, it even caches in memory to avoid HardDisk latency. But it’s not the thing that works out of the box, it takes time to configure, when you are using cookies, and you’ll have to learn VCL (varnish configuration language) and nginx In my opinion is the black horse in web servers market, that is so light, with high performance and stability and is an easily configured web server. By using varnish & nginx I could make use of the extra processor cores that I had and I already had lots of memory available for caching using Varnish.
After making all that, the load average dropped from around 5 to less than 0.5 which is great. Moreover, I have managed to find the attack pattern and detect it from nginx then closed the connection after serving 0 bytes. The attackers then became very aggressive, the attack waves reached 50k IPs, so I had to increase the
nginx workers up to 10 and increase the ulimit alot (made it 512k) and made lots of kernel tweaks in order to be able to handle the TCP load (like increasing TCP buffers, tcp fin_timout, … etc,) I even had to drop the IP_CONNTRACK kernel module and redesign the custom firewall that I made to use the TCP flags. The bottleneck at that phase was handling the TCP connections overhead on the kernel level.
After I made all the above, the site was still performing while the attack waves are active. Adding another stand-by server and automatic failover made the end user feels nothing about the attacks that were happening.
In few weeks an email from the attacker asking for $5,000 per month or else he’ll continue attacking. I Just didn’t answer him and he tried for a couple of more attack waves then everything became very stable, and the attacks were stopped, because they could not affect the website anymore :).
The attacks started in 2008, and it last lasted for about 8 months.