In this off-topic post, I’m going to discuss some behind-the-scenes “fun” that may happen when managing the web server used to host the CNX Software website. From time to time, the server becomes unreachable, but I can still access its console, and notice a very high CPU load (over 100) on a VPS with four cores, while the CPU load is typically 0.5 to 2 under normal circumstances.
That’s usually due to a DoS (denial of service), DDoS (distributed denial of service), or some bug I can’t identify. An easy way to solve this issue is to log in to Cloudflare and set the “Under Attack Mode” to on.
It will show all visitors a JavaScript challenge the first time they visit the website, and the CPU usage typically drops back to normal level within a minute or so. That means I can access my website and so do regular visitors. It’s now without drawback though, as the RSS feed won’t work anymore, and some browsers seem to have some issues with the JavaScript challenge going into a loop. So I typically turn it on temporarily for a few hours until the (D)DoS attack is gone.
In the case of a DoS attack where one or only a few hosts may be the cause of the high CPU usage, as opposed to a DDoS where many hosts are involved, it’s also possible to use GoAccess open-source web analytics application. to monitor the IP addresses with lots of requests. In a Ubuntu/Debian server, it can be installed as follows:
|
1 |
sudo apt install goaccess |
If one would want to analyze an access.log generated by nginx, we can run the following command:
|
1 |
goaccess access.log |
and select “Common Log Format”.
It will process all the entries in access.log and provide a summary of key metrics. But it’s not super useful to fight against DoS attacks since the data not only contains recent entries, and in my case, it was about 5 days of traffic, so the top IP addresses were from Googlebot and Bingbot which are both so-called “good bots”.
Luckily the tool can also only show real-time statistics. We’ll first need to edit /etc/goaccess/goaccess.conf configuration file to enable the parameters shown in the screenshot above. It’s just a question of uncommenting a few lines:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
# The following time format works with any of the # Apache/NGINX's log formats below. # time-format %H:%M:%S ... # The following date format works with any of the # Apache/NGINX's log formats below. # date-format %d/%b/%Y ... # Common Log Format (CLF) log-format %h %^[%d:%t %^] "%r" %s %b |
Now we can disable “Under Attack Mode” in CloudFlare and run the following command:
|
1 |
tail -f access.log | goaccess - |
We can let it run for a minute or two, and will show the most various metrics including the most accessed files, the top visitor IPs, and so on.
The next step is to check the IP addresses. For example, 2a06:98c0:3600::103 is a CloudFlare IPv6 address designed for workers. While it could be an issue, I’m not going to block that one. The 66.249.65.xx 66.249.75.xxx addresses typically come from various Google bots, so we’ll leave those alone too. But it’s unclear what 4.227.36.7 is for and some websites report it as a spam IP address, so I’ve added it to the IP Access Rules in Cloudflare->Security-WAF->Tools along with other suspicious IP addresses reported in GoAccess and enabled a “Managed Challenge” for each of them to try to mitigate the current DoS attack.
After a while, it looks like it partially worked as the CPU load dropped to 3 to 5. I’m not sure it’s the best solution, but that’s what I’m doing to try to keep the website running. It’s a continuous fight since the bots will change IP addresses and attack vectors, so I’m also using other mechanisms, but those are WordPress-specific.
Jean-Luc started CNX Software in 2010 as a part-time endeavor, before quitting his job as a software engineering manager, and starting to write daily news, and reviews full time later in 2011.
Support CNX Software! Donate via cryptocurrencies, become a Patron on Patreon, or purchase goods on Amazon or Aliexpress
