In recent days, many organizations have fallen victims to cybercrime. More sophisticated botnets and other attack methods have been developed to ratchet the speed of these attacks. The most common attacks are DDoS(distributed denial of service, account takeover(ATO), and scraping content from websites. These attacks have dire consequences for the target organizations and need to be avoided at all costs.
Normally, Bad Bots can be defined as software applications that execute automated tasks with malicious intentions over the internet. They masquerade in the system by disguising the user agents. These bad bots may be;
- Bots or Servers linked to viruses or malware
- Government surveillance bots
- Botnet Attack Networks (Mirai)
- Gambling and Porn WebSites
- Vulnerability scanners
- Spam Referrers
- Bad User-Agent Strings
- Browser Adware and Malware (Yontoo etc)
- Link Research and Backlink Testing Tools
- SEO companies that your competitors use to try to improve their SEO
- Image Hotlinking Sites and Image Thieves
- Link Ranking Bots
- E-mail harvesters
- Sites Linked to Lucrative Malware, Adware, and Ransomware Clickjacking Campaigns
- Stopping Google Analytics Ghost Spam
This guide demonstrates how you can block Bad Bots, Spam, User-Agents, and Ransomware on Nginx.
Step 1 – Download the Nginx Bad Bot Blocker
The Nginx Bad Bot Blocker was developed by Mitchell Krog to be used in the Nginx web server. It can be downloaded on your Linux system using the command:
### With wget
wget https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/install-ngxblocker
### With curl
sudo curl -sL https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/install-ngxblocker -o install-ngxblockerOnce the file has been downloaded, make it executable.
chmod +x install-ngxblockerOn FreeBSD systems, the package is available in the default repository and can be installed with the command:
pkg install www/nginx-ultimate-bad-bot-blockerStep 2 – Download the Required files.
On other Linux systems, use the downloaded file in the /usr/local/sbin directory as below:
sudo ./install-ngxblockerThe installation will proceed as below:
Checking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt
** Dry Run ** | not updating files | run as 'install-ngxblocker -x' to install files.
Creating directory: /etc/nginx/conf.d
Creating directory: /etc/nginx/bots.d
REPO = https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master
Downloading [FROM]=> [REPO]/conf.d/globalblacklist.conf [TO]=> /etc/nginx/conf.d/globalblacklist.conf
Downloading [FROM]=> [REPO]/conf.d/botblocker-nginx-settings.conf [TO]=> /etc/nginx/conf.d/botblocker-nginx-settings.conf
REPO = https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master
Downloading [FROM]=> [REPO]/bots.d/blockbots.conf [TO]=> /etc/nginx/bots.d/blockbots.conf
Downloading [FROM]=> [REPO]/bots.d/ddos.conf [TO]=> /etc/nginx/bots.d/ddos.conf
Downloading [FROM]=> [REPO]/bots.d/custom-bad-referrers.conf [TO]=> /etc/nginx/bots.d/custom-bad-referrers.conf
Downloading [FROM]=> [REPO]/bots.d/bad-referrer-words.conf [TO]=> /etc/nginx/bots.d/bad-referrer-words.conf
Downloading [FROM]=> [REPO]/bots.d/blacklist-ips.conf [TO]=> /etc/nginx/bots.d/blacklist-ips.conf
Downloading [FROM]=> [REPO]/bots.d/blacklist-user-agents.conf [TO]=> /etc/nginx/bots.d/blacklist-user-agents.conf
Downloading [FROM]=> [REPO]/bots.d/whitelist-domains.conf [TO]=> /etc/nginx/bots.d/whitelist-domains.conf
Downloading [FROM]=> [REPO]/bots.d/whitelist-ips.conf [TO]=> /etc/nginx/bots.d/whitelist-ips.conf
REPO = https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master
Downloading [FROM]=> [REPO]/setup-ngxblocker [TO]=> /usr/local/sbin/setup-ngxblocker
Downloading [FROM]=> [REPO]/update-ngxblocker [TO]=> /usr/local/sbin/update-ngxblocker
The above command lists the packages that will be downloaded, now use the command with the -x parameter to write these changes.
sudo ./install-ngxblocker -xThe output will be as shown:
Checking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt
Creating directory: /etc/nginx/conf.d
Creating directory: /etc/nginx/bots.d
REPO = https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master
Downloading [FROM]=> [REPO]/conf.d/globalblacklist.conf [TO]=> /etc/nginx/conf.d/globalblacklist.conf...OK
Downloading [FROM]=> [REPO]/conf.d/botblocker-nginx-settings.conf [TO]=> /etc/nginx/conf.d/botblocker-nginx-settings.conf...OK
REPO = https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master
Downloading [FROM]=> [REPO]/bots.d/blockbots.conf [TO]=> /etc/nginx/bots.d/blockbots.conf...OK
Downloading [FROM]=> [REPO]/bots.d/ddos.conf [TO]=> /etc/nginx/bots.d/ddos.conf...OK
Downloading [FROM]=> [REPO]/bots.d/custom-bad-referrers.conf [TO]=> /etc/nginx/bots.d/custom-bad-referrers.conf...OK
Downloading [FROM]=> [REPO]/bots.d/bad-referrer-words.conf [TO]=> /etc/nginx/bots.d/bad-referrer-words.conf...OK
Downloading [FROM]=> [REPO]/bots.d/blacklist-ips.conf [TO]=> /etc/nginx/bots.d/blacklist-ips.conf...OK
Downloading [FROM]=> [REPO]/bots.d/blacklist-user-agents.conf [TO]=> /etc/nginx/bots.d/blacklist-user-agents.conf...OK
Downloading [FROM]=> [REPO]/bots.d/whitelist-domains.conf [TO]=> /etc/nginx/bots.d/whitelist-domains.conf...OK
Downloading [FROM]=> [REPO]/bots.d/whitelist-ips.conf [TO]=> /etc/nginx/bots.d/whitelist-ips.conf...OK
REPO = https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master
Downloading [FROM]=> [REPO]/setup-ngxblocker [TO]=> /usr/local/sbin/setup-ngxblocker...OK
Downloading [FROM]=> [REPO]/update-ngxblocker [TO]=> /usr/local/sbin/update-ngxblocker...OK
Setting mode: 700 => /usr/local/sbin/install-ngxblocker
Setting mode: 700 => /usr/local/sbin/setup-ngxblocker
Setting mode: 700 => /usr/local/sbin/update-ngxblocker
Once the required files have been downloaded in the correct folders, proceed and make the setup and update scripts executable.
sudo chmod +x /usr/local/sbin/setup-ngxblocker
sudo chmod +x /usr/local/sbin/update-ngxblockerStep 3 – Install the Nginx Bad Bot Blocker
To install the Nginx Bad Bot Blocker, we will run the setup script. This script can be run in DRY-MODE to show the changes it will make and the files it will download as below.
sudo /usr/local/sbin/setup-ngxblockerThe Output:
/etc/nginx/sites-available/ssl.no-default.conf
/etc/nginx/sites-available/no-default.conf
/etc/nginx/sites-available/wordpress.example.com.conf
Configure every file above as a vhost ? [Y/N] : Y
Checking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt
** Dry Run ** | not updating files | run as 'setup-ngxblocker -x' to setup files.
inserting: include /etc/nginx/conf.d/globalblacklist.conf; => /etc/nginx/nginx.conf
inserting: include /etc/nginx/conf.d/botblocker-nginx-settings.conf; => /etc/nginx/nginx.conf
inserting: include /etc/nginx/bots.d/blockbots.conf; => /etc/nginx/sites-available/wordpress.example.com.conf
inserting: include /etc/nginx/bots.d/ddos.conf; => /etc/nginx/sites-available/wordpress.example.com.conf
Whitelisting ip: 88.99.92.81 => /etc/nginx/bots.d/whitelist-ips.conf
Web directory not found ('/var/www'): not automatically whitelisting domains.
Checking for missing includes:
Checking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt
Nothing to update for directory: /etc/nginx/conf.d
Nothing to update for directory: /etc/nginx/bots.d
Nothing to update for directory: /usr/local/sbin
Setting mode: 700 => /usr/local/sbin/install-ngxblocker
Setting mode: 700 => /usr/local/sbin/setup-ngxblocker
Setting mode: 700 => /usr/local/sbin/update-ngxblocker
To make the changes to the nginx.conf, you need to run the script with the -x parameter.
sudo ./setup-ngxblocker -xSample Output:
/etc/nginx/sites-available/ssl.no-default.conf
/etc/nginx/sites-available/no-default.conf
/etc/nginx/sites-available/wordpress.example.com.conf
Configure every file above as a vhost ? [Y/N] : y
Checking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt
inserting: include /etc/nginx/conf.d/globalblacklist.conf; => /etc/nginx/nginx.conf
inserting: include /etc/nginx/conf.d/botblocker-nginx-settings.conf; => /etc/nginx/nginx.conf
inserting: include /etc/nginx/bots.d/blockbots.conf; => /etc/nginx/sites-available/wordpress.example.com.conf
inserting: include /etc/nginx/bots.d/ddos.conf; => /etc/nginx/sites-available/wordpress.example.com.conf
Whitelisting ip: 88.99.92.81 => /etc/nginx/bots.d/whitelist-ips.conf
Web directory not found ('/var/www'): not automatically whitelisting domains.
Checking for missing includes:
Checking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt
Nothing to update for directory: /etc/nginx/conf.d
Nothing to update for directory: /etc/nginx/bots.d
Nothing to update for directory: /usr/local/sbin
Setting mode: 700 => /usr/local/sbin/install-ngxblocker
Setting mode: 700 => /usr/local/sbin/setup-ngxblocker
Setting mode: 700 => /usr/local/sbin/update-ngxblocker
The above command includes all the Nginx Virtual Host files on the server and whitelists your Ip address in the whitelist-ips.conf. You can make desired changes by editing the file /etc/nginx/bots.d/whitelist-ips.conf.
Basically, the script adds the “include” statements below into your Nginx virtual host files:
# Bad Bot Blocker
include /etc/nginx/bots.d/ddos.conf;
include /etc/nginx/bots.d/blockbots.conf;Alternatively, you can install the Blocker to a non-standard Nginx folder using by specifying the folders:
sudo ./install-ngxblocker -x -c /usr/local/nginx/conf.d -b /usr/local/nginx/bots.d
sudo ./setup-ngxblocker -x -c /usr/local/nginx/conf.d -b /usr/local/nginx/bots.d
sudo ./update-ngxblocker -c /usr/local/nginx/conf.d -b /usr/local/nginx/bots.d -e [email protected]Once complete, check the syntax of Nginx configurations.
$ sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successfulRestart Nginx for the changes to apply.
sudo systemctl restart nginxStep 4 – Configure Auto Update for Nginx Bad Bot Blocker
We will configure a Cron job so that automatically updates the blocker to guarantee you the latest version.
Open the file for editing:
sudo crontab -eIn the file, add the below lines replacing your email address appropriately.
00 22 * * * sudo /usr/local/sbin/update-ngxblocker -e [email protected]This command will auto-update the blocker every day at 2200hrs. You can also push out to 3-4 times a day by setting the Cron task to run after every 8 hours as below:
00 */8 * * * sudo /usr/local/sbin/update-ngxblocker -e [email protected]You can also disable email updates using the command:
00 */8 * * * sudo /usr/local/sbin/update-ngxblocker -nStep 5 – Customize Nginx Bad Bot Blocker
The Nginx Bad Bot Blocker can be customized to suit your environment by editing the “include” files. The good thing is that these files aren’t overwritten by an update.
The available files for editing are:
/etc/nginx/bots.d/whitelist-ips.conf
/etc/nginx/bots.d/whitelist-domains.conf
/etc/nginx/bots.d/blockbots.conf
/etc/nginx/bots.d/blacklist-domains.conf
/etc/nginx/bots.d/blacklist-user-agents.conf
/etc/nginx/bots.d/blacklist-ips.conf
/etc/nginx/bots.d/bad-referrer-words.conf
/etc/nginx/bots.d/custom-bad-referrers.conf
/etc/nginx/bots.d/ddos.confFor example, if you want to block GoogleBot from accessing your site, edit the file; /etc/nginx/bots.d/blacklist-user-agents.conf which overrides the default whitelist for GoogleBot. This can be done to any other whitelisted Bot.
Step 6 – Testing the Nginx Bad Bot Blocker
You can test your Nginx Bad Bot Blocker from the terminal on another system using your domain name as below:
curl -A "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" -I http://yourdomain.comYou can also use:
curl -A "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" -I http://yourdomain.comSample Output:
HTTP/1.1 200 OK
Server: nginx/1.14.1
Date: Wed, 27 Apr 2022 08:57:12 GMT
Content-Type: text/html
Content-Length: 3429
Last-Modified: Thu, 10 Jun 2021 09:09:03 GMT
Connection: keep-alive
ETag: "60c1d6af-d65"
Accept-Ranges: bytesThe below test should return the error below:
- curl: (52) Empty reply from server
- curl: (56) TCP connection reset by peer
- curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)
Use the commands below to test:
curl -A "Xenu Link Sleuth/1.3.8" -I http://yourdomain.com
curl -I http://yourdomain.com -e http://100dollars-seo.com
curl -I http://yourdomain.com -e http://zx6.ruSample Output:
That marks the end of this guide on how to Block Bad Bots, Spam, User-Agents, and Ransomware on Nginx. You can make the required adjustments to the “include” files to allow/block bots.
Interested in more?
- 6 Ways to Keep Hackers Out of Your Website
- 5 Most Unusual Telegram Bots That Ever Existed
- 4 Ways to Protect Your Website from Hackers
- Are Startups More Vulnerable to Cyberattacks?
