Thursday, December 26, 2024
Google search engine
HomeSecurity & TestingBlock Bad Bots, Spam, User-Agents, Ransomware on Nginx
Security & Testing

Block Bad Bots, Spam, User-Agents, Ransomware on Nginx

Ted Musemwa
By Ted Musemwa
0
0
.tdi_3.td-a-rec{text-align:center}.tdi_3 .td-element-style{z-index:-1}.tdi_3.td-a-rec-img{text-align:left}.tdi_3.td-a-rec-img img{margin:0 auto 0 0}@media(max-width:767px){.tdi_3.td-a-rec-img{text-align:center}}

In recent days, many organizations have fallen victims to cybercrime. More sophisticated botnets and other attack methods have been developed to ratchet the speed of these attacks. The most common attacks are DDoS(distributed denial of service, account takeover(ATO), and scraping content from websites. These attacks have dire consequences for the target organizations and need to be avoided at all costs.

Normally, Bad Bots can be defined as software applications that execute automated tasks with malicious intentions over the internet. They masquerade in the system by disguising the user agents. These bad bots may be;

  • Bots or Servers linked to viruses or malware
  • Government surveillance bots
  • Botnet Attack Networks (Mirai)
  • Gambling and Porn WebSites
  • Vulnerability scanners
  • Spam Referrers
  • Bad User-Agent Strings
  • Browser Adware and Malware (Yontoo etc)
  • Link Research and Backlink Testing Tools
  • SEO companies that your competitors use to try to improve their SEO
  • Image Hotlinking Sites and Image Thieves
  • Link Ranking Bots
  • E-mail harvesters
  • Sites Linked to Lucrative Malware, Adware, and Ransomware Clickjacking Campaigns
  • Stopping Google Analytics Ghost Spam

This guide demonstrates how you can block Bad Bots, Spam, User-Agents, and Ransomware on Nginx.

.tdi_2.td-a-rec{text-align:center}.tdi_2 .td-element-style{z-index:-1}.tdi_2.td-a-rec-img{text-align:left}.tdi_2.td-a-rec-img img{margin:0 auto 0 0}@media(max-width:767px){.tdi_2.td-a-rec-img{text-align:center}}

Step 1 – Download the Nginx Bad Bot Blocker

The Nginx Bad Bot Blocker was developed by Mitchell Krog to be used in the Nginx web server. It can be downloaded on your Linux system using the command:

### With wget
wget https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/install-ngxblocker

### With curl
sudo curl -sL https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/install-ngxblocker -o install-ngxblocker

Once the file has been downloaded, make it executable.

chmod +x install-ngxblocker

On FreeBSD systems, the package is available in the default repository and can be installed with the command:

pkg install www/nginx-ultimate-bad-bot-blocker

Step 2 – Download the Required files.

On other Linux systems, use the downloaded file in the /usr/local/sbin directory as below:

sudo ./install-ngxblocker

The installation will proceed as below:

Checking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt

** Dry Run ** | not updating files | run  as 'install-ngxblocker -x' to install files.

Creating directory: /etc/nginx/conf.d
Creating directory: /etc/nginx/bots.d

REPO = https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master

Downloading [FROM]=>  [REPO]/conf.d/globalblacklist.conf            [TO]=>  /etc/nginx/conf.d/globalblacklist.conf
Downloading [FROM]=>  [REPO]/conf.d/botblocker-nginx-settings.conf  [TO]=>  /etc/nginx/conf.d/botblocker-nginx-settings.conf

REPO = https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master

Downloading [FROM]=>  [REPO]/bots.d/blockbots.conf              [TO]=>  /etc/nginx/bots.d/blockbots.conf
Downloading [FROM]=>  [REPO]/bots.d/ddos.conf                   [TO]=>  /etc/nginx/bots.d/ddos.conf
Downloading [FROM]=>  [REPO]/bots.d/custom-bad-referrers.conf   [TO]=>  /etc/nginx/bots.d/custom-bad-referrers.conf
Downloading [FROM]=>  [REPO]/bots.d/bad-referrer-words.conf     [TO]=>  /etc/nginx/bots.d/bad-referrer-words.conf
Downloading [FROM]=>  [REPO]/bots.d/blacklist-ips.conf          [TO]=>  /etc/nginx/bots.d/blacklist-ips.conf
Downloading [FROM]=>  [REPO]/bots.d/blacklist-user-agents.conf  [TO]=>  /etc/nginx/bots.d/blacklist-user-agents.conf
Downloading [FROM]=>  [REPO]/bots.d/whitelist-domains.conf      [TO]=>  /etc/nginx/bots.d/whitelist-domains.conf
Downloading [FROM]=>  [REPO]/bots.d/whitelist-ips.conf          [TO]=>  /etc/nginx/bots.d/whitelist-ips.conf

REPO = https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master

Downloading [FROM]=>  [REPO]/setup-ngxblocker      [TO]=>  /usr/local/sbin/setup-ngxblocker
Downloading [FROM]=>  [REPO]/update-ngxblocker     [TO]=>  /usr/local/sbin/update-ngxblocker

The above command lists the packages that will be downloaded, now use the command with the -x parameter to write these changes.

sudo ./install-ngxblocker -x

The output will be as shown:

Checking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt

Creating directory: /etc/nginx/conf.d
Creating directory: /etc/nginx/bots.d

REPO = https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master

Downloading [FROM]=>  [REPO]/conf.d/globalblacklist.conf            [TO]=>  /etc/nginx/conf.d/globalblacklist.conf...OK
Downloading [FROM]=>  [REPO]/conf.d/botblocker-nginx-settings.conf  [TO]=>  /etc/nginx/conf.d/botblocker-nginx-settings.conf...OK

REPO = https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master

Downloading [FROM]=>  [REPO]/bots.d/blockbots.conf              [TO]=>  /etc/nginx/bots.d/blockbots.conf...OK
Downloading [FROM]=>  [REPO]/bots.d/ddos.conf                   [TO]=>  /etc/nginx/bots.d/ddos.conf...OK
Downloading [FROM]=>  [REPO]/bots.d/custom-bad-referrers.conf   [TO]=>  /etc/nginx/bots.d/custom-bad-referrers.conf...OK
Downloading [FROM]=>  [REPO]/bots.d/bad-referrer-words.conf     [TO]=>  /etc/nginx/bots.d/bad-referrer-words.conf...OK
Downloading [FROM]=>  [REPO]/bots.d/blacklist-ips.conf          [TO]=>  /etc/nginx/bots.d/blacklist-ips.conf...OK
Downloading [FROM]=>  [REPO]/bots.d/blacklist-user-agents.conf  [TO]=>  /etc/nginx/bots.d/blacklist-user-agents.conf...OK
Downloading [FROM]=>  [REPO]/bots.d/whitelist-domains.conf      [TO]=>  /etc/nginx/bots.d/whitelist-domains.conf...OK
Downloading [FROM]=>  [REPO]/bots.d/whitelist-ips.conf          [TO]=>  /etc/nginx/bots.d/whitelist-ips.conf...OK

REPO = https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master

Downloading [FROM]=>  [REPO]/setup-ngxblocker      [TO]=>  /usr/local/sbin/setup-ngxblocker...OK
Downloading [FROM]=>  [REPO]/update-ngxblocker     [TO]=>  /usr/local/sbin/update-ngxblocker...OK
Setting mode: 700 => /usr/local/sbin/install-ngxblocker
Setting mode: 700 => /usr/local/sbin/setup-ngxblocker
Setting mode: 700 => /usr/local/sbin/update-ngxblocker

Once the required files have been downloaded in the correct folders, proceed and make the setup and update scripts executable.

sudo chmod +x /usr/local/sbin/setup-ngxblocker
sudo chmod +x /usr/local/sbin/update-ngxblocker

Step 3 – Install the Nginx Bad Bot Blocker

To install the Nginx Bad Bot Blocker, we will run the setup script. This script can be run in DRY-MODE to show the changes it will make and the files it will download as below.

sudo /usr/local/sbin/setup-ngxblocker

The Output:

/etc/nginx/sites-available/ssl.no-default.conf
/etc/nginx/sites-available/no-default.conf
/etc/nginx/sites-available/wordpress.example.com.conf

Configure every file above as a vhost ? [Y/N] : Y
Checking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt

** Dry Run ** | not updating files | run  as 'setup-ngxblocker -x' to setup files.

inserting: include /etc/nginx/conf.d/globalblacklist.conf;            => /etc/nginx/nginx.conf
inserting: include /etc/nginx/conf.d/botblocker-nginx-settings.conf;  => /etc/nginx/nginx.conf
inserting: include /etc/nginx/bots.d/blockbots.conf;                  => /etc/nginx/sites-available/wordpress.example.com.conf
inserting: include /etc/nginx/bots.d/ddos.conf;                       => /etc/nginx/sites-available/wordpress.example.com.conf

Whitelisting ip:  88.99.92.81     => /etc/nginx/bots.d/whitelist-ips.conf

Web directory not found ('/var/www'): not automatically whitelisting domains.

Checking for missing includes:

Checking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt

Nothing to update for directory: /etc/nginx/conf.d
Nothing to update for directory: /etc/nginx/bots.d
Nothing to update for directory: /usr/local/sbin
Setting mode: 700 => /usr/local/sbin/install-ngxblocker
Setting mode: 700 => /usr/local/sbin/setup-ngxblocker
Setting mode: 700 => /usr/local/sbin/update-ngxblocker

To make the changes to the nginx.conf, you need to run the script with the -x parameter.

sudo ./setup-ngxblocker -x

Sample Output:

/etc/nginx/sites-available/ssl.no-default.conf
/etc/nginx/sites-available/no-default.conf
/etc/nginx/sites-available/wordpress.example.com.conf

Configure every file above as a vhost ? [Y/N] : y
Checking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt

inserting: include /etc/nginx/conf.d/globalblacklist.conf;            => /etc/nginx/nginx.conf
inserting: include /etc/nginx/conf.d/botblocker-nginx-settings.conf;  => /etc/nginx/nginx.conf
inserting: include /etc/nginx/bots.d/blockbots.conf;                  => /etc/nginx/sites-available/wordpress.example.com.conf
inserting: include /etc/nginx/bots.d/ddos.conf;                       => /etc/nginx/sites-available/wordpress.example.com.conf

Whitelisting ip:  88.99.92.81     => /etc/nginx/bots.d/whitelist-ips.conf

Web directory not found ('/var/www'): not automatically whitelisting domains.

Checking for missing includes:

Checking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt

Nothing to update for directory: /etc/nginx/conf.d
Nothing to update for directory: /etc/nginx/bots.d
Nothing to update for directory: /usr/local/sbin
Setting mode: 700 => /usr/local/sbin/install-ngxblocker
Setting mode: 700 => /usr/local/sbin/setup-ngxblocker
Setting mode: 700 => /usr/local/sbin/update-ngxblocker

The above command includes all the Nginx Virtual Host files on the server and whitelists your Ip address in the whitelist-ips.conf. You can make desired changes by editing the file /etc/nginx/bots.d/whitelist-ips.conf.

Basically, the script adds the “include” statements below into your Nginx virtual host files:

# Bad Bot Blocker
include /etc/nginx/bots.d/ddos.conf;
include /etc/nginx/bots.d/blockbots.conf;

Alternatively, you can install the Blocker to a non-standard Nginx folder using by specifying the folders:

sudo ./install-ngxblocker -x -c /usr/local/nginx/conf.d -b /usr/local/nginx/bots.d
sudo ./setup-ngxblocker -x -c /usr/local/nginx/conf.d -b /usr/local/nginx/bots.d
sudo ./update-ngxblocker -c /usr/local/nginx/conf.d -b /usr/local/nginx/bots.d -e [email protected]

Once complete, check the syntax of Nginx configurations.

$ sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Restart Nginx for the changes to apply.

sudo systemctl restart nginx

Step 4 – Configure Auto Update for Nginx Bad Bot Blocker

We will configure a Cron job so that automatically updates the blocker to guarantee you the latest version.

Open the file for editing:

sudo crontab -e

In the file, add the below lines replacing your email address appropriately.

00 22 * * * sudo /usr/local/sbin/update-ngxblocker -e [email protected]

This command will auto-update the blocker every day at 2200hrs. You can also push out to 3-4 times a day by setting the Cron task to run after every 8 hours as below:

00 */8 * * * sudo /usr/local/sbin/update-ngxblocker -e [email protected]

You can also disable email updates using the command:

00 */8 * * * sudo /usr/local/sbin/update-ngxblocker -n

Step 5 – Customize Nginx Bad Bot Blocker

The Nginx Bad Bot Blocker can be customized to suit your environment by editing the “include” files. The good thing is that these files aren’t overwritten by an update.

The available files for editing are:

/etc/nginx/bots.d/whitelist-ips.conf
/etc/nginx/bots.d/whitelist-domains.conf
/etc/nginx/bots.d/blockbots.conf
/etc/nginx/bots.d/blacklist-domains.conf
/etc/nginx/bots.d/blacklist-user-agents.conf
/etc/nginx/bots.d/blacklist-ips.conf
/etc/nginx/bots.d/bad-referrer-words.conf
/etc/nginx/bots.d/custom-bad-referrers.conf
/etc/nginx/bots.d/ddos.conf

For example, if you want to block GoogleBot from accessing your site, edit the file; /etc/nginx/bots.d/blacklist-user-agents.conf which overrides the default whitelist for GoogleBot. This can be done to any other whitelisted Bot.

Step 6 – Testing the Nginx Bad Bot Blocker

You can test your Nginx Bad Bot Blocker from the terminal on another system using your domain name as below:

curl -A "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" -I http://yourdomain.com

You can also use:

curl -A "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" -I http://yourdomain.com

Sample Output:

HTTP/1.1 200 OK
Server: nginx/1.14.1
Date: Wed, 27 Apr 2022 08:57:12 GMT
Content-Type: text/html
Content-Length: 3429
Last-Modified: Thu, 10 Jun 2021 09:09:03 GMT
Connection: keep-alive
ETag: "60c1d6af-d65"
Accept-Ranges: bytes

The below test should return the error below:

  • curl: (52) Empty reply from server
  • curl: (56) TCP connection reset by peer
  • curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)

Use the commands below to test:

curl -A "Xenu Link Sleuth/1.3.8" -I http://yourdomain.com
curl -I http://yourdomain.com -e http://100dollars-seo.com
curl -I http://yourdomain.com -e http://zx6.ru

Sample Output:

How To Block Bad Bots Spam User Agents and Ransomware on

That marks the end of this guide on how to Block Bad Bots, Spam, User-Agents, and Ransomware on Nginx. You can make the required adjustments to the “include” files to allow/block bots.

Interested in more?

.tdi_4.td-a-rec{text-align:center}.tdi_4 .td-element-style{z-index:-1}.tdi_4.td-a-rec-img{text-align:left}.tdi_4.td-a-rec-img img{margin:0 auto 0 0}@media(max-width:767px){.tdi_4.td-a-rec-img{text-align:center}}

Previous article
Secure Plex and Kodi Server using Let’s Encrypt SSL
Next article
Run Keycloak Server in Docker Containers with Let’s Encrypt SSL
Ted Musemwa
Ted Musemwa
RELATED ARTICLES

Most Popular

Load more

Recent Comments

강서구출장마사지 on How to store XML data into a MySQL database using Python?
금천구출장마사지 on How to store XML data into a MySQL database using Python?
nightwish.southeast.cz on Google says it won’t keep your Pixel during a repair if you’re caught using non-OEM parts
광명출장안마 on How to store XML data into a MySQL database using Python?
광명출장안마 on How to store XML data into a MySQL database using Python?
출장오피 on How to store XML data into a MySQL database using Python?
부천출장안마 on How to store XML data into a MySQL database using Python?
구월동출장안마 on How to store XML data into a MySQL database using Python?
강서구출장안마 on How to store XML data into a MySQL database using Python?
헬로출장 on How to store XML data into a MySQL database using Python?
오산출장안마 on How to store XML data into a MySQL database using Python?
광명출장마사지 on How to store XML data into a MySQL database using Python?
마포출장 on How to store XML data into a MySQL database using Python?
안양출장마사지 on How to store XML data into a MySQL database using Python?
gKTdhA on 5 reasons why I won’t switch away from Google Photos
부천출장안마 on How to store XML data into a MySQL database using Python?
동탄출장안마 on How to store XML data into a MySQL database using Python?
0a1Mq7 on Wander: An add-on for Apple’s Shortcuts app to install the Odyssey jailbreak
서울출장안마 on How to store XML data into a MySQL database using Python?
분당출장안마 on How to store XML data into a MySQL database using Python?
부천출장안마 on How to store XML data into a MySQL database using Python?
출장 오피 on How to store XML data into a MySQL database using Python?
화곡동출장마사지 on How to store XML data into a MySQL database using Python?
Gilda on Wander: An add-on for Apple’s Shortcuts app to install the Odyssey jailbreak
강서구출장마사지 on How to store XML data into a MySQL database using Python?
고양출장안마 on How to store XML data into a MySQL database using Python?
화성출장마사지 on How to store XML data into a MySQL database using Python?
천호동출장마사지 on How to store XML data into a MySQL database using Python?
June P. D. Alvarez on RedSn0w Updated to Fix iBooks DRM Issues
Litha on How to Install Siri on iPad 2

EDITOR PICKS

POPULAR POSTS

POPULAR CATEGORY

ABOUT US

Newspaper is your news, entertainment, music fashion website. We provide you with the latest breaking news and videos straight from the entertainment industry.

Contact us: hello@geeksforgeeks.org

FOLLOW US

© Lazyr0ar 2022