Nightmare is provided by Segment and is a high-level browser automation library. The goal for this Nightmare is to follow the path of the user actions (like goto, type, click, etc.), in order to expose few simple methods with an API that feels synchronous for each block of scripting, rather than deeply nested callbacks. It has been designed for automating tasks across sites that don’t have APIs, but nowadays is most often used for crawling, UI testing, etc.
Note: Nightmare is intended to be run on NodeJS 4.x or higher.
Alternatives to Nightmare.js: There are many alternatives for this, Some of the most common alternatives of Nightmate.js are:
- Puppeteer
- PhantomJS
- fancybox
- MomentJS
- Lodash
- Underscore
Project Setup and installing required modules:
-
Step 1: Install Nightmare.js and Jquery module using the following command:
npm install nightmare npm install jquery
-
Step 2: Create an index.js file and require the module using the following way:
const Nightmare = require('nightmare'); nightmare = Nightmare({ show: true});
This will create an instance of Nightmare which can navigate to the web. It is possible to put optional parameters in the JSON passed inside the Nightmare() as shown above.
Project Structure: It will look like the following.
Example: Let us see an example to use Nightmare.js for CraigLists scrapping:
Filename: index.js
index.js
var jquery = require('jquery') var Nightmare = require('nightmare'), nightmare = Nightmare() var city = process.argv[2] // Use the first argument passed as the city to be searched '.craigslist.org/search/cpg?is_paid=yes&postedToday=1') // Visits the city specified by the user and gets all computer // gigs posted that day .wait(2000) // Wait 2 seconds so page is guaranteed to be fully loaded .evaluate(function () { var gigs = []; // Create an array to hold all gigs gathered by following code $('.hdrlnk').each(function () { item = {} item["title"] = $(this).text() item["link"] = $(this).attr("href") gigs.push(item) }) // Create a gig object with title and link, then push // to the 'gigs' array return gigs // Pass the gigs array forward so it can be looped through later on }) .end() .then(function (result) { for (gig in result) { console.log(result[gig].title); } // Print each gig to the console in a neat format }) |
Run the index.js file using the following command:
node index.js cityname
For example, we can search for saltlakecity as shown below:
node index.js saltlakecity
Output: Wait for few seconds, and you will get the output.
