Nowadays Node.js is the most attractive technology in the field of backend development for developers around the globe. And if someone wishes to use something like Web Scraping using python modules or run some python scripts having some machine learning algorithms, then one need to know how to integrate these two.
We are going to get some data of a user through web scraping of leetcode . So, let’s get started.
Now, setup the Node.js server code first.
Javascript
//Import express.js module and create its variable.const express=require('express');const app=express();//Import PythonShell module.const {PythonShell} =require('python-shell');//Router to handle the incoming request.app.get("/", (req, res, next)=>{ //Here are the option object in which arguments can be passed for the python_test.js. let options = { mode: 'text', pythonOptions: ['-u'], // get print results in real-time scriptPath: 'path/to/my/scripts', //If you are having python_test.py script in same folder, then it's optional. args: ['shubhamk314'] //An argument which can be accessed in the script using sys.argv[1] }; PythonShell.run('python_test.py', options, function (err, result){ if (err) throw err; // result is an array consisting of messages collected //during execution of script. console.log('result: ', result.toString()); res.send(result.toString()) });});//Creates the server on default port 8000 and can be accessed through localhost:8000const port=8000;app.listen(port, ()=>console.log(`Server connected to ${port}`)); |
Now, Python Script. (Make sure you have installed the bs4 module and csv module.)
Python3
# codeimport sysimport requestsfrom bs4 import BeautifulSoupfrom csv import writer# bs4 module for web scraping and requests for making HTTPS requests using Python.soup = BeautifulSoup(response.text, 'html.parser')main_content = soup.select( '# base_content>div>div>div.col-sm-5.col-md-4>div:nth-child(3)>ul')list_items = main_content[0].select('li')items = ['Solved Question', 'Accepted Submission', 'Acceptance Rate']n = 0# It will create csv files named progress.csv in root folder once this is script is called.with open('progress.csv', 'w') as csv_file: csv_writer = writer(csv_file) headers = ['Name', 'Score'] csv_writer.writerow(headers) while(n < 3): name = items[n] score = list_items[n].find('span').get_text().strip() csv_writer.writerow([name, score]) n = n + 1print("csv file created for leetcode") |
After saving both the files, run the following command from its root folder :
node test.js
Now, send request through localhost:8000 in the browser.
If everything goes right, then the output will be :
Message: csv file created for leetcode.
Conclusion
This is the simple implementation of a how-to run python script with Node.js which can be useful in situations where you have a stack of Node.js application and you want to run a simple python script. If you want to know more about PythonShell module, then go through the given link.
