Prerequisite: Hadoop Installation, HDFS
Python Snakebite is a very popular Python library that we can use to communicate with the HDFS. Using the Python client library provided by the Snakebite package we can easily write Python code that works on HDFS. It uses protobuf messages to communicate directly with the NameNode. The python client library directly works with HDFS without making a system call to hdfs dfs. The Snakebite doesn’t support python3.
The hdfs dfs provides multiple commands through which we can perform multiple operations on HDFS. The client library that Snakebite provides will contain various methods that allow us to retrieve data from HDFS. text() method is used to simply read the data from a file available on our HDFS. So let’s perform a quick task to understand how we can retrieve data from a file from HDFS.
Task: Retrieving File Data From HDFS.
Step 1: Create a text file with the name data.txt and add some data to it.
cd Documents/ # Changing directory to Documents(You can choose as per your requirement) touch data.txt # touch command is used to create file in linux environment nano data.txt # nano is a command line text editor for Unix and Linux operating system cat data.txt # to see the content of a file
Step 2: Send this data.txt file to Hadoop HDFS with the help of copyFromLocal Command.
Syntax:
hdfs dfs -copyFromLocal /path 1 /path 2 .... /path n /destination
Using the command to sending data.txt to the root directory of HDFS.
hdfs dfs -copyFromLocal /home/dikshant/Documents/data.txt /
Now, Check whether the file is reached to the root directory of HDFS or not with the help of the below command.
hdfs dfs -ls /
You can check it manually by visiting http://localhost:50070/ then Utilities -> Browse the file system.
Step 3: Now our task is to read the data from data.txt we send to our HDFS. So create a file data_read.py in your local file system and add the below python code to it.
Python
# importing the libraryfrom snakebite.client import Client# the below line create client connection to the HDFS NameNodeclient = Client('localhost', 9000)# iterate over data.txt file and will show all the content of data.txtfor l in client.text(['/data.txt']): print l |
Client() method explanation:
The Client() method can accept all the below listed arguments:
- host(string): IP Address of NameNode.
- port(int): RPC port of Namenode.
- hadoop_version (int): Hadoop protocol version(by default it is: 9)
- use_trash (boolean): Use trash when removing the files.
- effective_use (string): Effective user for the HDFS operations (default user is current user).
Step 4: Run the read_data.py file and observe the result.
python read_data.py
We have successfully fetched the data from data.txt with the help of client library.
We can also copy any file from HDFS to our Local file system with the help of Snakebite. To copy a file from HDFS create a file fetch_file.py and copy the below python code to it. copyToLocal() method is used to achieve this.
Python
from snakebite.client import Clientclient = Client('localhost', 9000)for a in client.copyToLocal(['/data.txt'], '/home/dikshant/Desktop'): print a |
Now, run this python file you will see the below output.
python fetch_file.py
We can observe that the file now has been copied to my /home/dikshant/desktop directory.
