While working with a big Dataframe, Dataframe consists of any number of columns that are having different datatypes. For pre-processing the data to apply operations on it, we have to know the dimensions of the Dataframe and datatypes of the columns which are present in the Dataframe.
In this article, we are going to know how to verify the column type of the Dataframe. For verifying the column type we are using dtypes function. The dtypes function is used to return the list of tuples that contain the Name of the column and column type.
Syntax: df.dtypes()
where, df is the Dataframe
At first, we will create a dataframe and then see some examples and implementation.
Python
# importing necessary librariesfrom pyspark.sql import SparkSession# function to create new SparkSessiondef create_session(): spk = SparkSession.builder \ .master("local") \ .appName("Product_details.com") \ .getOrCreate() return spkdef create_df(spark,data,schema): df1 = spark.createDataFrame(data,schema) return df1if __name__ == "__main__": # calling function to create SparkSession spark = create_session() input_data = [("Mobile",112345,4.0,12499), ("LED TV",114567,4.2,49999), ("Refrigerator",123543,4.4,13899), ("Washing Machine",113465,3.9,6999), ("T-shirt",124378,4.1,1999), ("Jeans",126754,3.7,3999), ("Running Shoes",134565,4.7,1499), ("Face Mask",145234,4.6,999)] schema = ["Name","ID","Rating","Price"] # calling function to create dataframe df = create_df(spark,input_data,schema) # visualizing the dataframe df.show() |
Output:
Example 1: Verify the column type of the Dataframe using dtypes() function
In the below example code, we have created the Dataframe then for getting the column types of all the columns present in the Dataframe we have used dtypes function by writing df.dtypes using with f string while finding the datatypes of all the columns we have printed also. This gives a list of tuples that contains the name and datatype of the columns.
Python
# finding data type of the all the# column using dtype function and# printingprint(f'Data types of all the columns is : {df.dtypes}')# visualizing the dataframedf.show() |
Output:
Example 2: Verify the specific column datatype of the Dataframe
In the below code after creating the Dataframe we are finding the Datatype of the particular column using dtypes() function by writing dict(df.dtypes)[‘Rating’], here we are using dict because as we see in the above example df.dtypes return the list of tuples that contains the name and datatype of the column. So using dict we are typecasting tuple into the dictionary.
As we know in the dictionary the data is stored in key and value pair, while writing dict(df.dtypes)[‘Rating’] we are giving the key i.e, ‘Rating’ and extracting its value of that is double, which is the datatype of the column. So in this way, we can find out the datatype of column type while passing the specific name of the column.
Python
# finding data type of the Rating# column using dtype functiondata_type = dict(df.dtypes)['Rating']# printingprint(f'Data type of Rating is : {data_type}')# visualizing the dataframedf.show() |
Output:
Example 3: Verify the column type of the Dataframe using for loop
After creating the Dataframe, for finding the datatypes of the column with column name we are using df.dtypes which gives us the list of tuples.
While iterating we are getting the column name and column type as a tuple then printing the name of the column and column type using print(col[0],”,”,col[1]). In this way, we are getting every column name and column type using by iterating.
Python
print("Datatype of the columns with column names are:")# finding datatype of all column with# column name using for loopfor col in df.dtypes: # printing the column and datatype # of that column print(col[0],",",col[1])# visualizing the dataframedf.show() |
Output:
Example 4: Verify the column type of the Dataframe using schema
After creating the Dataframe for verifying the column type we are using printSchema() function by writing df.printSchema() through this function schema of the Dataframe is printed which contains the datatype of each and every column present in Dataframe. So, using printSchema() function also we can easily verify the column type of the PySpark Dataframe.
Python
# printing the schema of the Dataframe# using printschema functiondf.printSchema()# visualizing the dataframedf.show() |
Output:
