In this article, we are going to extract a single value from the pyspark dataframe columns. To do this we will use the first() and head() functions.
Single value means only one value, we can extract this value based on the column name
Syntax:
- dataframe.first()[‘column name’]
- Dataframe.head()[‘Index’]
Where,
- dataframe is the input dataframe and column name is the specific column
- Index is the row and columns.
So we are going to create the dataframe using the nested list.
Python3
# importing moduleimport pyspark # importing sparksession from pyspark.sql modulefrom pyspark.sql import SparkSession # creating sparksession and giving an app namespark = SparkSession.builder.appName('sparkdf').getOrCreate() # list of students data data =[["1","sravan","vignan"], ["2","ojaswi","vvit"], ["3","rohith","vvit"], ["4","sridevi","vignan"], ["1","sravan","vignan"], ["5","gnanesh","iit"]] # specify column namescolumns=['student ID','student NAME','college'] # creating a dataframe from the lists of datadataframe = spark.createDataFrame(data,columns) print("Actual data in dataframe")# show dataframedataframe.show() |
Output:
Actual data in dataframe +----------+------------+-------+ |student ID|student NAME|college| +----------+------------+-------+ | 1| sravan| vignan| | 2| ojaswi| vvit| | 3| rohith| vvit| | 4| sridevi| vignan| | 1| sravan| vignan| | 5| gnanesh| iit| +----------+------------+-------+
Example 1: Python program to extract a single value from a particular column using first().
Python3
# extract single value based on# column in the dataframedataframe.first()['student ID'] |
Output:
'1'
Example 2: Extract a single value using head().
Python3
# extract single value based# on column in the dataframedataframe.head()[0] |
Output:
'1'
Example 3: Extract a single value using head().
Python3
# extract single value based# on column in the dataframedataframe.head()[2] |
Output:
'vignan'
