In this article, we are going to see how to create an empty PySpark dataframe. Empty Pysaprk dataframe is a dataframe containing no data and may or may not specify the schema of the dataframe.
Creating an empty RDD without schema
We’ll first create an empty RDD by specifying an empty schema.
- emptyRDD() method creates an RDD without any data.
- createDataFrame() method creates a pyspark dataframe with the specified data and schema of the dataframe.
Code:
Python3
# Import necessary librariesfrom pyspark.sql import SparkSessionfrom pyspark.sql.types import *# Create a spark sessionspark = SparkSession.builder.appName('Empty_Dataframe').getOrCreate()# Create an empty RDDemp_RDD = spark.sparkContext.emptyRDD()# Create empty schemacolumns = StructType([])# Create an empty RDD with empty schemadata = spark.createDataFrame(data = emp_RDD, schema = columns)# Print the dataframeprint('Dataframe :')data.show()# Print the schemaprint('Schema :')data.printSchema() |
Output:
Dataframe : ++ || ++ ++ Schema : root
Creating an emptyRDD with schema
It is possible that we will not get a file for processing. However, we must still manually create a DataFrame with the appropriate schema.
- Specify the schema of the dataframe as columns = [‘Name’, ‘Age’, ‘Gender’].
- Create an empty RDD with an expecting schema.
Code:
Python3
# Import necessary librariesfrom pyspark.sql import SparkSessionfrom pyspark.sql.types import *# Create a spark sessionspark = SparkSession.builder.appName('Empty_Dataframe').getOrCreate()# Create an empty RDDemp_RDD = spark.sparkContext.emptyRDD()# Create an expected schemacolumns = StructType([StructField('Name', StringType(), True), StructField('Age', StringType(), True), StructField('Gender', StringType(), True)])# Create an empty RDD with expected schemadf = spark.createDataFrame(data = emp_RDD, schema = columns)# Print the dataframeprint('Dataframe :')df.show()# Print the schemaprint('Schema :')df.printSchema() |
Output :
Dataframe : +----+---+------+ |Name|Age|Gender| +----+---+------+ +----+---+------+ Schema : root |-- Name: string (nullable = true) |-- Age: string (nullable = true) |-- Gender: string (nullable = true)
Creating an empty dataframe without schema
- Create an empty schema as columns.
- Specify data as empty([]) and schema as columns in CreateDataFrame() method.
Code:
Python3
# Import necessary librariesfrom pyspark.sql import SparkSessionfrom pyspark.sql.types import *# Create a spark sessionspark = SparkSession.builder.appName('Empty_Dataframe').getOrCreate()# Create an empty schemacolumns = StructType([])# Create an empty dataframe with empty schemadf = spark.createDataFrame(data = [], schema = columns)# Print the dataframeprint('Dataframe :')df.show()# Print the schemaprint('Schema :')df.printSchema() |
Output:
Dataframe : ++ || ++ ++ Schema : root
Creating an empty dataframe with schema
- Specify the schema of the dataframe as columns = [‘Name’, ‘Age’, ‘Gender’].
- Specify data as empty([]) and schema as columns in CreateDataFrame() method.
Code:
Python3
# Import necessary librariesfrom pyspark.sql import SparkSessionfrom pyspark.sql.types import *# Create a spark sessionspark = SparkSession.builder.appName('Empty_Dataframe').getOrCreate()# Create an expected schemacolumns = StructType([StructField('Name', StringType(), True), StructField('Age', StringType(), True), StructField('Gender', StringType(), True)])# Create a dataframe with expected schemadf = spark.createDataFrame(data = [], schema = columns)# Print the dataframeprint('Dataframe :')df.show()# Print the schemaprint('Schema :')df.printSchema() |
Output :
Dataframe : +----+---+------+ |Name|Age|Gender| +----+---+------+ +----+---+------+ Schema : root |-- Name: string (nullable = true) |-- Age: string (nullable = true) |-- Gender: string (nullable = true)
