In this article, we are going to see how to append data to an empty DataFrame in PySpark in the Python programming language.
Method 1: Make an empty DataFrame and make a union with a non-empty DataFrame with the same schema
The union() function is the most important for this operation. It is used to mix two DataFrames that have an equivalent schema of the columns.
Syntax : FirstDataFrame.union(Second DataFrame)
Returns : DataFrame with rows of both DataFrames.
Example:
In this example, we create a DataFrame with a particular schema and data create an EMPTY DataFrame with the same scheme and do a union of these two DataFrames using the union() function in the python language.
Python
# Importing PySpark and the SparkSession# DataType functionalityimport pysparkfrom pyspark.sql import SparkSessionfrom pyspark.sql.types import *# Creating a spark sessionspark_session = SparkSession.builder.appName( 'Spark_Session').getOrCreate()# Creating an empty RDD to make a# DataFrame with no dataemp_RDD = spark_session.sparkContext.emptyRDD()# Defining the schema of the DataFramecolumns1 = StructType([StructField('Name', StringType(), False), StructField('Salary', IntegerType(), False)])# Creating an empty DataFramefirst_df = spark_session.createDataFrame(data=emp_RDD, schema=columns1)# Printing the DataFrame with no datafirst_df.show()# Hardcoded data for the second DataFramerows = [['Ajay', 56000], ['Srikanth', 89078], ['Reddy', 76890], ['Gursaidutt', 98023]]columns = ['Name', 'Salary']# Creating the DataFramesecond_df = spark_session.createDataFrame(rows, columns)# Printing the non-empty DataFramesecond_df.show()# Storing the union of first_df and# second_df in first_dffirst_df = first_df.union(second_df)# Our first DataFrame that was empty,# now has datafirst_df.show() |
Output :
+----+------+ |Name|Salary| +----+------+ +----+------+ +----------+------+ | Name|Salary| +----------+------+ | Ajay| 56000| | Srikanth| 89078| | Reddy| 76890| |Gursaidutt| 98023| +----------+------+ +----------+------+ | Name|Salary| +----------+------+ | Ajay| 56000| | Srikanth| 89078| | Reddy| 76890| |Gursaidutt| 98023| +----------+------+
Method 2: Add a singular row to an empty DataFrame by converting the row into a DataFrame
We can use createDataFrame() to convert a single row in the form of a Python List. The details of createDataFrame() are :
Syntax : CurrentSession.createDataFrame(data, schema=None, samplingRatio=None, verifySchema=True)
Parameters :
data :
- schema : str/list , optional: Contains a String or List of column names.
- samplingRatio : float, optional: A sample of rows for inference
- verifySchema : bool, optional: Verify data types of every row against the specified schema. The value is True by default.
Example:
In this example, we create a DataFrame with a particular schema and single row and create an EMPTY DataFrame with the same schema using createDataFrame(), do a union of these two DataFrames using union() function further store the above result in the earlier empty DataFrame and use show() to see the changes.
Python
# Importing PySpark and the SparkSession,# DataType functionalityimport pysparkfrom pyspark.sql import SparkSessionfrom pyspark.sql.types import *# Creating a spark sessionspark_session = SparkSession.builder.appName( 'Spark_Session').getOrCreate()# Creating an empty RDD to make a DataFrame# with no dataemp_RDD = spark_session.sparkContext.emptyRDD()# Defining the schema of the DataFramecolumns = StructType([StructField('Stadium', StringType(), False), StructField('Capacity', IntegerType(), False)])# Creating an empty DataFramedf = spark_session.createDataFrame(data=emp_RDD, schema=columns)# Printing the DataFrame with no datadf.show()# Hardcoded row for the second DataFrameadded_row = [['Motera Stadium', 132000]]# Creating the DataFrameadded_df = spark_session.createDataFrame(added_row, columns)# Storing the union of first_df and second_df# in first_dfdf = df.union(added_df)# Our first DataFrame that was empty,# now has datadf.show() |
Output :
+-------+--------+ |Stadium|Capacity| +-------+--------+ +-------+--------+ +--------------+--------+ | Stadium|Capacity| +--------------+--------+ |Motera Stadium| 132000| +--------------+--------+
Method 3: Convert the empty DataFrame into a Pandas DataFrame and use the append() function
We will use toPandas() to convert PySpark DataFrame to Pandas DataFrame. Its syntax is :
Syntax : PySparkDataFrame.toPandas()
Returns : Corresponding Pandas DataFrame
We will then use the Pandas append() function. Its syntax is :
Syntax : PandasDataFrame.append(other, ignore_index=False, verify_integrity=False, sort=False)
Parameters :
- other : Pandas DataFrame, Numpy Series etc: The data that has to be appended.
- ignore_index : bool: If indexed a ignored then the indexes of the new DataFrame will have no relations to the older ones.
- sort : bool: Sort the columns if alignment of the columns in other and PandasDataFrame is different.
Example:
Here we create an empty DataFrame where data is to be added, then we convert the data to be added into a Spark DataFrame using createDataFrame() and further convert both DataFrames to a Pandas DataFrame using toPandas() and use the append() function to add the non-empty data frame to the empty DataFrame and ignore the indexes as we are getting a new DataFrame.Finally, we convert our final Pandas DataFrame to a Spark DataFrame using createDataFrame().
Python
# Importing PySpark and the SparkSession,# DataType functionalityimport pysparkfrom pyspark.sql import SparkSessionfrom pyspark.sql.types import *# Creating a spark sessionspark_session = SparkSession.builder.appName( 'Spark_Session').getOrCreate()# Creating an empty RDD to make a DataFrame# with no dataemp_RDD = spark_session.sparkContext.emptyRDD()# Defining the schema of the DataFramecolumns = StructType([StructField('Stadium', StringType(), False), StructField('Capacity', IntegerType(), False)])# Creating an empty DataFramedf = spark_session.createDataFrame(data=emp_RDD, schema=columns)# Printing the DataFrame with no datadf.show()# Hardcoded row for the second DataFrameadded_row = [['Motera Stadium', 132000]]# Creating the DataFrame whose data# needs to be addedadded_df = spark_session.createDataFrame(added_row, columns)# converting our PySpark DataFrames to# Pandas DataFramespandas_added = added_df.toPandas()df = df.toPandas()# using append() function to add the datadf = df.append(pandas_added, ignore_index=True)# reconverting our DataFrame back# to a PySpark DataFramedf = spark_session.createDataFrame(df)# Printing resultant DataFramedf.show() |
Output :
+-------+--------+ |Stadium|Capacity| +-------+--------+ +-------+--------+ +--------------+--------+ | Stadium|Capacity| +--------------+--------+ |Motera Stadium| 132000| +--------------+--------+
