PySpark convert multiple columns to map

In this article, we are going to convert multiple columns to map using Pyspark in Python.

An RDD transformation that is used to apply the transformation function on every element of the data frame is known as a map. While working in the Pyspark data frame, we might encounter some circumstances in which we need to convert columns of the data frame to map columns of the data frame as the map keys.

Syntax: create_map( lit(“mapkey_1”),col(“column_1”))

Parameters:

• column_1: These are the column names which needs to be converted to map.
• mapkey_1: These are the names of the map keys to be given to data on creation of map.

Stepwise Implementation:

Step 1: First of all, import the required libraries, i.e., SparkSession, col, lit, and create_map. The SparkSession is used to create the session, while col is used to return a column based on the given column name. The lit is used to add a new column to the DataFrame by assigning a literal or constant value, while create_map is used to convert selected DataFrame columns to MapType. 

from pyspark.sql import SparkSession
from pyspark.sql.functions import col,lit,create_map

Step 2: Now, we create a spark session using getOrCreate() function. 

spark_session = SparkSession.builder.getOrCreate()

Step 3: Then, either read the CSV file for the data frame or create a new data frame using createDataFrame() function. 

data_frame=csv_file = spark_session.read.csv('#Path of CSV file',
                                             sep = ',', inferSchema = True,
                                             header = True)

or

data_frame=spark_session.createDataFrame(
                     [(column_1_data), (column_2_data), (column_3_data)],
                     ['column_name_1', 'column_name_2', 'column_name_3'])

Step 4: Further, create a column having map_column_name and create_map() function as arguments. The create_map() function will contain all the column names that need to be converted to map and the name to be given to mapkeys in the map. 

data_frame.withColumn("map_column_name", create_map(lit("mapkey_1"),
                       col("column_name_1"), lit("mapkey_2"),
                       col("column_name_2"), lit("mapkey_3"),
                       col("column_name_3"))).drop("column_name_1",
                                                   "column_name_2",
                                                   "column_name_3")

Step 5: Finally, display the updated data frame. 

data_frame.show(truncate=False)

Example 1:

In this example, we have used a data set (link), which is basically a 5×5 data frame as follows:

 

Then, we converted the columns ‘name,’ ‘class‘ and ‘fees‘ to map using create_map() function and stored them in the column ‘student_details‘ dropping the existing ‘name,’ ‘class‘ and ‘fees‘ columns.

Output:

 

Example 2:

In this example, we have created a data frame with columns emp_id, name, superior_emp_id, year_joined, emp_dept_id, gender, and salary as follows: 

 

Then, we converted the columns name, superior_emp_id, year_joined, emp_dept_id, gender, and salary to map using create_map() function and stored in the column ‘employee_details‘ dropping the existing name, superior_emp_id, year_joined, emp_dept_id, gender, and salary columns.

Output: