A Comprehensive Guide to Data Analysis using Pandas: Hands-On Data Analysis on IMDB movies data

Introduction

If you are an aspiring Data Analyst / Data Scientist, am sure you know that Data Wrangling is one of the most crucial steps for any Data Science or Machine Learning project and it’s the longest too.

Python has a powerful and most popular package ‘Pandas’ built on top of Numpy which has the implementation of many data objects and data operations. Pandas is one of the most famous data science tools and it’s definitely a game-changer for cleaning, manipulating, and analyzing data.

In this article, we will explore two of the most important data structures of pandas:

1. Series

2. DataFrame

We will also perform hands-on Data Analysis on an interesting dataset on movies. We will learn some of the most useful operations and functionalities that pandas offer by directly analyzing real data

Why Pandas?

Pandas provide tools for reading and writing data into data structures and files. It also provides powerful aggregation functions to manipulate data.

Pandas provide extended data structures to hold different types of labeled and relational data. This makes python highly flexible and extremely useful for data cleaning and manipulation.

Pandas is highly flexible and provides functions for performing operations like merging, reshaping, joining, and concatenating data.

Let’s first look at the two most used data structures provided by Pandas.

Series

A Series can be thought of as a 1-D array or a single column of a 2D array or matrix. You can think of it as one column in an excel sheet of data. A series is a set of data values attached to a particular label. It also has specific index values attached to each row. These index values are automatically defined when the series is created. We can also explicitly define these indexes.

Let’s dive in, create and explore Series by actually writing code in a Jupyter notebook.

Open your Jupyter notebook and follow along!

The Jupyter notebook with the code for this article can be accessed here.

How to create Series?

A series object can be created from either a list or an array of values or from a dictionary with key-value pairs.

pd.Series( ) is the method used to create Series. It can take a list, array, or dictionary as a parameter.

1. Create Series from List

Let’s create a Series using a list of values

Python Code:

Here, the indexes are generated by default, but we can also define custom indexes at the time of Series creation.

Below is a Series of ‘Marks’ and associated ‘Subjects’. The list of subjects is set as a row index.

s2 = pd.Series([80,93,78,85,97], index=['English','Science','Social','Tamil','Maths'])
print("The Marks obtained by student are", s2)

The Marks obtained by student are Subject
English    80
Science    93
Social     78
Tamil      85
Maths      97
Name: Student Marks, dtype: int64

Indexing and Slicing operation in Series

Data retrieval and manipulation are the most essential operations that we perform during data analysis. Data stored in a Series can be retrieved using slicing operation by square brackets [ ]

# slicing using default integer index
s1[1:4]

1    20
2    30
3    40
dtype: int64

85

2. Create Series from Dictionary

A dictionary is a core Python data structure that stores data as a set of Key-Value pairs. A Series is also similar to a dictionary in a way that it maps given indexes to a set of values.

I have a dictionary that stores data about fruits and their prices. Let’s see how to create Series from this dictionary.

dict_fruits = { 'Orange':80,
          'Apples':210,
          'Bananas':50,
         'Grapes':90,
         'Watermelon':70}

Let’s convert ‘dict_fruits’ to a Series

# Lets convert this dictionary into a series
fruits = pd.Series(dict_fruits)

print("Fruits and pricesn", fruits)

Fruits and prices
 Orange         80
Apples        210
Bananas        50
Grapes         90
Watermelon     70
dtype: int64

Data from this series can be retrieved as below:

# Slice the series and retrieve price of Grapes
print("The price per kg of grapes is:", fruits['Grapes'])

The price per kg of grapes is: 90

DataFrame

The next important data structure in pandas is the most widely used ‘DataFrame’.

A DataFrame can be thought of as a multi-dimensional table or a table of data in an excel file. It is a multi-dimensional table structure essentially made up of a collection of Series. It helps us store tabular data where each row is an observation and the columns represent variables.

pd.DataFrame( ) is the function used to create a dataframe.

A DataFrame can be created in multiple ways. Let’s look at each one of them

1. Create a Dataframe from Series object

A dataframe can be created by passing a series (or multiple) into the DataFrame creation method. The columns can be named using the optional input parameter ‘columns’

Let’s create a Dataframe using the Series we created in the above step:

df_marks = pd.DataFrame(s2, columns=['Student1'])
print("The dataframe created from series isn",df_marks)

The dataframe created from series is
          Student1
English        80
Science        93
Social         78
Tamil          85
Maths          97

2. Create DataFrame from a dictionary object

Let’s say we have 2 series of heights and weights of a set of persons and we want to put it together in a table.

# Create Height series (in feet)
height = pd.Series([5.3, 6.2,5.8,5.0,5.5], index=['Person 1','Person 2','Person 3','Person 4','Person 5'])
# Create Weight Series (in kgs)
weight = pd.Series([65,89,75,60,59], index=['Person 1','Person 2','Person 3','Person 4','Person 5'])

We will create a dictionary using both ‘height’ and ‘weight’ Series, and finally, create a dataframe using pd.DataFrame( ) method.

# Create dataframe
df_person = pd.DataFrame({'height': height, 'weight': weight})
print("The Person table details are:n", df_person)

The Person table details are:
           height  weight
Person 1     5.3      65
Person 2     6.2      89
Person 3     5.8      75
Person 4     5.0      60
Person 5     5.5      59

3. Create a dataframe by importing data from File

Pandas is extremely useful and comes in handy when we want to load data from various file formats like CSV, Excel, JSON, etc.

Here are few methods to read data into dataframe from other file objects

• read_table( )
• read_csv( )
• read_html( )
• read_json( )
• read_pickle( )

For the purpose of this article, we will consider only reading data from the CSV file.

Data Analysis of IMDB movies data

As we have a basic understanding of the different data structures in Pandas, let’s explore the fun and interesting ‘IMDB-movies-dataset’ and get our hands dirty by performing practical data analysis on real data.

It is an open-source dataset and you can download it from this link.

What’s more fun than performing hands-on data analysis?

So put on your hats as a Data Analyst/ Data Scientist and let’s GET.SET.GO

We will read the data from the .csv file and perform the following basic operations on movies data

1. Read data
2. View the data
3. Understand some basic information about the data
4. Data Selection – Indexing and Slicing data
5. Data Selection – Based on Conditional filtering
6. Groupby operations
7. Sorting operation
8. Dealing with missing values
9. Dropping columns and null values
10. Apply( ) functions

1. Read data

Load data from CSV file.

# Read data from .csv file
data = pd.read_csv('IMDB-Movie-Data.csv')

# Read data with specified explicit index.
# We will use this later in our analysis
data_indexed = pd.read_csv('IMDB-Movie-Data.csv', index_col="Title")

2. View data

Let’s do a quick preview of the data by using head( ) and tail( ) methods

head( ) 

• Returns the top 5 rows in the dataset by default 
• It can also take the number of rows to be viewed as a parameter

tail( )

• Returns the bottom 5 rows in the dataset by default
• It can also take the number of rows as an optional parameter

# Preview top 5 rows using head()
data.head()

Image Source : Output from Data Analysis

3. Understand basic information about the data

Pandas provide many functions to understand the shape, number of columns, indexes, and other information about the dataframe.

• info( ) is one of my favorite methods that gives all necessary information about different columns in a dataframe.

#Lets first understand the basic information about this data
data.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1000 entries, 0 to 999
Data columns (total 12 columns):
 #   Column              Non-Null Count  Dtype  
---  ------              --------------  -----  
 0   Rank                1000 non-null   int64  
 1   Title               1000 non-null   object 
 2   Genre               1000 non-null   object 
 3   Description         1000 non-null   object 
 4   Director            1000 non-null   object 
 5   Actors              1000 non-null   object 
 6   Year                1000 non-null   int64  
 7   Runtime (Minutes)   1000 non-null   int64  
 8   Rating              1000 non-null   float64
 9   Votes               1000 non-null   int64  
 10  Revenue (Millions)  872 non-null    float64
 11  Metascore           936 non-null    float64
dtypes: float64(3), int64(4), object(5)
memory usage: 93.9+ KB

• shape can be used to get the shape of dataframe
• columns gives us the list of columns in the dataframe

data.shape

(1000, 12)

This function tells us that there are 1000 rows and 12 columns in the dataset

• describe( ) method gives the basic statistical summaries of all numerical attributes in the dataframe.

data.describe()

Image Source : Output from Data Analysis

Some insights from the description table

• The min and max values in ‘Year’ depict the minimum and maximum release years. We can see that the dataset contains movies from 2006 to 2016.
• The average rating for the movies in this dataset is about 6.7 and the minimum rating is 1.9 and the maximum rating is 9.0
• The maximum revenue earned by a movie is 936.6 million

4. Data Selection – Indexing and Slicing

Extract data using columns

Extracting data from a dataframe is similar to Series. Here the column label is used to extract data from the columns.

Let’s quickly extract ‘Genre’ data from the dataframe

# Extract data as series
genre = data['Genre']

This operation will retrieve all the data from the ‘Genre’ column as Series. If we want to retrieve this data as a dataframe, then indexing must be done using double square brackets as below:

# Extract data as dataframe
data[['Genre']]

If we want to extract multiple columns from the data, simply add the column names to the list.

some_cols = data[['Title','Genre','Actors','Director','Rating']]

Extract data using rows

loc and iloc are two functions that can be used to slice data from specific row indexes.

loc – locates the rows by name

• loc performs slicing based explicit index.
• It takes string indexes to retrieve data from specified rows

iloc – locates the rows by integer index

• iloc performs slicing based on Python’s default numerical index.

In the beginning, when we read the data, we created a dataframe with ‘Title’ as the string index.

We will use the loc function to index and slice that dataframe using the specified ‘Title’.

data_indexed.loc[['Suicide Squad']][['Genre','Actors','Director','Rating','Revenue (Millions)']]

Here, iloc is used to slice data using integer indexes.

data.iloc[10:15][['Title','Rating','Revenue (Millions)']]

5. Data Selection – Based on Conditional Filtering

Pandas also enable retrieving data from dataframe based on conditional filters.

What if we want to pick only movies that are released from 2010 to 2016, have a rating of less than 6.0 but topped in terms of revenue?

It’s very simple and can be retrieved in a single line of code…

data[((data['Year'] >= 2010) & (data['Year'] <= 2016))
      & (data['Rating'] < 6.0)
      & (data['Revenue (Millions)'] > data['Revenue (Millions)'].quantile(0.95))]

Image Source : Output from Data Analysis

‘The Twilight Saga: Breaking Dawn – Part 2′ and ‘The Twilight Saga: Eclipse’ are the movies that topped in the box office, despite having lower ratings.

6. Groupby operation

Data can be grouped and operations can be performed on top of grouped data by using the groupby( ) method. This comes in handy when we want to apply aggregations and functions on top of grouped data.

data.groupby('Director')[['Rating']].mean().head()

7. Sorting operation

Sorting is yet another pandas operation that is heavily used in data analysis projects.

sort_values( ) method is used to perform sorting operation on a column or a list of multiple columns

In the above example, where we have listed the average rating for each ‘Director’, if we want to sort them from highly rated to lowest, we can perform the sorting operation.

data.groupby('Director')[['Rating']].mean().sort_values(['Rating'], ascending=False).head()

We can see that Director ‘Nitesh Tiwari’ has the highest average rating in this dataset

8. Dealing with missing values

Pandas has isnull( ) for detecting null values in a dataframe. Let’s see how to use these methods.

# To check null values row-wise
data.isnull().sum()

Rank                    0
Title                   0
Genre                   0
Description             0
Director                0
Actors                  0
Year                    0
Runtime (Minutes)       0
Rating                  0
Votes                   0
Revenue (Millions)    128
Metascore              64
dtype: int64

Here we know that ‘Revenue (Millions)’ and ‘Metascore’ are two columns where there are null values.

As we have seen null values in data, we can either choose to drop those or impute these values

9. Dropping columns and null values

Dropping columns/rows is yet another operation that is most important for data analysis.

drop( ) function can be used to drop rows or columns based on condition

# Use drop function to drop columns
data.drop('Metascore', axis=1).head()

Using the above code, the ‘Metascore’ column is dropped completely from data. Here axis= 1 specifies that column is to be dropped. These changes will not take place in actual data unless we specify inplace=True as a parameter in the drop( ) function.

We can also drop rows/ columns with null values by using dropna( ) function.

# Drops all rows containing missing data
data.dropna()

# Drop all columns containing missing data
data.dropna(axis=1)

data.dropna(axis=0, thresh=6)

In the above snippet, we are using thresh parameter to specify the minimum number of non-null values for the column/row to be held without dropping.

In our movies data, we know that there are some records where the Revenue is null.

We can impute these null values with mean Revenue (Millions).

fillna( ) –> function used to fill null values with specified values

revenue_mean = data_indexed['Revenue (Millions)'].mean()
print("The mean revenue is: ", revenue_mean)

The mean revenue is:  82.95637614678897

# We can fill the null values with this mean revenue
data_indexed['Revenue (Millions)'].fillna(revenue_mean, inplace=True)

Now, if we check the dataframe, there won’t be any null values in the Revenue column

10. Apply( ) function

The apply( ) function comes in handy when we want to apply any function to the dataset. It returns a value after passing each row of the dataframe to some function. The function can be built-in or user-defined.

For example, if we want to classify the movies based on their ratings, we can define a function to do so and then apply the function to the dataframe as shown below.

I will write a function that will classify movies into groups based on rating.

# Classify movies based on ratings
def rating_group(rating):
    if rating >= 7.5:
        return 'Good'
    elif rating >= 6.0:
        return 'Average'
    else:
        return 'Bad'

Now, I will apply this function to our actual dataframe and the ‘Rating_category’ will be computed for each row.

# Lets apply this function on our movies data
# creating a new variable in the dataset to hold the rating category
data['Rating_category'] = data['Rating'].apply(rating_group)

Here is the resultant data after applying the rating_group( ) function.

data[['Title','Director','Rating','Rating_category']].head(5)

We learned about most of the important operations in Pandas that are needed for data processing and manipulation. And it was great fun analyzing the IMBD movies data.

If you would like to delve deep into some of these operations, please have a look at this article’s Jupyter notebook from my Github.

This is my first article on Data Science. I hope it was helpful and  I would love to hear your valuable feedback through comments.

Happy Learning!

Lakshana G V

26 Jul 2022