A series of data points collected over the course of a time period, and that are time-indexed is known as Time Series data. These observations are recorded at successive equally spaced points in time. For Example, the ECG Signal, EEG Signal, Stock Market, Weather Data, etc., all are time-indexed and recorded over a period of time. Analyzing these data, and predicting future observations has a wider scope of research.
In this article, we will see how to implement EDA — Exploratory Data Analysis using Pandas Library in Python. We will try to infer the nature of the data over a specific period of time by plotting various graphs with matplotlib.pyplot, seaborn, statsmodels, and more packages.
For easy understanding of the plots and other functions, we will be creating a sample dataset with 16 rows and 5 columns which includes Date, A, B, C, D, and E columns.
Python3
import pandas as pd# Sample data which will be used# to create the dataframesample_timeseries_data = { 'Date': ['2020-01-25', '2020-02-25', '2020-03-25', '2020-04-25', '2020-05-25', '2020-06-25', '2020-07-25', '2020-08-25', '2020-09-25', '2020-10-25', '2020-11-25', '2020-12-25', '2021-01-25', '2021-02-25', '2021-03-25', '2021-04-25'], 'A': [102, 114, 703, 547, 641, 669, 897, 994, 1002, 974, 899, 954, 1105, 1189, 1100, 934], 'B': [1029, 1178, 723, 558, 649, 669, 899, 1000, 1012, 984, 918, 959, 1125, 1199, 1109, 954], 'C': [634, 422,152, 23, 294, 1452, 891, 990, 924, 960, 874, 548, 174, 49, 655, 914], 'D': [1296, 7074, 3853, 4151, 2061, 1478, 2061, 3853, 6379, 2751, 1064, 6263, 2210, 6566, 3918, 1121], 'E': [10, 17, 98, 96, 85, 89, 90, 92, 86, 84, 78, 73, 71, 65, 70, 60]}# Creating a dataframe using pandas# module with Date, A, B, C, D and E# as columns.dataframe = pd.DataFrame( sample_timeseries_data,columns=[ 'Date', 'A', 'B', 'C', 'D', 'E'])# Changing the datatype of Date, from# Object to datetime64dataframe["Date"] = dataframe["Date"].astype("datetime64")# Setting the Date as indexdataframe = dataframe.set_index("Date")dataframe |
Output:
Sample Time Series data frame
Plotting the Time-Series Data
Plotting Timeseries based Line Chart:
Line charts are used to represent the relation between two data X and Y on a different axis.
Syntax: plt.plot(x)
Example 1: This plot shows the variation of Column A values from Jan 2020 till April 2020. Note that the values have a positive trend overall, but there are ups and downs over the course.
Python3
import matplotlib.pyplot as plt# Using a inbuilt style to change# the look and feel of the plotplt.style.use("fivethirtyeight")# setting figure size to 12, 10plt.figure(figsize=(12, 10))# Labelling the axes and setting# a titleplt.xlabel("Date")plt.ylabel("Values")plt.title("Sample Time Series Plot")# plotting the "A" column aloneplt.plot(dataframe["A"]) |
Output:
Sample Time Series Plot
Example 2: Plotting with all variables.
Python3
plt.style.use("fivethirtyeight")dataframe.plot(subplots=True, figsize=(12, 15)) |
Output:
Plotting all Time-series data columns
Plotting Timeseries based Bar Plot:
A bar plot or bar chart is a graph that represents the category of data with rectangular bars with lengths and heights that is proportional to the values which they represent. The bar plots can be plotted horizontally or vertically. A bar chart describes the comparisons between the discrete categories. One of the axis of the plot represents the specific categories being compared, while the other axis represents the measured values corresponding to those categories.
Syntax: plt.bar(x, height, width, bottom, align)
This bar plot represents the variation of the ‘A’ column values. This can be used to compare the future and the fast values.
Python3
import matplotlib.pyplot as plt# Using a inbuilt style to change# the look and feel of the plotplt.style.use("fivethirtyeight")# setting figure size to 12, 10plt.figure(figsize=(15, 10))# Labelling the axes and setting a# titleplt.xlabel("Date")plt.ylabel("Values")plt.title("Bar Plot of 'A'")# plotting the "A" column aloneplt.bar(dataframe.index, dataframe["A"], width=5) |
Output:
Bar Plot for ‘A’ column
Plotting Timeseries based Rolling Mean Plots:
The mean of an n-sized window sliding from the beginning to the end of the data frame is known as Rolling Mean. If the window doesn’t have n observations, then NaN is returned.
Syntax: pandas.DataFrame.rolling(n).mean()
Example:
Python3
dataframe.rolling(window = 5).mean() |
Output:
The rolling mean of dataframe
Here, we will plot the time series with a rolling means plot:
Python3
import matplotlib.pyplot as plt# Using a inbuilt style to change# the look and feel of the plotplt.style.use("fivethirtyeight")# setting figure size to 12, 10plt.figure(figsize=(12, 10))# Labelling the axes and setting# a titleplt.xlabel("Date")plt.ylabel("Values")plt.title("Values of 'A' and Rolling Mean (2) Plot")# plotting the "A" column and "A" column# of Rolling Dataframe (window_size = 20)plt.plot(dataframe["A"])plt.plot(dataframe.rolling( window=2, min_periods=1).mean()["A"]) |
Output:
Rolling mean plot
Explanation:
- The Blue Plot Line represents the original ‘A’ column values while the Red Plot Line represents the Rolling mean of ‘A’ column values of window size = 2
- Through this plot, we infer that the rolling mean of a time-series data returns values with fewer fluctuations. The trend of the plot is retained but unwanted ups and downs which are of less significance are discarded.
- For plotting the decomposition of time-series data, box plot analysis, etc., it is a good practice to use a rolling mean data frame so that the fluctuations don’t affect the analysis, especially in forecasting the trend.
Time Series Decomposition:
It shows the observations and these four elements in the same plot:
- Trend Component: It shows the pattern of the data that spans across the various seasonal periods. It represents the variation of ‘A’ values over the period of 2 years with no fluctuations.
- Seasonal Component: This plot shows the ups and downs of the ‘A’ values i.e. the recurring normal variations.
- Residual Component: This is the leftover component after decomposing the ‘A’ values data into Trend and Seasonal Component.
- Observed Component: This trend and a seasonal component can be used to study the data for various purposes.
Example:
Python3
import statsmodels.api as smfrom pylab import rcParamsimport pandas as pdimport matplotlib.pyplot as pltimport seaborn as sns# Separating the Date Component into# Year and Monthdataframe['Date'] = dataframe.indexdataframe['Year'] = dataframe['Date'].dt.yeardataframe['Month'] = dataframe['Date'].dt.month# using inbuilt styleplt.style.use("fivethirtyeight")# Creating a dataframe with "Date" and "A"# columns only. This dataframe is date indexeddecomposition_dataframe = dataframe[['Date', 'A']].copy()decomposition_dataframe.set_index('Date', inplace=True)decomposition_dataframe.index = pd.to_datetime(decomposition_dataframe.index)# using sm.tsa library, we are plotting the# seasonal decomposition of the "A" column# Multiplicative Model : Y[t] = T[t] * S[t] * R[t]decomposition = sm.tsa.seasonal_decompose(decomposition_dataframe, model='multiplicative', freq=5)decomp = decomposition.plot()decomp.suptitle('"A" Value Decomposition')# changing the runtime configuration parameters to# have a desired plot of desired size, etcrcParams['figure.figsize'] = 12, 10rcParams['axes.labelsize'] = 12rcParams['ytick.labelsize'] = 12rcParams['xtick.labelsize'] = 12 |
Output:
‘A’ value decomposition
Plotting Timeseries based Autocorrelation Plot:
It is a commonly used tool for checking randomness in a data set. This randomness is ascertained by computing autocorrelation for data values at varying time lags. It shows the properties of a type of data known as a time series. These plots are available in most general-purpose statistical software programs. It can be plotted using the pandas.plotting.autocorrelation_plot().
Syntax: pandas.plotting.autocorrelation_plot(series, ax=None, **kwargs)
Parameters:
- series: This parameter is the Time series to be used to plot.
- ax: This parameter is a matplotlib axes object. Its default value is None.
Returns: This function returns an object of class matplotlib.axis.Axes
Considering the trend, seasonality, cyclic and residual, this plot shows the current value of the time-series data is related to the previous values. We can see that a significant proportion of the line shows an effective correlation with time, and we can use such correlation plots to study the internal dependence of time-series data.
Code:
Python3
from pandas.plotting import autocorrelation_plotautocorrelation_plot(dataframe['A']) |
Output:
Autocorrelation plot
Plotting Timeseries based Box Plot:
Box Plot is the visual representation of the depicting groups of numerical data through their quartiles. Boxplot is also used for detecting the outlier in data set. It captures the summary of the data efficiently with a simple box and whiskers and allows us to compare easily across groups. Boxplot summarizes a sample data using 25th, 50th and 75th percentiles.
Syntax: seaborn.boxplot(x=None, y=None, hue=None, data=None, order=None, hue_order=None, orient=None, color=None, palette=None, saturation=0.75, width=0.8, dodge=True, fliersize=5, linewidth=None, whis=1.5, ax=None, **kwargs)
Parameters:
x, y, hue: Inputs for plotting long-form data.
data: Dataset for plotting. If x and y are absent, this is interpreted as wide-form.
color: Color for all of the elements.Returns: It returns the Axes object with the plot drawn onto it.
Here, through these plots, we will be able to obtain an intuition of the ‘A’ value ranges of each year (Year-wise Box Plot) as well as each month (Month-wise Box Plot). Also, through the Month-wise Box Plot, we can observe that the value range is slightly higher in Jan and Feb, compared to other months.
Python3
# Splitting the plot into (1,2) subplots# and initializing them using fig and ax# variablesfig, ax = plt.subplots(nrows=1, ncols=2, figsize=(15, 6))# Using Seaborn Library for Box Plotsns.boxplot(dataframe['Year'], dataframe["A"], ax=ax[0])# Defining the title and axes namesax[0].set_title('Year-wise Box Plot for A', fontsize=20, loc='center')ax[0].set_xlabel('Year')ax[0].set_ylabel('"A" values')# Using Seaborn Library for Box Plotsns.boxplot(dataframe['Month'], dataframe["A"], ax=ax[1])# Defining the title and axes namesax[1].set_title('Month-wise Box Plot for A', fontsize=20, loc='center')ax[1].set_xlabel('Month')ax[1].set_ylabel('"A" values')# rotate the ticks and right align themfig.autofmt_xdate() |
Output:
box plot analysis of ‘A’ column values
Shift Analysis:
This plot the achieved by dividing the current value of the ‘A’ column by the shifted value of the ‘A’ column. Default Shift is by one value. This plot is used to analyze the value stability on a daily basis.
Python3
dataframe['Change'] = dataframe.A.div(dataframe.A.shift())dataframe['Change'].plot(figsize=(15, 10), xlabel = "Date", ylabel = "Value Difference", title = "Shift Plot") |
Output:
shift plot of the ‘A’ values
Plotting Timeseries based Heatmap:
We can interpret the trend of the “A” column values across the years sampled over 12 months, variation of values across different years, etc. We can also infer how the values have changed from the average value. This heatmap is a really useful visualization. This Heatmap shows the variation of temperature across Years as well as Months, differentiated using a Colormap.
Python3
import calendarimport seaborn as snsimport pandas as pddataframe['Date'] = dataframe.index# Splitting the Date into Year and Monthdataframe['Year'] = dataframe['Date'].dt.yeardataframe['Month'] = dataframe['Date'].dt.month# Creating a Pivot Table with "A"# column values and is Month indexed.table_df = pd.pivot_table(dataframe, values=["A"], index=["Month"], columns=["Year"], fill_value=0, margins=True)# Naming the index, can be generated# using calendar.month_abbr[i]mon_name = [['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep','Oct', 'Nov', 'Dec', 'All']]# Indexing using Month Namestable_df = table_df.set_index(mon_name)# Creating a heatmap using sns with Red,# Yellow & Green Colormap.ax = sns.heatmap(table_df, cmap='RdYlGn_r', robust=True, fmt='.2f', annot=True, linewidths=.6, annot_kws={'size':10}, cbar_kws={'shrink':.5, 'label':'"A" values'}) # Setting the Tick Labels, Title and x & Y labelsax.set_yticklabels(ax.get_yticklabels())ax.set_xticklabels(ax.get_xticklabels())plt.title('"A" Value Analysis', pad=14)plt.xlabel('Year')plt.ylabel('Months') |
Output:
heatmap table for “A” column
Heatmap Plot for the “A” column values
