YouTube is one of the oldest and most popular video distribution platforms in the world. We can’t even imagine the video content available here. It has billion of users and viewers, which keeps on increasing every passing minute.
Since its origins, YouTube and its content have transformed very much. Now we have SHORTS, likes, and many more features.
So here we will be doing the analysis for the neveropen Youtube channel, which includes the analysis of the time duration, likes, title of the video, etc.
Before that, we need the data. We can scrap the data using Web Scraping.
Web scraping the data
Web Scraping is the automation of the data extraction process from websites. Web Scrapers automatically load and extract data from websites based on user requirements. These can be custom-built to work for one site or can be configured to work with any website.
Here, we will be using Selenium and BeautifulSoup for web scraping.
After extracting the data, we will be converting it into an excel file. So for that, we will be using XLSXwriter library.
Python3
import time from selenium import webdriver from bs4 import BeautifulSoup import xlsxwriter |
The url of the main webpage/ youtube page must be provided in this format.
Python3
# provide the url of the channel whose data you want to fetch urls = [ ] |
Now , we will be creating the soup of the content extracted by the chrome driver.
You need to specify the path of chrome driver at the place of path_of_chrome_driver.
If you don’t have it, please install it and then specify the correct location.
Note : Mostly, the download location is ‘C:\Downloads\chromedriver.exe’
Python3
times = 0row = 0t = v = d = [] driver = webdriver.Chrome(executable_path='path_of_chrome_driver') for url in urls: driver.get('{}/videos?view=0&sort=p&flow=grid'.format(url)) while times < 5: time.sleep(1) driver.execute_script( "window.scrollTo(0, document.documentElement.scrollHeight);") times += 1 content = driver.page_source.encode('utf-8').strip() soup = BeautifulSoup(content, 'lxml') |
Now we will be extracting the title, duration, views as per their respective id/ class and storing them in a separate list. You can add more columns in a same way.
Python3
#Title titles = soup.findAll('a', id='video-title') t =[] for i in titles: t.append(i.text) #Views views = soup.findAll('span', class_='style-scope ytd-grid-video-renderer') v = [] count = 0for i in range(len(views)): if i%2 == 0: v.append(views[i].text) else: continue #Duration duration = soup.findAll( 'span', class_='style-scope ytd-thumbnail-overlay-time-status-renderer') d = [] for i in duration: d.append(i.text) |
Once we have the list, we are now ready to create the excel file.
Note : After creating an excel file and adding all the items. Please close it using workbook.close() command, else it will not show at the specified location.
Python3
workbook = xlsxwriter.Workbook('file.xlsx') worksheet = workbook.add_worksheet() worksheet.write(0, 0, "Title") worksheet.write(0, 1, "Views") worksheet.write(0, 2, "Duration") row = 1for title, view, dura in zip(t,v,d): worksheet.write(row, 0, title) worksheet.write(row, 1, view) worksheet.write(row, 2, dura) row += 1 workbook.close() |
Data Preprocessing
Data preprocessing involves the following steps:
- Removal of extra characters from columns (like spaces in duration)
- Conversion of values as per the requirement (e.g. 2.6k must be in form 2600)
- Conversion of duration column into categories.
For implementing the above steps, let’s start with loading the excel file we created above.
Python3
import pandas as pd data = pd.read_excel('file.xlsx') data.head() |
Output:
Removal of extra character from views column is done by checking if there is ‘k’ in the value and removing it. Then, converting it into float value then multiply it with 1000. Refer the below code for the same.
Python3
data['Views'] = data['Views'].str.replace(" views","") new = [] for i in data['Views']: if(i.endswith('K')): i = i.replace('K','') new.append(float(i) * 1000) else : new.append(i) data['Views'] = new |
Removal of extra character from Duration column is done by removing ‘\n’ . Then we need to convert it into seconds. For that, we will use loop and multiply the hour value with 3600 and minute value with 60 and add them with seconds value.
Python3
#Duration column cleaning data['Duration'] = data['Duration'].str.replace("\n","") new2 = [] for i in data['Duration']: if(i=='SHORTS' or len(i.split(':'))==1): new2.append(i) elif(len(i.split(':'))==2): i = i.split(':') tim = int(i[0])*60 + int(i[1]) new2.append(tim) elif(len(i.split(':'))==3): i = i.split(':') tim = int(i[0])*3600 + int(i[1])*60 + int(i[2]) new2.append(tim) data['Duration'] = new2 |
Once we get the seconds, we can easily categorize the values. In this article, we have taken 4 section :
- SHORTS
- Mini-Videos
- Long-Videos
- Very-Long-Videos
You can take more or less, as per your choice.
Python3
#Duration column categorization for i in data['Duration'].index: val = data['Duration'].iloc[i] if(val==' SHORTS'): continue elif(val in range(0,900)): data.loc[i,'Duration'] = 'Mini-Videos' elif(val in range(901,3600)): data.loc[i,'Duration'] = 'Long-Videos' else: data.loc[i,'Duration'] = 'Very-Long-Videos' |
After all the preprocessing, let’s check the new dataset.
Python3
data.head() |
Output:
Text Preprocessing
Text preprocessing refers to the cleaning of the text data by doing the following steps :
- Removal of punctuations
- Lowercase the characters
- Create tokens
- Remove Stopwords
We can do all these using NLTK Library.
Python3
import re from tqdm import tqdm import nltk nltk.download('punkt') nltk.download('stopwords') from nltk.corpus import stopwords from nltk.tokenize import word_tokenize from nltk.stem.porter import PorterStemmer |
After importing the libraries, run the below code for processing the Title column.
Python3
def preprocess_text(text_data): preprocessed_text = [] for sentence in tqdm(text_data): sentence = re.sub(r'[^\w\s]', '', sentence) preprocessed_text.append(' '.join(token.lower() for token in str(sentence).split() if token not in stopwords.words('english'))) return preprocessed_text preprocessed_review = preprocess_text(data['Title'].values) data['Title'] = preprocessed_review |
Data Visualization
Data visualization is the graphical representation of information and data in a pictorial or graphical format. Here we will be using WordCloud for checking the maximum used words in the whole data.
We will also use Seaborn library for better visualization.
Let’s import the libraries for that.
Python3
from wordcloud import WordCloud import matplotlib.pyplot as plt import seaborn as sns |
Wordcloud for the title column.
Python3
consolidated = ' '.join(word for word in data['Title'].astype(str)) wordCloud = WordCloud(width=1600, height=800, random_state=21, max_font_size=110, collocations=False) plt.figure(figsize=(15, 10)) plt.imshow(wordCloud.generate(consolidated), interpolation='bilinear') plt.axis('off') plt.show() |
Output:
Countplot for Duration column.
Python3
sns.countplot(data['Duration']) |
Output:
