Python – Efficient Text Data Cleaning

Gone are the days when we used to have data mostly in row-column format, or we can say Structured data. In present times, the data being collected is more unstructured than structured. We have data in the form of text, images, audio etc and the ratio of Structured to Unstructured data has decreased over the years. Unstructured data is increasing at 55-65% every year.

Thus, we need to learn how to work with unstructured data to be able to extract relevant information from it and make it useful. While working with text data it is very important to pre-process it before using it for predictions or analysis.
In this article, we will be learning various text data cleaning techniques using python.

Let’s take a tweet for example:

I enjoyd the event which took place yesteday & I luvd it ! The link to the show is 
http://t.co/4ftYom0i It's awesome you'll luv it #HadFun #Enjoyed BFN GN

We will be performing data cleaning on this tweet step-wise.

Steps for Data Cleaning

1) Clear out HTML characters: A Lot of HTML entities like ' ,& ,< etc can be found in most of the data available on the web. We need to get rid of these from our data. You can do this in two ways:

• By using specific regular expressions or
• By using modules or packages available(htmlparser of python)

We will be using the module already available in python. 

Code: 

Output:

2) Encoding & Decoding Data: It is the process of converting information from simple understandable characters to complex symbols and vice versa. There are different forms of encoding &decoding like “UTF8″,”ascii” etc. available for text data. We should keep our data in a standard encoding format. The most common format is the UTF-8 format.

The given tweet is already in the UTF-8 format so we encoded it to ascii format and then decoded it to UTF-8 format to explain the process.

Code: 

 Output:

3) Removing URLs, Hashtags and Styles: In our text dataset, we can have hyperlinks, hashtags or styles like retweet text for twitter dataset etc. These provide no relevant information and can be removed. In hashtags, only the hash sign ‘#’ will be removed. For this, we will use the re library to perform regular expression operations.

Code: 

Output:

4) Contraction Replacement: The text data might contain apostrophe’s used for contractions. Example- “didn’t” for “did not” etc. This can change the sense of the word or sentence. Hence we need to replace these apostrophes with the standard lexicons. To do so we can have a dictionary which consists of the value with which the word needs to be replaced and use that.

Few of the contractions used are:-
n't --> not        'll --> will
's  --> is        'd  --> would
'm  --> am        've --> have
're --> are

Code: 

Output:

5) Split attached words:  Some words are joined together for example – “ForTheWin”. These need to be separated to be able to extract the meaning out of it. After splitting, it will be “For The Win”. 

Code: 

Output:

6 )Convert to lower case: Convert your text to lower case to avoid case sensitivity related issues.

Code: 

Output:

7) Slang lookup: There are many slang words which are used nowadays, and they can be found in the text data. So we need to replace them with their meanings. We can use a dictionary of slang words as we did for the contraction replacement, or we can create a file consisting of the slang words. Examples of slang words are:-

asap --> as soon as possible
b4   --> before
lol  --> laugh out loud
luv  --> love
wtg  --> way to go

We are using a file which consists of the words. You can download the file slang.txt. Source of this file was taken from here.

Code: 

Output:

8) Standardizing and Spell Check: There might be spelling errors in the text or it might not be in the correct format. For example – “drivng” for “driving” or “I misssss this” for “I miss this”. We can correct these by using the autocorrect library for python. There are other libraries available which you can use as well. First, you will have to install the library by using the command-

#install autocorrect library
 pip install autocorrect

Code: 

Output:

9) Remove Stopwords: Stop words are the words which occur frequently in the text but add no significant meaning to it. For this, we will be using the nltk library which consists of modules for pre-processing data. It provides us with a list of stop words. You can create your own stopwords list as well according to the use case.

First, make sure you have the nltk library installed. If not then download it using the command-

#install nltk library
 pip install nltk

Code: 

Output:

10) Remove Punctuations: Punctuations consists of !,<@#&$ etc. 

Code: 

Output:

These were some data cleaning techniques which we usually perform on the text data format. You can also perform some advanced data cleaning like grammar check etc.