Introduction to NLTK: Tokenization, Stemming, Lemmatization, POS Tagging

Natural Language Processing (NLP) is a broad subfield of Artificial Intelligence that deals with processing and predicting textual data. While Python is known for the extensive libraries it offers for various ML/DL tasks – it certainly doesn’t fail to do so for NLP tasks.

Natural Language Toolkit (NLTK) is one of the vastest Python libraries for performing various Natural Language Processing tasks. From rudimentary tasks such as text pre-processing to tasks like vectorized representation of text – NLTK’s API has covered everything. In this article, we will accustom ourselves to the basics of NLTK and perform some crucial NLP tasks: Tokenization, Stemming, Lemmatization, and POS Tagging.

Natural Language Toolkit (NLTK)

As discussed earlier, NLTK is Python’s API library for performing an array of tasks in human language. It can perform a variety of operations on textual data, such as classification, tokenization, stemming, tagging, parsing, semantic reasoning, etc.

Installation:
NLTK can be installed simply using pip or by running the following code.

pip install nltk

Accessing Additional Resources:
To incorporate the usage of additional resources, such as recourses of languages other than English – you can run the following in a python script. It has to be done only once when you are running it for the first time in your system.

Now, having installed NLTK successfully in our system, let’s perform some basic operations on text data using NLTK. 

Tokenization

Tokenization refers to break down the text into smaller units. It entails splitting paragraphs into sentences and sentences into words. It is one of the initial steps of any NLP pipeline. Let us have a look at the two major kinds of tokenization that NLTK provides:

Work Tokenization

It involves breaking down the text into words.

 "I study Machine Learning on Lazyroar." will be word-tokenized as
  ['I', 'study', 'Machine', 'Learning', 'on', 'Lazyroar', '.']. 

Sentence Tokenization

It involves breaking down the text into individual sentences. 

Example:
"I study Machine Learning on Lazyroar. Currently, I'm studying NLP"
 will be sentence-tokenized as 
 ['I study Machine Learning on Lazyroar.', 'Currently, I'm studying NLP.']

In Python, both these tokenizations can be implemented in NLTK as follows:

Output:

['Lazyroar', 'is', 'a', 'great', 'learning', 'platform', '.',
'It', 'is', 'one', 'of', 'the', 'best', 'for', 'Computer', 'Science', 'students', '.']
['Lazyroar is a great learning platform.',
 'It is one of the best for Computer Science students.']

Stemming and Lemmatization 

When working with Natural Language, we are not much interested in the form of words – rather, we are concerned with the meaning that the words intend to convey. Thus, we try to map every word of the language to its root/base form. This process is called canonicalization. 

E.g. The words ‘play’, ‘plays’, ‘played’, and ‘playing’  convey the same action – hence, we can map them all to their base form i.e. ‘play’.

Now, there are two widely used canonicalization techniques: Stemming and Lemmatization.

Stemming 

Stemming generates the base word from the inflected word by removing the affixes of the word. It has a set of pre-defined rules that govern the dropping of these affixes. It must be noted that stemmers might not always result in semantically meaningful base words.  Stemmers are faster and computationally less expensive than lemmatizers. 

In the following code, we will be stemming words using Porter Stemmer – one of the most widely used stemmers:

Output:

play
play
play
play

We can see that all the variations of the word ‘play’ have been reduced to the same word  – ‘play’. In this case, the output is a meaningful word, ‘play’. However, this is not always the case. Let us take an example. 

Please note that these groups are stored in the lemmatizer; there is no removal of affixes as in the case of a stemmer.

Output:

commun

The stemmer reduces the word ‘communication’ to a base word ‘commun’ which is meaningless in itself.

Lemmatization

Lemmatization involves grouping together the inflected forms of the same word. This way, we can reach out to the base form of any word which will be meaningful in nature. The base from here is called the Lemma.

Lemmatizers are slower and computationally more expensive than stemmers. 

Example:
'play', 'plays', 'played', and 'playing' have 'play' as the lemma. 

In Python, both these tokenizations can be implemented in NLTK as follows:

Output:

play
play
play
play

Please note that in lemmatizers, we need to pass the Part of Speech of the word along with the word as a function argument.

Also, stemmers always result in meaningful base words. Let us take the same example as we took in the case for stemmers.

Output:

Communication

Part of Speech Tagging

Part of Speech (POS) tagging refers to assigning each word of a sentence to its part of speech. It is significant as it helps give a better syntactic overview of a sentence. 

Example:
"Lazyroar is a Computer Science platform."
Let's see how NLTK's POS tagger will tag this sentence.

In Python, both these tokenizations can be implemented in NLTK as follows:

Output: 

[('Lazyroar', 'NNP'),
 ('is', 'VBZ'),
 ('a', 'DT'),
 ('Computer', 'NNP'),
 ('Science', 'NNP'),
 ('platform', 'NN'),
 ('.', '.')]

One can refer to the official documentation for the meaning of the tags assigned by the pos_tag function.