With the help of nltk.tokenize.word_tokenize() method, we are able to extract the tokens from string of characters by using tokenize.word_tokenize() method. It actually returns the syllables from a single word. A single word can contain one or two syllables.
Syntax :
tokenize.word_tokenize()
Return : Return the list of syllables of words.
Example #1 :
In this example we can see that by using tokenize.word_tokenize() method, we are able to extract the syllables from stream of words or sentences.
# import SyllableTokenizer() method from nltkfrom nltk import word_tokenize     # Create a reference variable for Class word_tokenizetk = SyllableTokenizer()     # Create a string inputgfg = "Antidisestablishmentarianism"     # Use tokenize methodgeek = tk.tokenize(gfg)     print(geek) |
Output :
[‘An’, ‘ti’, ‘dis’, ‘es’, ‘ta’, ‘blish’, ‘men’, ‘ta’, ‘ria’, ‘nism’]
Example #2 :
# import SyllableTokenizer() method from nltkfrom nltk.tokenize import word_tokenize     # Create a reference variable for Class word_tokenizetk = SyllableTokenizer()     # Create a string inputgfg = "Gametophyte"     # Use tokenize methodgeek = tk.tokenize(gfg)     print(geek) |
Output :
[‘Ga’, ‘me’, ‘to’, ‘phy’, ‘te’]
