With the help of nltk.tokenize.LineTokenizer() method, we are able to extract the tokens from string of sentences in the form of single line by using tokenize.LineTokenizer() method.
Syntax :
tokenize.LineTokenizer()
Return : Return the tokens of line from stream of sentences.
Example #1 :
In this example we can see that by using tokenize.LineTokenizer() method, we are able to extract the tokens from stream of sentences into small lines.
# import LineTokenizer() method from nltk from nltk.tokenize import LineTokenizer # Create a reference variable for Class LineTokenizer tk = LineTokenizer() # Create a string input gfg = "Lazyroar...$$&* \nis\n for Lazyroar" # Use tokenize method geek = tk.tokenize(gfg) print(geek) |
Output :
[‘Lazyroar…$$&* ‘, ‘is’, ‘ for Lazyroar’]
Example #2 :
# import LineTokenizer() method from nltk from nltk.tokenize import LineTokenizer # Create a reference variable for Class LineTokenizer tk = LineTokenizer(blanklines ='keep') # Create a string input gfg = "The price\n\n of burger \nin BurgerKing is Rs.36.\n" # Use tokenize method geek = tk.tokenize(gfg) print(geek) |
Output :
[‘The price’, ”, ‘ of burger ‘, ‘in BurgerKing is Rs.36.’]
