Topic Modeling:
Topic modeling is a way of abstract modeling to discover the abstract ‘topics’ that occur in the collections of documents. The idea is that we will perform unsupervised classification on different documents, which find some natural groups in topics. We can answer the following question using topic modeling.
- What is the topic/main idea of the document?
- Given a document, can we find another document with a similar topic?
- How do topics field change over time?
Topic modeling can help in optimizing the search process. In this article, we will be discussing Latent Dirichlet Allocation, a topic modeling process.
Latent Dirichlet Allocation
Latent Dirichlet allocation is one of the most popular methods for performing topic modeling. Each document consists of various words and each topic can be associated with some words. The aim behind the LDA to find topics that the document belongs to, on the basis of words contains in it. It assumes that documents with similar topics will use a similar group of words. This enables the documents to map the probability distribution over latent topics and topics are probability distribution.
Setting up Generative Model:
- Let’s suppose we have D documents using the vocabulary of V-word types. Each document consists of an N-words token (can be removed or padded ). Now, we assume K topics, this required a K-dimensional vector that represents the topic distribution for the document.
- Each topic has a V-dimensional multinomial beta_k over words with a common symmetric prior.
- For each topic 1…k:
- Draw a multinomial over words
.
- Draw a multinomial over words
- For each document 1…d:
- Draw a multinomial over topics
- For each word
:
- Draw a topic
with ![Z_{N_d} \epsilon [1..K]](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20120%2027'%3E%3C/svg%3E "Rendered by QuickLaTeX.com")
- Draw a word W_{N_d} \sim Mult(\varphi).
- Draw a topic
- Draw a multinomial over topics
.
:
with ![Z_{N_d} \epsilon [1..K]](https://geeksforgeeks.org/wp-content/uploads/2023/10/quicklatex.com-c257571e9c0a9f3d0affc496421b1eb5_l3-1.png "Rendered by QuickLaTeX.com")
Graphical Model of LDA:
- In the above equation, the LHS represents the probability of generating the original document from the LDA machine.
- On the right side of the equation, there are 4 probability terms, the first two terms represent Dirichlet distribution and the other two represent the multinomial distribution. The first and third terms represent the distribution of topics but the second and fourth represent the word distribution. We will discuss the Dirichlet distribution first.
Dirichlet Distribution
- Dirichlet’s distribution can be defined as a probability density for a vector-valued input having the same characteristics as our multinomial parameter
. It has non-zero values such that:
. It has non-zero values such that:
- The Dirichlet distribution is parameterized by the vector α, which has the same number of elements K as the multinomial parameter θ.
- We can interpret p(θ|α) as answering the question “what is the probability density associated with multinomial distribution θ, given that our Dirichlet distribution has parameter α?”.
Dirichlet distribution
- Above is the visualization of the Dirichlet distribution, for our purpose, we can assume that corners/vertices represent the topics with words inside the triangle (the word is closer to the topic if it frequently relates with it. ) or vice-versa.
- This distribution can be extended to more than 3-dimensions. For 4-dimension we can use tetrahedron and for further dimension. We can use k-1 dimensions simplex.
Inference:
- The inference problem in LDA to compute the posterior of the hidden variables given the document and corpus parameter \alpha and \beta. That is to compute the P(
Example:
- Let’s consider we have two categories of topics, we have a word vector for each topic consisting of some words. Following are the words that represented different topics:
| words | P(words | topic =1) | P(words | topic =2) |
|---|---|---|
| Heart | 0.2 | 0 |
| Love | 0.2 | 0 |
| Soul | 0.2 | 0 |
| Tears | 0.2 | 0 |
| Joy | 0.2 | 0 |
| Scientific | 0 | 0.2 |
| Knowledge | 0 | 0.2 |
| Work | 0 | 0.2 |
| Research | 0 | 0.2 |
| Mathematics | 0 | 0.2 |
- Now, we have some document, and we scan some documents for these words:
| Words in Document | {P(topic=1), P(topic=2)} |
|---|---|
| MATHEMATICS KNOWLEDGE RESEARCH WORK MATHEMATICS RESEARCH WORK SCIENTIFIC MATHEMATICS WORK | {1,0} |
| SCIENTIFIC KNOWLEDGE MATHEMATICS SCIENTIFIC HEART LOVE TEARS KNOWLEDGE HEART | {0.25, 0.75} |
| MATHEMATICS HEART RESEARCH LOVE MATHEMATICS WORK TEARS SOUL KNOWLEDGE HEART | {0.5, 0.5} |
| WORK JOY SOUL TEARS MATHEMATICS TEARS LOVE LOVE LOVE SOUL | {0.75, 0.25} |
| TEARS LOVE JOY SOUL LOVE TEARS SOUL SOUL TEARS JOY | {1,0} |
- Now, we update the above words to topics matrix using the probabilities from document matrix below.
Implementation
In this implementation, we use scikit-learn and pyLDAvis. For datasets, we use yelp reviews datasets that can be found on the Yelp website.
Python3
# install pyldavis!pip install pyldavis# imports!pip install gensim pyLDAvis! python3 -m spacy download en_core_web_smimport pandas as pdimport numpy as npimport stringimport spacyimport nltkimport gensimfrom gensim import corporaimport matplotlib.pyplot as pltimport pyLDAvisimport pyLDAvis.gensim_modelsnltk.download('wordnet')from nltk.corpus import wordnet as wnnltk.download('stopwords')from nltk.corpus import stopwordsimport spacy.clispacy.cli.download("en_core_web_md")import en_core_web_md# fetch yelp review dataset and clean ityelp_review = pd.read_csv('/content/yelp.csv')yelp_review.head()# print number of document and topicsprint(len(yelp_review))print("Unique Business")print(len(yelp_review.groupby('business_id')))print("Unique User")print(len(yelp_review.groupby('user_id')))# clean the document and remove punctuationdef clean_text(text): delete_dict = {sp_char: '' for sp_char in string.punctuation} delete_dict[' '] =' ' table = str.maketrans(delete_dict) text1 = text.translate(table) textArr= text1.split() text2 = ' '.join([w for w in textArr if ( not w.isdigit() and ( not w.isdigit() and len(w)>3))]) return text2.lower()yelp_review['text'] = yelp_review['text'].apply(clean_text)yelp_review['Num_words_text'] = yelp_review['text'].apply(lambda x:len(str(x).split())) print('-------Reviews By Stars --------')print(yelp_review['stars'].value_counts())print(len(yelp_review))print('-------------------------')max_review_data_sentence_length = yelp_review['Num_words_text'].max()# print short review (mask = (yelp_review['Num_words_text'] < 100) & (yelp_review['Num_words_text'] >=20)df_short_reviews = yelp_review[mask]df_sampled = df_short_reviews.groupby('stars') .apply(lambda x: x.sample(n=100)).reset_index(drop = True)print('No of Short reviews')print(len(df_short_reviews))# function to remove stopwordsdef remove_stopwords(text): textArr = text.split(' ') rem_text = " ".join([i for i in textArr if i not in stop_words]) return rem_text# remove stopwords from the textstop_words = stopwords.words('english')df_sampled['text']=df_sampled['text'].apply(remove_stopwords)# perform Lemmatizationlp = en_core_web_md.load(disable=['parser', 'ner'])def lemmatization(texts,allowed_postags=['NOUN', 'ADJ']): output = [] for sent in texts: doc = nlp(sent) output.append([token.lemma_ for token in doc if token.pos_ in allowed_postags ]) return outputtext_list=df_sampled['text'].tolist()print(text_list[2])tokenized_reviews = lemmatization(text_list)print(tokenized_reviews[2])# convert to document term frequency:dictionary = corpora.Dictionary(tokenized_reviews)doc_term_matrix = [dictionary.doc2bow(rev) for rev in tokenized_reviews]# Creating the object for LDA model using gensim libraryLDA = gensim.models.ldamodel.LdaModel# Build LDA modellda_model = LDA(corpus=doc_term_matrix, id2word=dictionary, num_topics=10, random_state=100, chunksize=1000, passes=50,iterations=100)# print lda topics with respect to each word of documentlda_model.print_topics()# calculate perplexity and coherenceprint('\Perplexity: ', lda_model.log_perplexity(doc_term_matrix, total_docs=10000)) # calculate coherencecoherence_model_lda = CoherenceModel(model=lda_model, texts=tokenized_reviews, dictionary=dictionary , coherence='c_v')coherence_lda = coherence_model_lda.get_coherence()print('Coherence: ', coherence_lda)# Now, we use pyLDA vis to visualize itpyLDAvis.sklearn.prepare(lda_tf, dtm_tf, tf_vectorizer) |
Total reviews 10000 Unique Business 4174 Unique User 6403 -------------- -------Reviews by stars -------- 4 3526 5 3337 3 1461 2 927 1 749 Name: stars, dtype: int64 10000 ------------------------- No of Short reviews 6276 ------------------------- # review and tokenized version decided completely write place three times tried closed website posted hours open wants drive suburbs youd better call first place cannot trusted wasted time spent hungry minutes walking disappointed vitamin fail said ['place', 'time', 'closed', 'website', 'hour', 'open', 'drive', 'suburb', 'first', 'place', 'time', 'hungry', 'minute', 'vitamin'] --------------------------- # LDA print topics [(0, '0.015*"food" + 0.013*"good" + 0.010*"gelato" + 0.008*"sandwich" + 0.008*"chocolate" + 0.005*"wife" + 0.005*"next" + 0.005*"bad" + 0.005*"night" + 0.005*"sauce"'), (1, '0.030*"food" + 0.021*"great" + 0.019*"place" + 0.019*"good" + 0.016*"service" + 0.011*"time" + 0.011*"nice" + 0.008*"lunch" + 0.008*"dish" + 0.007*"staff"'), (2, '0.023*"food" + 0.023*"good" + 0.018*"place" + 0.014*"great" + 0.009*"star" + 0.009*"service" + 0.008*"store" + 0.007*"salad" + 0.007*"well" + 0.006*"pizza"'), (3, '0.035*"good" + 0.025*"place" + 0.023*"food" + 0.020*"time" + 0.015*"service" + 0.012*"great" + 0.009*"friend" + 0.008*"table" + 0.008*"chicken" + 0.007*"hour"'), (4, '0.020*"food" + 0.019*"time" + 0.012*"good" + 0.009*"restaurant" + 0.009*"great" + 0.008*"service" + 0.007*"order" + 0.006*"small" + 0.006*"hour" + 0.006*"next"'), (5, '0.012*"drink" + 0.009*"star" + 0.006*"worth" + 0.006*"place" + 0.006*"friend" + 0.005*"great" + 0.005*"kid" + 0.005*"drive" + 0.005*"simple" + 0.005*"experience"'), (6, '0.024*"place" + 0.015*"time" + 0.012*"food" + 0.011*"price" + 0.009*"good" + 0.009*"great" + 0.009*"kid" + 0.008*"staff" + 0.008*"nice" + 0.007*"happy"'), (7, '0.028*"place" + 0.019*"service" + 0.015*"good" + 0.014*"pizza" + 0.014*"time" + 0.013*"food" + 0.013*"great" + 0.011*"well" + 0.009*"order" + 0.007*"price"'), (8, '0.032*"food" + 0.026*"good" + 0.026*"place" + 0.015*"great" + 0.009*"service" + 0.008*"time" + 0.006*"price" + 0.006*"meal" + 0.006*"shop" + 0.006*"coffee"'), (9, '0.020*"food" + 0.014*"place" + 0.011*"meat" + 0.010*"line" + 0.009*"good" + 0.009*"minute" + 0.008*"time" + 0.008*"chicken" + 0.008*"wing" + 0.007*"hour"')] ------------------------------
PyLDAvis Visualization
