Agglomerative clustering is a type of Hierarchical clustering that works in a bottom-up fashion. Metrics play a key role in determining the performance of clustering algorithms. Choosing the right metric helps the clustering algorithm to perform better. This article discusses agglomerative clustering with different metrics in Scikit Learn.
Scikit learn provides various metrics for agglomerative clusterings like Euclidean, L1, L2, Manhattan, Cosine, and Precomputed. Let us take a look at each of these metrics in detail:
- Euclidean Distance: It measures the straight line distance between 2 points in space.
- Manhattan Distance: It measures the sum of absolute differences between 2 points/vectors in all dimensions.
- Cosine Similarity: It measures the angular cosine similarity between 2 vectors.
Agglomerative Clustering
Two kinds of datasets are considered, low dimensional and high dimensional. High-dimensional data has more features than data records. For low-dimensional data, the customer shopping dataset is considered. This dataset has 5 features namely, Customer Id, Gender, Age, Annual Income (k$), and Spending Score (1-100). We will form clusters based on Annual Income (k$) and Spending Score (1-100) as scatter plots between other features do not show promising patterns. For high dimensional data, the forest cover type dataset is considered that has 55 features and 5,81,012 data records. However, to convert this dataset into a high-dimensional dataset only 50 records are considered for clustering.
Python3
import numpy as npimport pandas as pdimport matplotlib.pyplot as pltfrom sklearn import preprocessingimport scipy.cluster.hierarchy as schfrom sklearn.cluster import AgglomerativeClusteringfrom sklearn.metrics import silhouette_score |
Load the datasets using the panda’s data frame.
Python3
# low dimensional datadf = pd.read_csv("Mall_Customers.csv")# high dimensional datahd_df = pd.read_csv("covtype.csv")hd_data = hd_df.head(50) |
Let’s take a look at the first five rows of the low-dimensional dataset.
Python3
df.head() |
Output:
First five rows of the dataset
Function for implementing agglomerative clustering using different metrics.
Python3
# generates clusters using agglomerative clustering algorithm# uses average linkage method with different affinity (metric) valuesdef agg_clustering(data, num_clusters, metric): cluster_model = AgglomerativeClustering(n_clusters=num_clusters, affinity=metric, linkage='average') clusters = cluster_model.fit_predict(data) score = silhouette_score(data, cluster_model.labels_, metric='euclidean') return clusters, score |
Scaling the data and invoking the above function.
Python3
X = df.iloc[:, [3, 4]].valuesscaler = preprocessing.StandardScaler()scaled_X = scaler.fit_transform(X)y_euclidean, euclidean_score = agg_clustering(scaled_X, 5, 'euclidean')y_l1, l1_score = agg_clustering(scaled_X, 5, 'l1')y_l2, l2_score = agg_clustering(scaled_X, 5, 'l2')y_manhattan, manhattan_score = agg_clustering(scaled_X, 5, 'manhattan')y_cosine, cosine_score = agg_clustering(scaled_X, 5, 'cosine') |
Let’s plot the clusters.
Python3
# plots the clusters formed using different metrics# y contains cluster labels for each data recorddef plot_clusters(data, y, metric): plt.scatter(data[y==0, 0], data[y==0, 1], s=100, c='red', label ='Cluster 1') plt.scatter(data[y==1, 0], data[y==1, 1], s=100, c='blue', label ='Cluster 2') plt.scatter(data[y==2, 0], data[y==2, 1], s=100, c='green', label ='Cluster 3') plt.scatter(data[y==3, 0], data[y==3, 1], s=100, c='purple', label ='Cluster 4') plt.scatter(data[y==4, 0], data[y==4, 1], s=100, c='orange', label ='Cluster 5') plt.title(f'Clusters of Customers (using {metric} distance metric)') plt.xlabel('Annual Income(k$)') plt.ylabel('Spending Score(1-100)') plt.legend() plt.show() |
Python3
plot_clusters(X, y_euclidean, 'euclidean')plot_clusters(X, y_l1, 'l1')plot_clusters(X, y_l2, 'l2')plot_clusters(X, y_manhattan, 'manhattan')plot_clusters(X, y_cosine, 'cosine') |
Output:
It is a bit difficult to figure out the differences between the clusters formed using different metrics just by looking at the above plots. Hence, we make use of silhouette scores to compare the above clusters.
Python3
silhouette_scores = {'euclidean': euclidean_score, 'l1': l1_score, 'l2': l2_score, 'manhattan': manhattan_score, 'cosine': cosine_score}plt.bar(list(silhouette_scores.keys()), list(silhouette_scores.values()), width=0.4) |
Output:
Comparison of different metrics for clusters formed
We can observe that Manhattan or L1 metric and Euclidean or L2 metric give good silhouette scores. However, the cosine metric performs poorly in this case. Cosine metric gives a poor performance with low dimensional data and should be avoided. Also, data must be scaled before using Euclidean or L2 distance metric.
Similarly, clusters are formed for high dimensional data after scaling the features.
Python3
numerical_features = ["Elevation", "Aspect", "Slope", "Horizontal_Distance_To_Hydrology", "Vertical_Distance_To_Hydrology", "Horizontal_Distance_To_Roadways", "Hillshade_9am", "Hillshade_Noon", "Hillshade_3pm", "Horizontal_Distance_To_Fire_Points"]hd_data[numerical_features] = scaler.fit_transform(hd_data[numerical_features])y_hd_euclidean, euclidean_score_hd = agg_clustering(hd_data, 5, 'euclidean')y_hd_l1, l1_score_hd = agg_clustering(hd_data, 5, 'l1')y_hd_l2, l2_score_hd = agg_clustering(hd_data, 5, 'l2')y_hd_manhattan, manhattan_score_hd = agg_clustering(hd_data, 5, 'manhattan')y_hd_cosine, cosine_score_hd = agg_clustering(hd_data, 5, 'cosine') |
Let’s take a look at the silhouette scores.
Python3
silhouette_scores_hd = {'euclidean': euclidean_score_hd, 'l1': l1_score_hd, 'l2': l2_score_hd, 'manhattan': manhattan_score_hd, 'cosine': cosine_score_hd}plt.bar(list(silhouette_scores_hd.keys()), list(silhouette_scores_hd.values()), width=0.4) |
Output:
Comparison of different metrics for clusters formed
In this case, the cosine metric performs pretty well. Hence, cosine is generally used with high-dimensional data. Manhattan or L1 metric also performs well on high dimensional data. However, the Euclidean metric does not perform very well on high-dimensional data due to the “curse of dimensionality”.
