This article was published as a part of the Data Science Blogathon
Whenever we saw our childhood diaries, we always wanted to recognize all the places we had visited at least once in our childhood. But we don’t remember their names. Most Indian people visit various temples but they always forget the names of these temples. Sometimes this also puts us in a shy moment where we can’t even tell our peers, oh yeah even I visit this place. Even we forget, Who had built this monument? The solution to these problems is Landmark Detection, which helps us memorize the names of these places. Do you know the working of the landmarks detection? In this blog, we are going to create a deep learning project on landmark detection with Python.
What is Landmark Detection?
The mechanism of detecting the famous human-made sculptures, buildings, and monuments inside an image is defined as Landmark Detection. You can simply compare it with the famous application of Google known as Google Landmark Detection, which is used by Google Maps.
At the end of this blog, you will be able to create your own landmark detector like Google using the Keras library of Deep learning.
Dataset
Our task is to build neural networks to recognize the landmarks inside the images using the Python programming language. The most critical task for any project is to choose an appropriate dataset for model training. We choose Kaggle’s dataset for our deep learning project. The dataset consists of image URLs that are publicly available online. The dataset contains three CSV files including test images, training images, and index images. The test images are for the purpose of image recognition and landmark labelling predicted by the deep learning model. The training images are already defined and associated with landmark labels that are used to train models for accurate landmark recognition. The use of index images is found in the image retrieval task. You can download the dataset from https://www.kaggle.com/google/google-landmarks-dataset.
Google Landmark Detection with Keras
Step One: Import the libraries
To begin with this task, our first step is to import all the required python libraries that we need to create our Deep Learning model for the sake of landmark detection:
import numpy as np import pandas as pd import keras import cv2 from matplotlib import pyplot as plt import os import random from PIL import Image
Step Two: Import dataset
So after importing libraries our next task is to import the landmark datasets containing images:
samples = 20000
df = pd.read_csv("train.csv")
df = df.loc[:samples,:]
num_classes = len(df["landmark_id"].unique())
num_data = len(df)
Now we’ll check the size of the training data and the number of unique classes present in the training data:
print("Size of training data:", df.shape)
print("Number of unique classes:", num_classes)
Output
Size of training data: (20001, 2)
Number of unique classes: 1020
The number of training sets available in the training samples is 20,001, they belong to around 1,020 different classes, which gives us on an average 19.6 images per class, however, this distribution might not be the case, so let’s observe the distribution of samples by class:
data = pd.DataFrame(df['landmark_id'].value_counts()) #index the data frame data.reset_index(inplace=True) data.columns=['landmark_id','count'] print(data.head(10)) print(data.tail(10))
Output
landmark_id count
0 1924 944
1 27 504
2 454 254
3 1346 244
4 1127 201
5 870 193
6 2185 177
7 1101 162
8 389 140
9 219 139
landmark_id count
1010 499 2
1011 1942 2
1012 875 2
1013 2297 2
1014 611 2
1015 1449 2
1016 1838 2
1017 604 2
1018 374 2
1019 991 2
As we can see, the range of 10 most frequent landmarks from 139 data points to 944 data points while the last 10 landmarks have only 2 data points.
print(data['count'].describe())#statistical data for the distribution
plt.hist(data['count'],100,range = (0,944),label = 'test')#Histogram of the distribution
plt.xlabel("Amount of images")
plt.ylabel("Occurences")
Output
count 1020.000000
mean 19.608824
std 41.653684
min 2.000000
25% 5.000000
50% 9.000000
75% 21.000000
max 944.000000
Name: count, dtype: float64
Text(0, 0.5, ‘Occurences’)
After observing the above histogram, we can conclude that the huge majority of classes are not associated with a lot of images.
print("Amount of classes with less than or equal to five datapoints:", (data['count'].between(0,5)).sum())
print("Amount of classes between five and 10 datapoints:", (data['count'].between(5,10)).sum())
n = plt.hist(df["landmark_id"],bins=df["landmark_id"].unique())
freq_info = n[0]
plt.xlim(0,data['landmark_id'].max())
plt.ylim(0,data['count'].max())
plt.xlabel('Landmark ID')
plt.ylabel('Number of images')
Output
Amount of classes with less than or equal to five data points: 322
Amount of classes between five and 10 data points: 342
Text(0, 0.5, 'Number of images')
The above graph depicts that around 50% of the 1020 classes have fewer than 10 images, which can create a problem while training a classifier.
In terms of the number of images they have, there are some “outliers”, which represents that there might have a higher chance of getting a correct “guess” with the highest amount in these classes.
Training Model
Now, our task is to train the Deep Learning model using the Python programming language to detect the landmark which will work the same as the Google landmark detection model.
from sklearn.preprocessing import LabelEncoder
lencoder = LabelEncoder()
lencoder.fit(df["landmark_id"])
def encode_label(lbl):
return lencoder.transform(lbl)
def decode_label(lbl):
return lencoder.inverse_transform(lbl)
def get_image_from_number(num):
fname, label = df.loc[num,:]
fname = fname + ".jpg"
f1 = fname[0]
f2 = fname[1]
f3 = fname[2]
path = os.path.join(f1,f2,f3,fname)
im = cv2.imread(os.path.join(base_path,path))
return im, label
print("4 sample images from random classes:")
fig=plt.figure(figsize=(16, 16))
for i in range(1,5):
a = random.choices(os.listdir(base_path), k=3)
folder = base_path+'/'+a[0]+'/'+a[1]+'/'+a[2]
random_img = random.choice(os.listdir(folder))
img = np.array(Image.open(folder+'/'+random_img))
fig.add_subplot(1, 4, i)
plt.imshow(img)
plt.axis('off')
plt.show()
from keras.applications import VGG19
from keras.layers import *
from keras import Sequential
# Parameters
# learning_rate = 0.0001
# decay_speed = 1e-6
# momentum = 0.09
# loss_function = "sparse_categorical_crossentropy"
source_model = VGG19(weights=None)
#new_layer = Dense(num_classes, activation=activations.softmax, name='prediction')
drop_layer = Dropout(0.5)
drop_layer2 = Dropout(0.5)
model = Sequential()
for layer in source_model.layers[:-1]: # go through until last layer
if layer == source_model.layers[-25]:
model.add(BatchNormalization())
model.add(layer)
# if layer == source_model.layers[-3]:
# model.add(drop_layer)
# model.add(drop_layer2)
model.add(Dense(num_classes, activation="softmax"))
model.summary()
optim1 = keras.optimizers.RMSprop(learning_rate = 0.0001, momentum = 0.09)
optim2 = keras.optimizers.Adam(learning_rate=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-07)
model.compile(optimizer=optim1,
loss="sparse_categorical_crossentropy",
metrics=["accuracy"])
sgd = SGD(lr=learning_rate, decay=decay_speed, momentum=momentum, nesterov=True)
rms = keras.optimizers.RMSprop(lr=learning_rate, momentum=momentum)
model.compile(optimizer=rms,
loss=loss_function,
metrics=["accuracy"])
print("Model compiled! n")
Output
Model: “sequential”
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
batch_normalization (BatchNo (None, 224, 224, 3) 12
_________________________________________________________________
block1_conv1 (Conv2D) (None, 224, 224, 64) 1792
_________________________________________________________________
block1_conv2 (Conv2D) (None, 224, 224, 64) 36928
_________________________________________________________________
block1_pool (MaxPooling2D) (None, 112, 112, 64) 0
_________________________________________________________________
block2_conv1 (Conv2D) (None, 112, 112, 128) 73856
_________________________________________________________________
block2_conv2 (Conv2D) (None, 112, 112, 128) 147584
_________________________________________________________________
block2_pool (MaxPooling2D) (None, 56, 56, 128) 0
_________________________________________________________________
block3_conv1 (Conv2D) (None, 56, 56, 256) 295168
_________________________________________________________________
block3_conv2 (Conv2D) (None, 56, 56, 256) 590080
_________________________________________________________________
block3_conv3 (Conv2D) (None, 56, 56, 256) 590080
_________________________________________________________________
block3_conv4 (Conv2D) (None, 56, 56, 256) 590080
_________________________________________________________________
block3_pool (MaxPooling2D) (None, 28, 28, 256) 0
_________________________________________________________________
block4_conv1 (Conv2D) (None, 28, 28, 512) 1180160
_________________________________________________________________
block4_conv2 (Conv2D) (None, 28, 28, 512) 2359808
_________________________________________________________________
block4_conv3 (Conv2D) (None, 28, 28, 512) 2359808
_________________________________________________________________
block4_conv4 (Conv2D) (None, 28, 28, 512) 2359808
_________________________________________________________________
block4_pool (MaxPooling2D) (None, 14, 14, 512) 0
_________________________________________________________________
block5_conv1 (Conv2D) (None, 14, 14, 512) 2359808
_________________________________________________________________
block5_conv2 (Conv2D) (None, 14, 14, 512) 2359808
_________________________________________________________________
block5_conv3 (Conv2D) (None, 14, 14, 512) 2359808
_________________________________________________________________
block5_conv4 (Conv2D) (None, 14, 14, 512) 2359808
_________________________________________________________________
block5_pool (MaxPooling2D) (None, 7, 7, 512) 0
_________________________________________________________________
flatten (Flatten) (None, 25088) 0
_________________________________________________________________
fc1 (Dense) (None, 4096) 102764544
_________________________________________________________________
fc2 (Dense) (None, 4096) 16781312
_________________________________________________________________
dense (Dense) (None, 1020) 4178940
=================================================================
Total params: 143,749,192
Trainable params: 143,749,186
Non-trainable params: 6
#Function used to process the data, fitted into a data generator.
def get_image_from_number(num, df):
fname, label = df.iloc[num,:]
fname = fname + ".jpg"
f1 = fname[0]
f2 = fname[1]
f3 = fname[2]
path = os.path.join(f1,f2,f3,fname)
im = cv2.imread(os.path.join(base_path,path))
return im, label
def image_reshape(im, target_size):
return cv2.resize(im, target_size)
def get_batch(dataframe,start, batch_size):
image_array = []
label_array = []
end_img = start+batch_size
if end_img > len(dataframe):
end_img = len(dataframe)
for idx in range(start, end_img):
n = idx
im, label = get_image_from_number(n, dataframe)
im = image_reshape(im, (224, 224)) / 255.0
image_array.append(im)
label_array.append(label)
label_array = encode_label(label_array)
return np.array(image_array), np.array(label_array)
batch_size = 16
epoch_shuffle = True
weight_classes = True
epochs = 15
# Split train data up into 80% and 20% validation
train, validate = np.split(df.sample(frac=1), [int(.8*len(df))])
print("Training on:", len(train), "samples")
print("Validation on:", len(validate), "samples")
for e in range(epochs):
print("Epoch: ", str(e+1) + "/" + str(epochs))
if epoch_shuffle:
train = train.sample(frac = 1)
for it in range(int(np.ceil(len(train)/batch_size))):
X_train, y_train = get_batch(train, it*batch_size, batch_size)
model.train_on_batch(X_train, y_train)
model.save("Model.h5")
Output
Training on: 16000 samples
Validation on: 4001 samples
Epoch: 1/15
Epoch: 2/15
Epoch: 3/15
Epoch: 4/15
Epoch: 5/15
Epoch: 6/15
Epoch: 7/15
Epoch: 8/15
Epoch: 9/15
Epoch: 10/15
Epoch: 11/15
Epoch: 12/15
Epoch: 13/15
Epoch: 14/15
Epoch: 15/15
Now we are done with successful model training. Our next step is to test the model, let’s see the results of our trained landmark detection model:
### Test on the training set
batch_size = 16
errors = 0
good_preds = []
bad_preds = []
for it in range(int(np.ceil(len(validate)/batch_size))):
X_train, y_train = get_batch(validate, it*batch_size, batch_size)
result = model.predict(X_train)
cla = np.argmax(result, axis=1)
for idx, res in enumerate(result):
print("Class:", cla[idx], "- Confidence:", np.round(res[cla[idx]],2), "- GT:", y_train[idx])
if cla[idx] != y_train[idx]:
errors = errors + 1
bad_preds.append([batch_size*it + idx, cla[idx], res[cla[idx]]])
else:
good_preds.append([batch_size*it + idx, cla[idx], res[cla[idx]]])
print("Errors: ", errors, "Acc:", np.round(100*(len(validate)-errors)/len(validate),2))
#Good predictions
good_preds = np.array(good_preds)
good_preds = np.array(sorted(good_preds, key = lambda x: x[2], reverse=True))
fig=plt.figure(figsize=(16, 16))
for i in range(1,6):
n = int(good_preds[i,0])
img, lbl = get_image_from_number(n, validate)
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
fig.add_subplot(1, 5, i)
plt.imshow(img)
lbl2 = np.array(int(good_preds[i,1])).reshape(1,1)
sample_cnt = list(df.landmark_id).count(lbl)
plt.title("Label: " + str(lbl) + "nClassified as: " + str(decode_label(lbl2)) + "nSamples in class " + str(lbl) + ": " + str(sample_cnt))
plt.axis('off')
plt.show()
Conclusion
You can see the model output, how monument images are classified depending on the classes and labels. It uses the Keras library of deep learning to create a convolutional network which in turn trains the model. Hope you linked the blog, for any doubt please hit the box.
