Convolutional Neural Networks (ConvNets or CNNs) are a category of Neural Networks that have proven very effective in areas such as image recognition and classification. Understanding the behavior of ConvNets can be a complex task, especially when working with large image datasets. To help with this, we can visualize the activations of individual neurons in ConvNets, which can provide us with insights into the features that the model is learning. In this article, we will explore how to visualize ConvNets using PyTorch, a popular deep-learning framework.
ConvNets: Convolutional Neural Networks are a type of neural network that is designed to work with image data. ConvNets are composed of several layers, including convolutional layers and pooling layers, which extract and reduce features from the input image.
Example 1 :
Step 1:
Install the necessary libraries
sudo apt install graphviz # [Ubuntu] winget install graphviz # [Windows] sudo port install graphviz # [Mac] pip install torchviz
Step 2:
Import the necessary libraries, including PyTorch, Numpy, and Matplotlib.
Python3
import torch import torchvision import torchvision.transforms as transforms import torch.nn as nn import torch.nn.functional as F from PIL import Image from torch.autograd import Variable import torch.optim as optim import matplotlib.pyplot as plt import numpy as np from torchviz import make_dot |
Step 3:
Define the simple Convolution Neural Network model :
Python3
class convents(nn.Module): def __init__( self ): super ().__init__() self .conv1 = nn.Conv2d( 3 , 32 , kernel_size = 4 , padding = 1 ) self .conv2 = nn.Conv2d( 32 , 64 , kernel_size = 3 , padding = 1 ) self .pool = nn.MaxPool2d( 2 , 2 ) self .dropout1 = nn.Dropout2d( 0.25 ) self .dropout2 = nn.Dropout2d( 0.5 ) self .fc1 = nn.Linear( 64 * 7 * 7 , 128 ) self .fc2 = nn.Linear( 128 , 10 ) def forward( self , x): x = self .pool(F.relu( self .conv1(x))) x = self .dropout1(x) x = self .pool(F.relu( self .conv2(x))) x = self .dropout2(x) x = x.view( - 1 , 64 * 7 * 7 ) x = F.relu( self .fc1(x)) x = self .fc2(x) return x net = convents() # Loss functions and optimizer criterion = nn.CrossEntropyLoss() optimizer = optim.SGD(net.parameters(), lr = 0.001 , momentum = 0.9 ) net |
Output:
convents( (conv1): Conv2d(3, 32, kernel_size=(4, 4), stride=(1, 1), padding=(1, 1)) (conv2): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False) (dropout1): Dropout2d(p=0.25, inplace=False) (dropout2): Dropout2d(p=0.5, inplace=False) (fc1): Linear(in_features=3136, out_features=128, bias=True) (fc2): Linear(in_features=128, out_features=10, bias=True) )
In the first conv1, the input will be 3 channels and the output will be 32 convolved images
Step 4:
Load the image
Python3
img = Image. open ( "Ganesh.jpg" ) img |
Output:
Step 5:
Transform the image to Pytorch Tensor
Python3
# transform the image to pytorch tensor transform = transforms.Compose([ transforms.ToTensor(), ]) img = transform(img) img.shape |
Output:
torch.Size([3, 788, 716])
Step 6:
Get the first convolutional layer from our created model, and apply it to the image.
Python3
# Get activations of the first convolutional layer conv1 = net.conv1 print ( 'First convolution Layer :' ,conv1) # APPLY THE FIRST Convolutional layer to the image y = conv1(img) print ( 'Output Shape :' ,y.shape) |
Output:
First convolution Layer : Conv2d(3, 32, kernel_size=(4, 4), stride=(1, 1), padding=(1, 1)) Output Shape : torch.Size([32, 787, 715])
We can visualize the convolution by torchviz make_dot functions. To use this graphviz should be installed in the system
Python3
make_dot(y.mean(), params = dict (conv1.named_parameters())) |
Output:
Step 7:
Get the weights of the first convolutional layer of the model.
Python3
# Get weights of the first convolutional layer weights = conv1.weight.detach().numpy() weights.shape |
Output:
(32, 3, 4, 4)
Step 8:
Plot the Gray Scale original, Convolved, and Convolution weight with matplotlib
Python3
plt.figure(figsize = ( 12 , 5 )) # Plot the original grayscale image plt.subplot( 1 , 3 , 1 ) plt.imshow(img[ 0 ],cmap = 'gray' ) plt.title( 'Original' ) plt.axis( 'off' ) img = Variable(img.unsqueeze( 0 ), requires_grad = True ) #Plot the convolved grayscale image # Squeeze tensor to numpy image img_conv1 = y.detach().numpy() img_conv1 = np.squeeze(img_conv1) plt.subplot( 1 , 3 , 2 ) plt.imshow(img_conv1[ 0 ], cmap = 'gray' ) plt.axis( 'off' ) plt.title( 'After convolutions' ) # Plot the weights of the first convolutional layer # Squeeze tensor to numpy image weights = np.squeeze(weights) plt.subplot( 1 , 3 , 3 ) plt.imshow(weights[ 0 , 0 ,:,:], cmap = 'gray' ) plt.axis( 'off' ) plt.title( 'First convolutions weights' ) plt.show() |
Output:
Example 2:
Let’s try a pre-trained model
- Import the necessary packages
- Load a pre-trained VGG16 model
- Load an image and transformed it
- Plot Original & convolved Grayscale image with First convolutions weights
- visualize the convolution graph by torchviz
Input Image:
Code Implementations
Python3
# Import necessary libraries import torch import torchvision import torchvision.transforms as transforms import torchvision.models as models import torch.nn as nn import torch.nn.functional as F from PIL import Image from torch.autograd import Variable import torch.optim as optim import matplotlib.pyplot as plt import numpy as np from torchviz import make_dot # Load a pre-trained VGG16 model model = models.vgg16(pretrained = True ) # Transformations transform = transforms.Compose([ transforms.Resize( 256 ), transforms.CenterCrop( 224 ), transforms.ToTensor(), transforms.Normalize(mean = [ 0.485 , 0.456 , 0.406 ], std = [ 0.229 , 0.224 , 0.225 ]) ]) # Load an example image and transformed it img = transform(Image. open ( "GFG.img" )) #img = Variable(img.unsqueeze(0), requires_grad=True) plt.figure(figsize = ( 12 , 5 )) # Plot the original grayscale image plt.subplot( 1 , 3 , 1 ) plt.imshow(img[ 0 ],cmap = 'gray' ) plt.axis( 'off' ) plt.title( 'Original' ) img = Variable(img.unsqueeze( 0 ), requires_grad = True ) # Get activations of the first convolutional layer conv1 = model.features[ 0 ] # APPLY THE FIRST Convolutional layer y = conv1(img) img_conv1 = y.detach().numpy() img_conv1 = np.squeeze(img_conv1) # Plot the convolved grayscale image plt.subplot( 1 , 3 , 2 ) plt.imshow(img_conv1[ 0 ], cmap = 'gray' ) plt.axis( 'off' ) plt.title( 'After convolutions' ) # Get weights of the first convolutional layer weights = conv1.weight.detach().numpy() weights = np.squeeze(weights) # Plot the weights of the first convolutional layer plt.subplot( 1 , 3 , 3 ) plt.imshow(weights[ 0 , 0 ,:,:], cmap = 'gray' ) plt.axis( 'off' ) plt.title( 'First convolutions weights' ) plt.show() make_dot(y.mean(), params = dict (conv1.named_parameters())) |
Output:
Other ways to visualize ConvNets in PyTorch include plotting the weights of the convolutional layers, visualizing the filters in the convolutional layers, and plotting the feature maps produced by the activations of the convolutional layers. The choice of visualization will depend on the specific goals and questions you have about your ConvNet model.
In conclusion, visualizing the activations of ConvNets in PyTorch can provide valuable insights into the features that the model is learning and can help with understanding the behavior of the model.