According to the paper, previous methods to interpret neural network suffer from several disadvantages: (a) of low-resolution, (b) not class-discriminative and (c) limited to a specific type of network. Grad-CAM is an improvement of CAM. Since the information is abstracted layer by layer and the fully-connectedly layer cannot maintain the spatial information, they select the last convolutional layer for analysis.
For a class and a filter
, they compute the contribution of this filter to the class classification as:
where the class score for class
(before softmax) and
is an element of the last convoluted output. The final Grad-CAM a weighted sum across all channels:
ReLU is employed to keep the positive influence, which means the remain activations have a positive relationship with the given class label. By reversing the positive and negative of ReLU layer, the Gram-CAM can localize the counterfactual areas, based on which the neural network "believes" current label is wrong. Besides, to have a higher resolution, they combine Grad-CAM with Guided Backpropagation to refine the result.
Although the model is very simple, they conduct extensive experiments and interpreted neural networks in a variety of tasks. I strongly suggest you read the experiments yourselves.
[DR020]: TBA