Occlusion Sensitivity
Description
Occlusion sensitivity shows which parts of an input a model's prediction is sensitive to. A region of the input, such as a square patch of an image, is hidden by replacing it with a neutral value like grey, and the change in the model's output is recorded, usually the probability of the class of interest. Sliding the patch across the whole input gives a map of how much the output changes when each region is hidden. A large drop marks a region the prediction depends on. The method treats the model as a black box, needing only its inputs and outputs, and is most often used on images.
Example Use Cases
Explainability
Testing which regions of a chest X-ray are critical for pneumonia detection by systematically covering different areas with grey patches and measuring how much the model's confidence drops for each occluded region.
Evaluating whether a facial recognition system relies on specific facial features by masking eyes, nose, mouth, or other regions to identify which areas cause the biggest drop in recognition accuracy.
Analysing agricultural crop disease classification models to verify that predictions focus on visible disease symptoms (discolouration, lesions, wilting) rather than background elements like soil or irrigation equipment, so that a model relying on the background is caught before use on farms with different surroundings.
Limitations
- Computationally expensive as it requires running inference multiple times for each region tested, scaling poorly with input size.
- The choice of patch size, shape and stride changes the map. A patch that is too small may miss features that are informative only together, and one that is too large covers several relevant regions at once and blurs the result.
- Cannot capture interactions between multiple regions that jointly contribute to the prediction but are individually less important. If two regions carry the same evidence, hiding either one alone may change little, so both can look unimportant.
- The occluding patch is itself an unnatural input. A change in the output may reflect the model reacting to the grey or black patch and not to the missing content, and the map can change with the fill value chosen.
Resources
Research Papers
Visualizing and Understanding Convolutional Networks
Large Convolutional Network models have recently demonstrated impressive classification performance on the ImageNet benchmark Krizhevsky et al. [18]. However there is no clear understanding of why they perform so well, or how they might be improved. In this paper we explore both issues. We introduce a novel visualization technique that gives insight into the function of intermediate feature layers and the operation of the classifier. Used in a diagnostic role, these visualizations allow us to find model architectures that outperform Krizhevsky et al on the ImageNet classification benchmark. We also perform an ablation study to discover the performance contribution from different model layers. We show our ImageNet model generalizes well to other datasets: when the softmax classifier is retrained, it convincingly beats the current state-of-the-art results on Caltech-101 and Caltech-256 datasets.