Occlusion Sensitivity

Description

Occlusion sensitivity tests which parts of the input are important by occluding (masking or removing) them and seeing how the model's prediction changes. For example, portions of an image can be covered up in a sliding window fashion; if the model's confidence drops significantly when a certain region is occluded, that region was important for the prediction.

Example Use Cases

Explainability

Testing which regions of a chest X-ray are critical for pneumonia detection by systematically covering different areas with grey patches and measuring how much the model's confidence drops for each occluded region.

Evaluating whether a facial recognition system relies on specific facial features by masking eyes, nose, mouth, or other regions to identify which areas cause the biggest drop in recognition accuracy.

Limitations

Computationally expensive as it requires running inference multiple times for each region tested, scaling poorly with input size.
Choice of occlusion size and shape can significantly bias results - too small may miss important features, too large may occlude multiple relevant regions simultaneously.
Cannot capture interactions between multiple regions that jointly contribute to the prediction but are individually less important.
Results may be misleading if the model adapts to occlusion patterns or if occluded regions are filled with unrealistic pixel values.

Resources

kazuto1011/grad-cam-pytorch

Software Package

Occlusion Sensitivity Analysis with Augmentation Subspace Perturbation in Deep Feature Space

Research Paper•Pedro Valois, Koichiro Niinuma, and Kazuhiro Fukui•Nov 25, 2023

Occlusion Sensitivity — tf-explain documentation

Documentation

Adaptive occlusion sensitivity analysis for visually explaining video recognition networks

Research Paper•Tomoki Uchiyama et al.•Jul 26, 2022

sicara/tf-explain

Software Package

Related Techniques

Name	Description	Assurance Goals
Ridge Regression Surrogates	This technique approximates a complex model by training a ridge regression (a linear model with L2 regularization) on the original model's predictions. The ridge regression serves as a global surrogate that balances fidelity and interpretability, capturing the main linear relationships that the complex model learned while ignoring noise due to regularization.	Explainability Transparency
Prompt Sensitivity Analysis	Prompt Sensitivity Analysis systematically evaluates how variations in input prompts affect large language model outputs, providing insights into model robustness, consistency, and interpretability. This technique involves creating controlled perturbations of prompts whilst maintaining semantic meaning, then measuring how these changes influence model responses. It encompasses various types of prompt modifications including lexical substitutions, syntactic restructuring, formatting changes, and contextual variations. The analysis typically quantifies sensitivity through metrics such as output consistency, semantic similarity, and statistical measures of variance across prompt variations.	Explainability Reliability Safety
Causal Mediation Analysis in Language Models	Causal mediation analysis in language models is a mechanistic interpretability technique that systematically investigates how specific internal components (neurons, attention heads, or layers) causally contribute to model outputs. By performing controlled interventions—such as activating, deactivating, or modifying specific components—researchers can trace the causal pathways through which information flows and transforms within the model. This approach goes beyond correlation to establish causal relationships, enabling researchers to understand not just what features influence outputs, but how and why they do so through specific computational pathways.	Explainability Reliability Safety
Permutation Importance	Permutation Importance quantifies a feature's contribution to a model's performance by randomly shuffling its values and measuring the resulting drop in predictive accuracy. If shuffling a feature significantly degrades the model's performance, that feature is considered important. This model-agnostic technique helps identify which inputs are genuinely driving predictions, rather than just being correlated with the outcome.	Explainability Reliability
Gradient-weighted Class Activation Mapping	Grad-CAM creates visual heatmaps showing which regions of an image a convolutional neural network focuses on when making a specific classification. Unlike pixel-level techniques, Grad-CAM produces coarser region-based explanations by using gradients from the predicted class to weight the CNN's final feature maps, then projecting these weighted activations back to create an overlay on the original image. This provides intuitive visual explanations of where the model is 'looking' for evidence of different classes.	Explainability Fairness
Individual Conditional Expectation Plots	ICE plots display the predicted output for individual instances as a function of a feature, with all other features held fixed for each instance. Each line on an ICE plot represents one instance's prediction trajectory as the feature of interest changes, revealing whether different instances are affected differently by that feature.	Explainability

Tags

Applicable Models:

Data Requirements:

No Special Requirements

Data Type:

Evidence Type:

Quantitative Metric

Expertise Needed:

Explanatory Scope:

Lifecycle Stage:

Model Development

Technique Type: