Layer-wise Relevance Propagation

Description

Layer-wise Relevance Propagation (LRP) explains a neural network's prediction by working backwards from the output to the inputs. It treats the prediction score as a quantity called relevance, then passes that relevance back through each layer in proportion to how much each neuron contributed in the forward pass. The rules for this redistribution are chosen to keep the total relevance roughly constant from layer to layer, so the input scores add up to the prediction apart from small amounts absorbed by bias terms and stabilisers. Different rules can be used for different types of layer. The result is a relevance score for each input, such as a pixel or a feature, with positive values for inputs that supported the prediction and negative values for inputs that counted against it.

Example Use Cases

Explainability

Identifying which pixels in chest X-rays contribute to pneumonia detection, helping radiologists verify AI diagnoses by highlighting anatomical regions the model considers relevant.

A perception team investigating a misclassified traffic-sign image draws the relevance map for the wrong class. The map shows the relevance sitting on a sticker and the background rather than on the sign itself, which tells the team to add training images with such clutter before the model is retested.

Transparency

A bank's model-risk reviewers receive LRP relevance scores for a sample of declined applications scored by a neural network. They check that the inputs with the most relevance for the decline output are affordability measures and not proxy fields such as postcode, and file the scores with the decision record for audit.

Limitations

Each layer type needs its own propagation rule, and a new layer type has none until someone derives one, so adapting LRP to a new architecture takes expertise and testing.
Rule selection (LRP-ε, LRP-γ and others) changes the resulting maps, so two analysts can produce different explanations of the same prediction, and choosing among them takes expertise.
Heatmaps are hard to validate. The original paper judges them by eye and with a pixel-flipping test, and notes that quality beyond visual assessment is an open question, so a plausible-looking map does not prove the model relies on the highlighted inputs.
Standard rules do not cover attention layers. Extensions for transformers, such as AttnLRP in the LXT library, exist but are recent and specific to certain architectures, so users of transformers or large language models must adopt them and confirm they match their model.
Bias handling and epsilon stabilisation can absorb or inject relevance, so input scores do not always sum exactly to the selected output. A report should document the accounting residual before claiming conservation.

Resources

Research Papers

On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation

Sebastian Bach et al.•Jul 10, 2015

Understanding and interpreting classification decisions of automated image classification systems is of high value in many applications, as it allows to verify the reasoning of the system and provides additional information to the human expert. Although machine learning methods are solving very successfully a plethora of tasks, they have in most cases the disadvantage of acting as a black box, not providing any information about what made them arrive at a particular decision. This work proposes a general solution to the problem of understanding classification decisions by pixel-wise decomposition of nonlinear classifiers. We introduce a methodology that allows to visualize the contributions of single pixels to predictions for kernel-based classifiers over Bag of Words features and for multilayered neural networks. These pixel contributions can be visualized as heatmaps and are provided to a human expert who can intuitively not only verify the validity of the classification decision, but also focus further analysis on regions of potential interest. We evaluate our method for classifiers trained on PASCAL VOC 2009 images, synthetic image data containing geometric shapes, the MNIST handwritten digits data set and for the pre-trained ImageNet model available as part of the Caffe open source package.

Software Packages

LRP-eXplains-Transformers

Dec 5, 2023

Layer-wise Relevance Propagation for Large Language Models and Vision Transformers [ICML 2024]

lrp_toolbox

May 26, 2016

The LRP Toolbox provides simple and accessible stand-alone implementations of LRP for artificial neural networks supporting Matlab and Python. The Toolbox realizes LRP functionality for the Caffe Deep Learning Framework as an extension of Caffe source code published in 10/2015.

Zennit (chr5tphr/zennit)

Sep 19, 2026

Zennit is a high-level framework in Python using PyTorch for explaining/exploring neural networks using attribution methods like LRP.

Tutorials

Tutorial: Implementing Layer-Wise Relevance Propagation

GitLab Enterprise Edition

Layer-Wise Relevance Propagation: An Overview

Grégoire Montavon et al.•Jan 1, 2019

Documentations

Getting started — zennit documentation

Zennit Developers

Layer-wise Relevance Propagation eXplains Transformers (LXT) documentation

Lxt Developers•Jan 1, 2024

Related Techniques

Name	Description	Assurance Goals
DeepLIFT	DeepLIFT (Deep Learning Important FeaTures) explains a single prediction of a neural network by dividing the difference between the model's output and its output on a reference input among the input features. The reference is a neutral input chosen for the problem, such as an all-zero image or a blurred copy of the input. The method measures how far each neuron's activation is from its value on the reference, then passes those differences backwards through the layers in one pass, applying a rule for each layer type. A plain gradient (the rate of change of the output with respect to an input) is zero where the output has levelled off, but DeepLIFT can still assign credit there. The scores add up to the output difference and mean something only relative to the reference chosen.	Explainability Transparency
Contextual Decomposition	Contextual Decomposition explains why an LSTM, a neural network that reads text one word at a time, made a particular prediction. The user picks a word or phrase, and the method splits the network's output score into two parts: the part due to that phrase alone, and the part due to everything else, including how the phrase interacts with the words around it. Comparing a phrase's score with the scores of its parts reveals effects that word-by-word methods miss, such as a negation reversing the sentiment of the word that follows it. The split is tracked through the LSTM's internal gates, so the model itself is not changed or retrained. The output is a numerical score for each phrase examined.	Explainability Transparency
Taylor Decomposition	Taylor Decomposition explains one prediction of a neural network by splitting its output score into a contribution from each input, such as each pixel. It starts from a reference input, called a root point, that is close to the real input but for which the network's score is zero. Each input's contribution is the slope of the score with respect to that input at the root point, multiplied by how far the input sits from the root, and terms that involve several inputs at once are left out. Finding a good root point is slow for large networks, so the deep variant applies the same idea to one neuron at a time and passes the contributions backwards through the layers, which gives rules close to those of Layer-wise Relevance Propagation. The result is a relevance score for every input that can be drawn as a heatmap.	Explainability