DeepLIFT

Description

DeepLIFT (Deep Learning Important FeaTures) explains a single prediction of a neural network by dividing the difference between the model's output and its output on a reference input among the input features. The reference is a neutral input chosen for the problem, such as an all-zero image or a blurred copy of the input. The method measures how far each neuron's activation is from its value on the reference, then passes those differences backwards through the layers in one pass, applying a rule for each layer type. A plain gradient (the rate of change of the output with respect to an input) is zero where the output has levelled off, but DeepLIFT can still assign credit there. The scores add up to the output difference and mean something only relative to the reference chosen.

Example Use Cases

Explainability

A genomics group training a neural network to predict where a regulatory protein binds DNA computes DeepLIFT scores for each predicted binding region against a reference sequence with background base frequencies. They read off which short stretches of bases carry the score, compare them with known binding motifs, and decide whether the model has learned real regulatory sequence before using it to prioritise experiments.

A hospital imaging team is investigating a chest X-ray classifier that flags healthy patients as abnormal. For the false positives it computes DeepLIFT scores against a blurred copy of each image and checks whether the weight sits on the image border, where scanner labels are printed, rather than on lung tissue. It then decides whether to mask the labels and retrain.

A regional geohazard agency uses a neural network to rate landslide susceptibility from terrain, rainfall and land-cover layers. For a slope the model rates as high risk, DeepLIFT scores show which input layers drove the rating, and the agency compares them with the field investigation report before using the map for planning.

Transparency

A lender's credit team sends each declined applicant a short statement of the factors that most lowered their score. For its neural network scorer it computes DeepLIFT scores against a documented reference applicant, the median approved applicant, and the statement lists the factors with the largest negative scores, which are the ones that pulled this score below the reference.

Limitations

Every score is relative to a reference input, and the choice changes the picture. In the original paper an all-zero reference on CIFAR-10 images highlighted hard-to-interpret background pixels, while a blurred copy of the image highlighted object outlines. State the reference used and repeat the analysis with at least one alternative before drawing conclusions.
The Captum implementation supports only the Rescale rule, which can give all the credit to one of two inputs that jointly drive the output. In the paper's min(i1, i2) example Rescale scores only the smaller input, while the RevealCancel rule shares the credit. Where inputs interact, the scores can hide a dependency that a more complete rule would show.
Propagation rules must support the network operations. Captum implements the Rescale rule and a limited set of nonlinearities; inaccessible operations in built-in recurrent modules can prevent attribution unless a supported implementation is used.
Two networks that compute exactly the same function can receive different DeepLIFT scores, because differences between a neuron and its reference do not obey the chain rule that gradients obey (Sundararajan et al., 2017, App. B). The scores therefore describe how the model is built as well as what it computes, which matters when comparing a model with a re-implemented or compressed version of it.
Feature contributions account for a particular output difference, not a causal effect in the world or a guaranteed change after deleting that feature. Correlated genomic bases or financial variables make that distinction important.

Resources

Research Papers

Learning Important Features Through Propagating Activation Differences

Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje•Apr 10, 2017

The purported "black box" nature of neural networks is a barrier to adoption in applications where interpretability is essential. Here we present DeepLIFT (Deep Learning Important FeaTures), a method for decomposing the output prediction of a neural network on a specific input by backpropagating the contributions of all neurons in the network to every feature of the input. DeepLIFT compares the activation of each neuron to its 'reference activation' and assigns contribution scores according to the difference. By optionally giving separate consideration to positive and negative contributions, DeepLIFT can also reveal dependencies which are missed by other approaches. Scores can be computed efficiently in a single backward pass. We apply DeepLIFT to models trained on MNIST and simulated genomic data, and show significant advantages over gradient-based methods. Video tutorial: http://goo.gl/qKb7pL, ICML slides: bit.ly/deeplifticmlslides, ICML talk: https://vimeo.com/238275076, code: http://goo.gl/RM8jvH.

Software Packages

captum

Aug 27, 2019

Model interpretability and understanding for PyTorch

DeepLIFT (kundajelab/deeplift)

Sep 30, 2026

Public facing deeplift repo

Tutorials

Tutorial A3: DeepLIFT/SHAP — tangermeme v0.1.0 documentation

Tangermeme Developers•Jan 1, 2023

Interpreting a CIFAR-10 TorchVision model with Captum

Model Interpretability for PyTorch

Documentations

DeepLIFT Documentation - Captum

Jan 1, 2019

Related Techniques

Name	Description	Assurance Goals
Layer-wise Relevance Propagation	Layer-wise Relevance Propagation (LRP) explains a neural network's prediction by working backwards from the output to the inputs. It treats the prediction score as a quantity called relevance, then passes that relevance back through each layer in proportion to how much each neuron contributed in the forward pass. The rules for this redistribution are chosen to keep the total relevance roughly constant from layer to layer, so the input scores add up to the prediction apart from small amounts absorbed by bias terms and stabilisers. Different rules can be used for different types of layer. The result is a relevance score for each input, such as a pixel or a feature, with positive values for inputs that supported the prediction and negative values for inputs that counted against it.	Explainability Transparency
Taylor Decomposition	Taylor Decomposition explains one prediction of a neural network by splitting its output score into a contribution from each input, such as each pixel. It starts from a reference input, called a root point, that is close to the real input but for which the network's score is zero. Each input's contribution is the slope of the score with respect to that input at the root point, multiplied by how far the input sits from the root, and terms that involve several inputs at once are left out. Finding a good root point is slow for large networks, so the deep variant applies the same idea to one neuron at a time and passes the contributions backwards through the layers, which gives rules close to those of Layer-wise Relevance Propagation. The result is a relevance score for every input that can be drawn as a heatmap.	Explainability
Integrated Gradients	Integrated Gradients assigns input features scores for a prediction relative to a chosen reference input, called the baseline. It evaluates gradients, rates of output change, at points along a straight path from that reference to the actual input. Each feature's average gradient is multiplied by its change from the reference, and the resulting scores sum approximately to the output difference when enough numerical steps are used. For text, the path runs through continuous token representations rather than discrete word identifiers, and scores can be combined into highlighted words. In a protected-term variant, the reviewer fixes before the run a list of words or columns that name or record a protected characteristic, and reads the scores those inputs receive: a large score shows that the output rests partly on the characteristic, but a small score does not show that the model ignores it.	Explainability Fairness