Contextual Decomposition
Description
Contextual Decomposition explains why an LSTM, a neural network that reads text one word at a time, made a particular prediction. The user picks a word or phrase, and the method splits the network's output score into two parts: the part due to that phrase alone, and the part due to everything else, including how the phrase interacts with the words around it. Comparing a phrase's score with the scores of its parts reveals effects that word-by-word methods miss, such as a negation reversing the sentiment of the word that follows it. The split is tracked through the LSTM's internal gates, so the model itself is not changed or retrained. The output is a numerical score for each phrase examined.
Example Use Cases
Explainability
Analysing why an LSTM-based spam filter flagged an email by decomposing contributions from individual words ('free', 'urgent') versus their contextual interactions ('free trial' together).
A hospital analytics team checks an LSTM that flags clinical notes mentioning possible pneumonia. Contextual decomposition shows whether negated findings such as 'no evidence of pneumonia' push the score down, as intended, or whether the model reacts to the word 'pneumonia' alone, which tells the team if it needs retraining on negated examples before use.
Transparency
A platform's trust-and-safety team handles an appeal against an LSTM toxicity classifier's removal of a post. The reviewing moderator is shown the phrases with the highest contextual decomposition scores, including whether a negation such as 'not stupid' was read correctly, so the moderator can judge whether the removal rested on the post's actual meaning.
Limitations
- Developed for LSTMs, whose gates the method uses to separate a phrase's own effect from its interactions. The source does not cover plain recurrent networks, transformers or attention models, so teams using those cannot apply the published method as it stands.
- Needs internal states, gate parameters and classifier weights. Output labels or probabilities alone do not reveal the phrase-related and remaining-input components.
- The user must choose each phrase to examine, and the published method handles contiguous phrases. The number of possible phrases grows quickly with text length, so scanning a whole document for interactions is costly and easy to leave incomplete.
- The split relies on linearising the gate and activation functions and averaging over orderings of the terms. Interaction scores are therefore approximate, and the source validates them only on sentiment tasks (Stanford Sentiment Treebank, Yelp), so scores on other tasks need checking independently.
- Components combine additively in hidden states and class scores before the probability transformation. They should not be read as independent probabilities or as the exact effect of deleting a phrase.
Resources
Research Papers
Beyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMs
The driving force behind the recent success of LSTMs has been their ability to learn complex and non-linear relationships. Consequently, our inability to describe these relationships has led to LSTMs being characterized as black boxes. To this end, we introduce contextual decomposition (CD), an interpretation algorithm for analysing individual predictions made by standard LSTMs, without any changes to the underlying model. By decomposing the output of a LSTM, CD captures the contributions of combinations of words or variables to the final prediction of an LSTM. On the task of sentiment analysis with the Yelp and SST data sets, we show that CD is able to reliably identify words and phrases of contrasting sentiment, and how they are combined to yield the LSTM's final prediction. Using the phrase-level labels in SST, we also demonstrate that CD is able to successfully extract positive and negative negations from an LSTM, something which has not previously been done.