TimeSHAP
Description
TimeSHAP explains one prediction of a model that reads a sequence of events, such as a recurrent neural network that scores a bank transaction using the account's history, by giving each past event, each input feature and selected event-feature cells a Shapley value: a share of the score, taken from cooperative game theory, such that all the shares add up to the difference between the model's score and its score with every event and feature removed. It adapts KernelSHAP, the sampling estimator in SHAP, which explains a single input vector and so, applied to the latest event of a recurrent model, gives earlier events no credit; TimeSHAP instead switches off whole rows or columns of the event-by-feature input matrix, replacing them with a background value (the training-data average in the original paper), queries the model on many such perturbed sequences, and fits a weighted linear model whose coefficients are the attributions. Because the number of event combinations doubles with every event, a pruning step first walks back from the most recent event and merges all older events into a single player once their combined attribution falls below a chosen tolerance, and cell-level values are computed only where the most important events and features intersect, with the other cells grouped. The model is treated as a black box that takes a batch of sequences and returns one score per sequence, and the output is event, feature and cell attributions for each explained sequence, plus plots that aggregate them across many sequences into a global view.
Example Use Cases
Explainability
A bank's fraud operations team runs TimeSHAP on each alert from its recurrent account-takeover model, so the analyst reviewing an alert sees which earlier events (for example a password change, then a login from a new device) and which features carried the score, not only the payment being held. The analyst uses this to decide whether to release the payment or contact the customer, and the model team samples the explanations to check that the drivers match known fraud patterns.
A hospital data-science team validating a recurrent model that predicts next-day escalation of treatment for patients in intensive care computes event- and feature-level TimeSHAP values for test-set patients and reviews them with clinicians. Drivers the clinicians judge implausible, such as an indicator that a measurement is missing outweighing the measurements themselves, are investigated before the model is used in a prospective study.
An airport's forecasting team using a recurrent model to warn of fog in the next six hours from the past 48 hours of station observations uses the pruning statistics and global event plots to find how far back the model actually draws on, and the feature attributions to check that humidity, temperature and wind lead each warning. The findings set the input window for the next retraining and go into the model's validation report.
Limitations
- Every attribution is measured against the background chosen to stand for a removed event or feature, the training-data average in the original paper. A sequence containing averaged events may be unlike anything the model has seen, and another background gives other attributions; the original authors note that a feature scoring zero for every sequence may reflect the background rather than the model. Report the background with the results.
- Exact values need every combination of events, which doubles with each event. On the original fraud data (median sequence about 138 events), exact values were possible for only 10% of sequences with 32,000 sampled combinations, and repeated runs disagreed (relative standard deviation 1.71 without pruning, 0.98 at the chosen tolerance). With tens of thousands of model calls per explanation, explaining every scored event in real time is impractical; the alerts people review are the usual scope.
- Pruning trades detail on older events for speed and stability. All events before the cut-off share one attribution. The cut-off test compares the older block's net, signed attribution with a user-set tolerance, so by construction an early event that pushed the score up can pass unseen if others pushed it down; the original paper does not measure how often this happens. Results change with the tolerance, so fix it in advance and report it.
- It explains one score per sequence: the reference implementation expects the model to return a single value for each sequence. A model with many outputs, such as a forecast for several lead times or locations, needs a separate run or a chosen summary for each. The attributions show what the model relies on, not causes in the world: in the original study a high attribution to customer age led to a bias audit, which was needed to establish the disparity.
- Each event and feature receives one number, so an effect that arises only from a combination, such as a settings change followed by a login from a new device, is shared out among its parts: an analyst sees each part with some credit but cannot tell that they matter only together. Cell-level values are computed only where the top-ranked events and features intersect. A clinical study using the method found escalation effects hard to see and suggested interactions as the cause.