GNNExplainer

Description

GNNExplainer explains one prediction of a graph neural network, a model that passes information along the edges of a graph such as a molecule, a payment network or a mesh of weather grid points, by finding a small subgraph and a small set of input features that most support that prediction. Explanation methods for flat inputs, such as SHAP and LIME, give each input feature its own score and have no built-in notion of a connection between two nodes; this method scores the connections themselves, restricted to the part of the graph the model actually reads for the prediction (its computation graph, the neighbourhood within as many hops as the model has message-passing layers). It learns a mask with a value between 0 and 1 for each edge of that computation graph and for each node-feature dimension by gradient descent: the masked graph is fed to the model, and the masks are adjusted to keep the predicted label as likely as possible while penalties keep the total mask small and push values towards 0 or 1. Edges and features whose mask values fall below a threshold are then dropped, leaving a compact subgraph and feature subset that can be drawn over the input graph; the same procedure explains node labels, predicted links and whole-graph labels, and the original paper also aligns the explanations of many nodes of one class into a summary pattern. It needs gradient access to the model, and each explanation is a separate optimisation.

Example Use Cases

Explainability

A pharmaceutical company's computational toxicology team uses a graph neural network to flag candidate molecules as likely mutagenic. For each flagged compound it reviews the substructure and atom features GNNExplainer selects and checks them against known structural alerts such as nitro groups, to decide whether a flag rests on chemistry the team recognises before the compound is deprioritised.

A bank's financial-crime unit scores accounts with a graph neural network over its payment network to flag likely money-mule accounts. Investigators receive with each flag the explanation subgraph of counterparties and transfers and the account features it selected, and use them to decide whether to escalate; model validators sample explanations to check that flags do not rest mainly on features such as branch or account-opening channel.

A weather service's model evaluation team runs the regression mode of the PyTorch Geometric implementation on a small set of high-impact forecasts from a graph-based forecasting model, such as the predicted rainfall total at one location. Forecasters check whether the connections and input variables selected, for example humidity upwind, are physically plausible, and the team deletes the selected connections to confirm the forecast changes, before such forecasts feed a warning service.

Limitations

Each explanation is a separate optimisation, run for a set number of steps (100 by default in the PyTorch Geometric implementation), so explaining many nodes or graphs is slow and explanations of similar cases are not required to agree. A view of the model as a whole needs an extra step, such as the class-level summary in the original paper, or a method that trains one explainer for all instances.
The masks are soft during optimisation, so the model is run on graphs whose edges and features are scaled by fractions it never saw in training, and these values can add meaning or noise of their own. The objective is not convex and the masks start at random: in a 2024 benchmark, the ten-edge explanations from runs with different seeds overlapped by a Jaccard index of only 0.18 to 0.80 across six datasets. The final subgraph also depends on the threshold used to cut the mask.
The size of the explanation is set by the user, not found. The original experiments fixed it at the known size of the planted pattern on synthetic graphs and at 10 edges and 5 features on real ones, and the PyTorch Geometric implementation advises tuning the size penalty to the graph's average degree. A compact explanation can therefore leave out structure the model uses.
Most evidence that it finds the right structure comes from synthetic graphs with planted patterns, and a trained model need not use the planted edges, so agreement with them can mislead. A 2023 benchmark found graph explainers, this one included, less faithful when linked nodes tend to differ and when the deciding structure is large. Check fidelity on your own model, for example by deleting the explanation and measuring the change.
It assumes the computation graph is small. The computation graph grows with every message-passing layer, so in a mesh-based forecasting model that passes messages many times along long-range edges (GraphCast uses 16 such layers) it probably spans most of the mesh: the mask then covers most edges and compactness comes from the size penalty, not the model. This has not been tested. PyTorch Geometric applies the mask inside its own message-passing layers, so the model must use them or be adapted.

Resources

Research Papers

GNNExplainer: Generating Explanations for Graph Neural Networks

Zhitao Ying et al.•Jan 1, 2019

Graph Neural Networks (GNNs) are a powerful tool for machine learning on graphs.GNNs combine node feature information with the graph structure by recursively passing neural messages along edges of the input graph. However, incorporating both graph structure and feature information leads to complex models, and explaining predictions made by GNNs remains unsolved. Here we propose GNNExplainer, the first general, model-agnostic approach for providing interpretable explanations for predictions of any GNN-based model on any graph-based machine learning task. Given an instance, GNNExplainer identifies a compact subgraph structure and a small subset of node features that have a crucial role in GNN's prediction. Further, GNNExplainer can generate consistent and concise explanations for an entire class of instances. We formulate GNNExplainer as an optimization task that maximizes the mutual information between a GNN's prediction and distribution of possible subgraph structures. Experiments on synthetic and real-world graphs show that our approach can identify important graph structures as well as node features, and outperforms baselines by 17.1% on average. GNNExplainer provides a variety of benefits, from the ability to visualize semantically relevant structures to interpretability, to giving insights into errors of faulty GNNs.

Software Packages

PyTorch Geometric (pyg-team/pytorch_geometric)

Jan 1, 2025

Graph Neural Network Library for PyTorch

Tutorials

Explaining Graph Neural Networks (PyG tutorial)

Documentations

torch_geometric.explain.algorithm.GNNExplainer (PyG documentation)

Related Techniques

Name	Description	Assurance Goals
Contrastive Explanation Method	The Contrastive Explanation Method (CEM) explains one prediction of a classifier, such as a neural network, by describing what is present and what is absent in the input. A pertinent positive is the smallest set of features already present that is enough for the model to give the same prediction. A pertinent negative is the smallest set of features not currently present that would change the prediction to a different class if they were added. CEM finds both by solving an optimisation problem that searches for the smallest change to the input, and it can use an autoencoder (a model trained to reconstruct realistic inputs) to keep the changed inputs realistic. An explanation reads like 'this digit is a 3 because these strokes are present and because a short bar at the top is absent', since adding that bar would make it a 5.	Explainability Transparency
Occlusion Sensitivity	Occlusion sensitivity shows which parts of an input a model's prediction is sensitive to. A region of the input, such as a square patch of an image, is hidden by replacing it with a neutral value like grey, and the change in the model's output is recorded, usually the probability of the class of interest. Sliding the patch across the whole input gives a map of how much the output changes when each region is hidden. A large drop marks a region the prediction depends on. The method treats the model as a black box, needing only its inputs and outputs, and is most often used on images.	Explainability
TimeSHAP	TimeSHAP explains one prediction of a model that reads a sequence of events, such as a recurrent neural network that scores a bank transaction using the account's history, by giving each past event, each input feature and selected event-feature cells a Shapley value: a share of the score, taken from cooperative game theory, such that all the shares add up to the difference between the model's score and its score with every event and feature removed. It adapts KernelSHAP, the sampling estimator in SHAP, which explains a single input vector and so, applied to the latest event of a recurrent model, gives earlier events no credit; TimeSHAP instead switches off whole rows or columns of the event-by-feature input matrix, replacing them with a background value (the training-data average in the original paper), queries the model on many such perturbed sequences, and fits a weighted linear model whose coefficients are the attributions. Because the number of event combinations doubles with every event, a pruning step first walks back from the most recent event and merges all older events into a single player once their combined attribution falls below a chosen tolerance, and cell-level values are computed only where the most important events and features intersect, with the other cells grouped. The model is treated as a black box that takes a batch of sequences and returns one score per sequence, and the output is event, feature and cell attributions for each explained sequence, plus plots that aggregate them across many sequences into a global view.	Explainability