Explainability
Internal Mechanisms
Reveals how the model processes information internally
12 techniques in this subcategory
12 techniques
| Goals | Models | Data Types | Description | |||
|---|---|---|---|---|---|---|
| Layer-wise Relevance Propagation | Algorithmic | Requirements/model Internals Architecture/neural Networks +3 | Any | Layer-wise Relevance Propagation (LRP) explains a neural network's prediction by working backwards from the output to... | ||
| Contextual Decomposition | Algorithmic | Architecture/neural Networks/recurrent Requirements/white Box +1 | Text | Contextual Decomposition explains why an LSTM, a neural network that reads text one word at a time, made a particular... | ||
| Taylor Decomposition | Algorithmic | Requirements/model Internals Architecture/neural Networks +3 | Any | Taylor Decomposition explains one prediction of a neural network by splitting its output score into a contribution from... | ||
| Gradient-weighted Class Activation Mapping | Algorithmic | Architecture/neural Networks/convolutional Requirements/architecture Specific +3 | Image | Grad-CAM (Gradient-weighted Class Activation Mapping) draws a heatmap over an image showing which regions most raise a... | ||
| Attention Weight Analysis | Algorithmic | Architecture/neural Networks/recurrent Architecture/neural Networks/transformer +2 | Image Text | Attention weight analysis reads the weights an attention-based neural network assigns to the parts of its input, to see... | ||
| Factor Analysis | Algorithmic | Architecture/model Agnostic Paradigm/unsupervised +1 | Tabular | Factor analysis looks for a small number of hidden factors that explain why a larger set of measured variables, such as... | ||
| t-SNE | Algorithmic | Architecture/model Agnostic Requirements/black Box | Any | t-SNE (t-distributed Stochastic Neighbour Embedding) turns high-dimensional data into a two- or three-dimensional map... | ||
| Neuron Activation Analysis | Algorithmic | Architecture/neural Networks Requirements/model Internals +1 | Text | Neuron activation analysis examines when individual neurons in a neural network respond strongly. The analyst runs a set... | ||
| Causal Mediation Analysis in Language Models | Algorithmic | Architecture/neural Networks/transformer Architecture/neural Networks/transformer/llm +3 | Text | Causal mediation analysis traces which internal components of a language model carry the effect of a change in the input... | ||
| Concept Activation Vectors | Algorithmic | Architecture/neural Networks Requirements/gradient Access +2 | Any | A concept activation vector (CAV) is a direction inside a neural network that stands for a human idea such as 'stripes'.... | ||
| Chain-of-Thought Faithfulness Evaluation | Testing | Architecture/neural Networks/transformer/llm Paradigm/generative +1 | Text | Chain-of-thought faithfulness evaluation tests whether the step-by-step reasoning a language model writes out is what... | ||
| Sparse Autoencoder Feature Analysis | Algorithmic | Architecture/neural Networks/transformer Architecture/neural Networks/transformer/llm +2 | Text | Sparse autoencoder feature analysis looks for human-interpretable units inside a neural network, usually a language... |
Rows per page
Page 1 of 1