Sparse Autoencoder Feature Analysis

Description

Sparse autoencoder feature analysis looks for human-interpretable units inside a neural network, usually a language model, by learning from the activations at one layer a large dictionary of directions, called features, such that each activation vector can be rebuilt from only a few of them. Unlike neuron activation analysis it does not take the network's own neurons as the units, since single neurons often respond to several unrelated things, and unlike concept activation vectors it needs no labelled examples of a concept chosen in advance, because the features are found without supervision. The analyst records activations at the chosen layer over a large body of ordinary text and trains an autoencoder, a network with one hidden layer usually several times wider than the layer studied, to reconstruct them while a sparsity penalty keeps most hidden units at zero; the decoder weights of each hidden unit then give one feature direction. Each feature is labelled from the inputs on which it fires most strongly, by a person or by a language model that writes a description, which is then scored on how well it predicts the feature's activations on other text, and the label is tested by removing or amplifying the feature inside the running model and checking the change in output. The output is a catalogue of labelled features with example activations, interpretability scores and measured effects, plus a reconstruction score saying how much of the layer's behaviour the dictionary accounts for; the features can then serve as units for causal mediation analysis, which traces pathways, while this method finds and checks the units themselves.

Example Use Cases

Explainability

Before deploying an open-weights language model as a customer-service assistant, a bank's AI assurance team uses sparse autoencoders published for that model to find features active on prompts where the assistant should refuse to give investment advice. It tests each candidate by suppressing it and measuring the change in refusal rate, and records which labelled features the refusal behaviour depends on in the model's assurance file.

A bioinformatics group using a protein language model to annotate newly sequenced proteins trains sparse autoencoders on its embeddings and compares the features with database annotations of binding sites, structural motifs and domains. Features that match known biology are used to propose candidate annotations for curators to verify, features that fire widely with no clear meaning are flagged for review, and the model's own annotation accuracy is still tested on held-out proteins.

A team that fine-tuned a language model to triage clinical referral letters lists the sparse autoencoder features that most change the 'urgent' score on a validation set, and checks whether they describe clinical content or properties of the letter, such as the referring practice's template. A template feature among the leading drivers is recorded as a finding and leads to retraining on reformatted letters.

Limitations

The dictionary does not capture everything the layer carries, and what it misses cannot be inspected. In the originating study, replacing one layer of a small language model with its reconstruction raised perplexity from 25 to 40; a concurrent study recovered 79% of a layer's contribution to the loss in its main run. Not finding a feature for a concept does not show that the model lacks it.
The features found are not unique. Larger dictionaries split a feature into narrower ones (one base64 feature at 512 features became three at 4,096), and for a 131,000-feature autoencoder on one layer of Llama 3 8B only about 30% of features were shared by two runs differing only in random seed. A conclusion resting on one feature should name the autoencoder used and be checked on a second run.
Labels are hypotheses drawn from the top-activating examples, and automated scores are modest: the first five features reported in the originating study scored between -0.11 and 0.57, and the advantage over simpler decompositions faded in later layers. Back a label with an intervention that changes the output as the label predicts before citing it as evidence.
It needs the model's internal activations, plus storage and compute to train an autoencoder for each layer studied (one concurrent study trained on 8 billion activation vectors from a single layer). For large models practitioners often rely on autoencoders released for particular open-weights models, so the method is not available for a model reached only through an API.
Being interpretable does not make the features better for a given task. In a 2025 study of detecting concepts from model activations, probes built on autoencoder features did not beat logistic regression on the raw activations on average, and the spurious correlations and data problems they exposed could also be found with simpler baselines. Where detection is the goal, compare against such baselines.

Resources

Research Papers

Sparse Autoencoders Find Highly Interpretable Features in Language Models

Robert Huben et al.

Towards Monosemanticity: Decomposing Language Models With Dictionary Learning

Software Packages

SAELens (decoderesearch/SAELens)

Oct 12, 2026

Training Sparse Autoencoders on Language Models

Tutorials

Loading and Analysing Pre-Trained Sparse Autoencoders (SAELens tutorial)

Training Sparse Autoencoders on Language Models. Contribute to decoderesearch/SAELens development by creating an account on GitHub.

Documentations

SAELens documentation

Docs for Sparse Autoencoder Training and Analysis Library

Related Techniques

Name	Description	Assurance Goals
Neuron Activation Analysis	Neuron activation analysis examines when individual neurons in a neural network respond strongly. The analyst runs a set of inputs through the model, finds the inputs that activate a chosen neuron most and least, and looks for the concept that separates them, such as buildings in an image classifier or French text in a language model. Some methods test the match automatically, for example by scoring how closely a neuron's activation lines up with labelled examples of a concept. The result is a list of neurons with candidate meanings and the evidence behind each. These meanings are correlations between inputs and activations, so they show what a neuron responds to, not what it does to the model's output.	Explainability
Attention Weight Analysis	Attention weight analysis reads the weights an attention-based neural network assigns to the parts of its input, to see which positions it draws on at each step. The weights come from an ordinary forward pass and are plotted as heatmaps, one row per output step or token. In recurrent encoder-decoder models, such as early neural machine translation or image captioning, each output step has a row over the source words or image regions, read as a soft alignment in which weight is shared across several source positions. In transformers, each attention head (one of several parallel attention units in a layer) has its own matrix at every layer; the analyst views these per head and layer, or averages them over many inputs to profile a head, for example by how much attention it gives each kind of token, or how often its strongest attention lands on a labelled relation such as a verb and its object. The heatmaps show where each step looks; on their own they do not measure how much each input contributed to the prediction, because the weights leave out what is done with the attended values and information mixes between layers.	Explainability
Concept Activation Vectors	A concept activation vector (CAV) is a direction inside a neural network that stands for a human idea such as 'stripes'. The analyst supplies example inputs showing the concept and random counter-examples, and trains a simple linear classifier to separate the network's activations (the values a chosen layer produces) for the two sets. The CAV is the direction at right angles to that classifier's dividing boundary, pointing towards the concept examples. Testing with CAVs (TCAV) then reports, for one output class, the share of its inputs whose class score would rise if the activations moved in the concept's direction, and repeating this with many random counter-example sets and applying a significance test guards against a meaningless direction. Concept sets built from examples of a protected characteristic, such as gender or ethnicity, turn the test into a check of whether a class is sensitive to that characteristic, as the original paper demonstrates.	Explainability Fairness Transparency