Continual Learning Stability Testing
Description
Continual learning stability testing checks whether a model that is updated over time with new data or tasks keeps what it learned earlier, by scoring it after each update on a test set for every task in the sequence. Backward transfer is the average change in accuracy on earlier tasks, and a large fall is called catastrophic forgetting. Forward transfer is accuracy on a task the model has not yet been trained on, compared with an untrained model. Scoring only at the end of each update can hide a sharp but temporary drop, so stricter tests also score the model during training and report its worst result. The scores can also show whether methods meant to protect old knowledge, such as replaying stored examples of earlier data or penalising changes to the weights that matter for earlier tasks, are working.
Example Use Cases
Reliability
A trust-and-safety team retrains its abuse classifier every month on newly reported content. Before release it scores the new version on a held-out test set for each earlier abuse type and holds the release if detection of any type has fallen by more than the agreed margin.
A bank updates its fraud model weekly on newly confirmed cases. To see whether replaying stored examples of older fraud actually protects detection of old patterns, the team runs the same update with and without replay and compares how well each version still catches each earlier fraud type.
Safety
A hospital team lets a triage model keep learning from new cases while it is in use. Because users meet any dip as it happens, the team scores the model on its urgent conditions at regular points during each update, not only at the end, and stops the update if the lowest score falls below the agreed floor.
A factory retrains a camera-based hazard detector when a new machine is installed. Before it returns to the line, the safety engineer scores the retrained detector on the stored test images for every previously known hazard and signs off only if none is detected less reliably than before.
Limitations
- Every update means scoring the model on a test set for each earlier task, so total cost grows with the number of tasks as well as the number of updates. Old test sets also have to be kept, which can conflict with data-retention rules, and scoring only a sample of tasks cannot detect forgetting in the rest.
- Scores taken only at the end of each update can miss a sharp, temporary drop on earlier tasks while the model is still learning. This matters for a system that stays in service during training, because users meet the dip. Catching it needs repeated scoring during training, which adds to the cost.
- Stability scores measure only one side of the stability-plasticity trade-off. A model can score perfectly on earlier tasks by refusing to learn anything new, so results must be read alongside accuracy on the new tasks.
- Deciding how much forgetting is acceptable depends on the application, and the test cannot settle it. A threshold set too tight blocks adaptation to changing conditions, and one set too loose lets real losses through.
- Results hold for the sequence of tasks tested, and assume tasks with clear boundaries and a labelled test set for each. Real data streams often drift gradually, and how much a model forgets depends on how different consecutive tasks are, so a gentler test sequence understates forgetting.
Resources
Research Papers
Continual evaluation for lifelong learning: Identifying the stability gap
Time-dependent data-generating distributions have proven to be difficult for gradient-based training of neural networks, as the greedy updates result in catastrophic forgetting of previously learned knowledge. Despite the progress in the field of continual learning to overcome this forgetting, we show that a set of common state-of-the-art methods still suffers from substantial forgetting upon starting to learn new tasks, except that this forgetting is temporary and followed by a phase of performance recovery. We refer to this intriguing but potentially problematic phenomenon as the stability gap. The stability gap had likely remained under the radar due to standard practice in the field of evaluating continual learning models only after each task. Instead, we establish a framework for continual evaluation that uses per-iteration evaluation and we define a new set of metrics to quantify worst-case performance. Empirically we show that experience replay, constraint-based replay, knowledge-distillation, and parameter regularization methods are all prone to the stability gap; and that the stability gap can be observed in class-, task-, and domain-incremental learning benchmarks. Additionally, a controlled experiment shows that the stability gap increases when tasks are more dissimilar. Finally, by disentangling gradients into plasticity and stability components, we propose a conceptual explanation for the stability gap.
Gradient Episodic Memory for Continual Learning
One major obstacle towards AI is the poor ability of models to solve new problems quicker, and without forgetting previously acquired knowledge. To better understand this issue, we study the problem of continual learning, where the model observes, once and one by one, examples concerning a sequence of tasks. First, we propose a set of metrics to evaluate models learning over a continuum of data. These metrics characterize models not only by their test accuracy, but also in terms of their ability to transfer knowledge across tasks. Second, we propose a model for continual learning, called Gradient Episodic Memory (GEM) that alleviates forgetting, while allowing beneficial transfer of knowledge to previous tasks. Our experiments on variants of the MNIST and CIFAR-100 datasets demonstrate the strong performance of GEM when compared to the state-of-the-art.