Out-of-Distribution Detector for Neural Networks

Description

ODIN (Out-of-Distribution Detector for Neural Networks) flags inputs that look unlike the data a neural network classifier was trained on, and it works on an already trained network without retraining it. A classifier reports a confidence score, the highest probability it gives to any class, but that score is often high even for unfamiliar inputs. ODIN adjusts the score in two ways so that familiar and unfamiliar inputs separate better. It divides the classifier's raw output scores by a large constant, called the temperature, before turning them into probabilities, and it nudges the input slightly in the direction that raises the score, which lifts familiar inputs more than unfamiliar ones. An input whose adjusted score falls below a chosen threshold is flagged as out-of-distribution, so its prediction can be withheld or referred for review.

Example Use Cases

Reliability

A hospital screens each chest X-ray with ODIN before the diagnostic classifier's result is shown. Scans that are not chest X-rays, or that carry heavy artefacts, are flagged for human review instead of receiving a confident diagnosis, and the false positive rate is measured on a sample of such scans first.

Safety

An automated driving team scores each camera frame with ODIN and passes frames below the threshold to a fallback routine, such as slowing the vehicle and requesting remote assistance. The team measures how many unfamiliar frames, such as sensor faults or unusual scenes, are wrongly accepted before relying on the detector.

Limitations

Temperature and perturbation size are tuned on a validation set that includes out-of-distribution examples, so some representative unfamiliar data is needed before deployment. The paper found settings transferred reasonably between validation sets, but only on image benchmarks.
Detection gets harder as unfamiliar inputs resemble the training data. In the original paper, at 95% acceptance of familiar images, the share of unfamiliar natural images wrongly accepted rose from at most about 11% with a 10-class training set to between 19% and 58% with a 100-class one, so performance must be measured on the deployment's own data.
Not designed to resist inputs deliberately crafted to look in-distribution, and the original paper does not evaluate adversarial evasion, so it gives no protection against a deliberate attacker.
Needs access to the network's gradients to compute the input perturbation, so it cannot be applied to a model that is reachable only through an API.
Each input needs at least one extra forward pass and one backward pass before the final scoring pass, which can matter for real-time or high-volume use.

Resources

Research Papers

Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks

Shiyu Liang, Yixuan Li, and R. Srikant•Jun 8, 2017

We consider the problem of detecting out-of-distribution images in neural networks. We propose ODIN, a simple and effective method that does not require any change to a pre-trained neural network. Our method is based on the observation that using temperature scaling and adding small perturbations to the input can separate the softmax score distributions between in- and out-of-distribution images, allowing for more effective detection. We show in a series of experiments that ODIN is compatible with diverse network architectures and datasets. It consistently outperforms the baseline approach by a large margin, establishing a new state-of-the-art performance on this task. For example, ODIN reduces the false positive rate from the baseline 34.7% to 4.3% on the DenseNet (applied to CIFAR-10) when the true positive rate is 95%.

Software Packages

odin

Feb 10, 2018

A simple and effective method for detecting out-of-distribution images in neural networks.

Tutorials

ODIN - CIFAR10 - pytorch-ood

Reproduces the ODIN benchmark for OOD detection, from the paper Enhancing the reliability of out-of-distribution image detection in neural networks. This produces a table with the following output:...

Documentations

ODIN - pytorch-ood

classification no segmentation

Related Techniques

Name	Description	Assurance Goals
Out-of-Domain Detection	Out-of-domain detection checks whether a user's query falls inside the topics and tasks a conversational AI system is meant to handle. Scope is defined by examples, such as an assistant's supported intents (the actions it can recognise), and a detector then scores how closely each new query resembles them. Typical detectors set a minimum confidence on an intent classifier, add an explicit out-of-scope class trained on examples of unsupported queries, or measure how far a query's embedding, a numerical summary of its meaning, lies from the in-scope examples. A query judged out of scope can be declined, handed to a person or another service, or answered with a request to rephrase, instead of receiving an unreliable or invented answer. The detector tests resemblance to the defined scope, not what the system actually knows, so it needs evaluating on realistic out-of-scope queries.	Reliability Safety Transparency
Monte Carlo Dropout	Monte Carlo dropout estimates how uncertain a neural network is about each prediction. Dropout randomly switches off some of the network's units and is normally used only in training, but here it stays on at prediction time. Each forward pass (one run of an input through the network) therefore uses a different random pattern of units, and the input is run through typically tens to a hundred times. The average gives the prediction, and the spread across runs is read as epistemic uncertainty, meaning uncertainty from the model's limited knowledge rather than noise in the data. The method approximates Bayesian inference, but the spread is an estimate whose reliability should be checked against held-out errors.	Reliability
Runtime Monitoring and Circuit Breakers	Runtime monitoring and circuit breakers observe a deployed AI service and restrict model calls when configured signals breach specified limits. In the closed state, requests pass through and recent failures are counted, while an open breaker rejects requests or provides a configured fallback, such as a cached answer, a simpler model, a rule-based default or referral to a person. After a waiting period, the half-open state permits a few trial calls and closes again only if they succeed. Model-related triggers can include immediately available signals such as input-drift scores (how far incoming data has moved from the training data) or low prediction confidence, since accuracy usually needs later outcome labels. Logs record state changes and triggering signals, supporting review of why the service was restricted without guaranteeing that the fallback is safe.	General Reliability Safety Transparency