Temperature Scaling

Description

Temperature scaling adjusts a model's confidence by applying a single parameter (temperature) to its predictions. When a model is too confident in its wrong answers, temperature scaling can fix this by making the predictions more realistic. It works by dividing the model's outputs by the temperature value before converting them to probabilities. Higher temperatures make the model less confident, whilst lower temperatures increase confidence. The technique maintains the model's accuracy whilst ensuring that when it says it's 90% confident, it's actually right about 90% of the time.

Example Use Cases

Reliability

Adjusting a deep learning image classifier's confidence scores to be realistic, ensuring that when it's 90% confident, it's right 90% of the time.

Transparency

Making medical diagnosis model predictions more trustworthy by providing realistic confidence scores that doctors can interpret and use to make informed decisions about patient care.

Fairness

Ensuring fair treatment across patient demographics by calibrating confidence scores equally across different groups, preventing systematic over-confidence in predictions for certain populations.

Limitations

Only addresses calibration at the overall dataset level, not subgroup-specific miscalibration issues.
Does not improve the rank ordering or accuracy of predictions, only adjusts confidence levels.
Assumes that calibration errors are consistent across different types of inputs and feature values.
Requires a separate validation set for temperature parameter optimisation, which may not be available in small datasets.

Resources

gpleiss/temperature_scaling

Software Package

Exploring the Impact of Temperature Scaling in Softmax for Classification and Adversarial Robustness

Research Paper•Hao Xuan, Bokai Yang, and Xingyu Li•Feb 28, 2025

Neural Clamping: Joint Input Perturbation and Temperature Scaling for Neural Network Calibration

Research Paper•Yung-Chen Tang, Pin-Yu Chen, and Tsung-Yi Ho•Jul 24, 2024

On Calibration of Modern Neural Networks | arXiv

Research Paper•Chuan Guo et al.•Jun 14, 2017

On the Limitations of Temperature Scaling for Distributions with Overlaps

Research Paper•Muthu Chidambaram and Rong Ge•Jun 1, 2023

Related Techniques

Name	Description	Assurance Goals
Model Distillation	Model distillation transfers knowledge from a large, complex model (teacher) to a smaller, more efficient model (student) by training the student to mimic the teacher's behaviour. The student learns from the teacher's soft predictions and intermediate representations rather than just hard labels, capturing nuanced decision boundaries and uncertainty. This produces models that are faster, require less memory, and are often more interpretable whilst maintaining much of the original performance. Beyond compression, distillation can improve model reliability by regularising training and enable deployment in resource-constrained environments.	Explainability Reliability Safety
Contextual Decomposition	Contextual Decomposition explains LSTM and RNN predictions by decomposing the final hidden state into contributions from individual inputs and their interactions. Unlike simpler attribution methods, it separates the direct contribution of specific words or phrases from the contextual effects of surrounding words. This is particularly useful for understanding how sequential models process language, as it can identify whether a word's influence comes from its individual meaning or from its interaction with nearby words in the sequence.	Explainability Transparency
SHapley Additive exPlanations	SHAP explains model predictions by quantifying how much each input feature contributes to the outcome. It assigns an importance score to every feature, indicating whether it pushes the prediction towards or away from the average. The method systematically evaluates how predictions change as features are included or excluded, drawing on game theory concepts to ensure a fair distribution of contributions.	Explainability Fairness Reliability
Out-of-DIstribution detector for Neural networks	ODIN (Out-of-Distribution Detector for Neural Networks) identifies when a neural network encounters inputs significantly different from its training distribution. It enhances detection by applying temperature scaling to soften the model's output distribution and adding small, carefully calibrated perturbations to the input that push in-distribution samples towards higher confidence predictions. By measuring the maximum softmax probability after these adjustments, ODIN can effectively distinguish between in-distribution and out-of-distribution inputs, flagging potentially unreliable predictions before they cause downstream errors.	Explainability Reliability Safety
Homomorphic Encryption	Homomorphic encryption allows computation on encrypted data without decrypting it first, producing encrypted results that, when decrypted, match the results of performing the same operations on the plaintext. This enables secure outsourced computation where sensitive data remains encrypted throughout processing. By allowing ML operations on encrypted data, it provides strong privacy guarantees for applications involving highly sensitive information.	Privacy Safety Transparency Security
Permutation Tests	Permutation tests assess the statistical significance of observed results (such as model accuracy, feature importance, or group differences) by comparing them to what would occur purely by chance. The technique randomly shuffles labels or data thousands of times, recalculating the metric of interest each time to build an empirical null distribution. If the actual observed result falls in the extreme tail of this distribution (typically beyond the 95th or 99th percentile), it provides strong evidence that the relationship is genuine rather than due to random chance, without requiring parametric assumptions about data distributions.	Explainability Reliability