Operating-Point ROC Analysis

Description

Operating-point ROC analysis measures how often a classifier makes each kind of mistake at the decision threshold it will actually use, which is the evidence behind claims such as 'the false-positive rate stays within acceptable bounds'. A classifier that outputs a score is turned into a yes-or-no decision by a threshold, and moving the threshold trades one kind of error against the other. On a labelled held-out set, the receiver operating characteristic (ROC) curve plots the true-positive rate (the share of actual positives that are flagged, also called sensitivity) against the false-positive rate (the share of actual negatives wrongly flagged, equal to one minus specificity) at every possible threshold; the operating threshold is then chosen from the curve to meet a stated limit on one error rate or to minimise the expected cost of errors, or an existing threshold is confirmed. At that one threshold the analysis reports the confusion matrix (the counts of true positives, false positives, false negatives and true negatives) and the false-positive and false-negative rates with their complements, specificity and sensitivity. The output is a set of error rates tied to a named threshold and a named test set, against which a requirement on error rates can be checked.

Example Use Cases

Reliability

A hospital's clinical informatics team evaluating a vendor's sepsis alert scores a year of its own admissions, plots the ROC curve and fixes the alert threshold its operations committee chose from the vendor's recommended range. At that threshold it reports sensitivity, specificity, positive predictive value and the share of all admissions that would trigger an alert, and uses the false-positive count and alert volume to decide whether ward staff can act on the alerts.

A card-payments company re-runs the analysis every month on transactions its fraud analysts have labelled, confirms that the false-positive rate at the live threshold is still under the limit agreed with its customer-service team, and, whenever the rate has drifted above that limit, reopens the threshold choice on the latest ROC curve and confirms the new threshold on the following month's labelled sample.

A manufacturer's quality engineers choose the threshold for a visual-inspection model by weighing the cost of scrapping a good part against the cost of shipping a defective one, using the ROC curve and the line's defect rate, then report the false-reject and defect-escape rates that the chosen threshold gives on a separate held-out production batch.

Safety

A rail operator's safety team assessing a camera-based obstacle detector for level crossings sets the decision threshold on a tuning set so that the miss rate meets the requirement allocated to the detector. It then measures the false-negative rate at that threshold on a separate set of recorded crossings, broken down into daylight, night and rain, and enters the confusion matrices in the safety case as verification evidence.

Limitations

The rates are evidence only if they are measured on labelled cases that played no part in choosing the threshold. A threshold set to give a 1 per cent false-positive rate on a tuning set gives exactly that rate there by construction, and with a few hundred negative tuning cases its true rate on new data can be half as high again. Choosing the threshold on one set and reporting rates on another, or nesting the choice inside cross-validation, avoids this at the cost of more labelled data.
False-positive and true-positive rates do not change with how common the positive class is, but the share of alerts that are correct does. A false-positive rate within its limit can still mean most alerts are false when the condition is rare: at 90 per cent sensitivity and a 5 per cent false-positive rate, only about 15 per cent of alerts are correct when 1 case in 100 is positive. Claims about alert burden need the positive predictive value at the deployment prevalence as well.
A rate from a finite test set is an estimate, and with few negatives or few errors it can fall well below the true rate by chance. A claim that a rate is within a limit needs a confidence interval (exact binomial for a single rate; see Bootstrap Confidence Intervals for Evaluation Metrics for other metrics) narrower than the margin to the limit. With no misses, about 3,000 positives show a miss rate below 1 in 1,000, and 30,000 below 1 in 10,000, at 95 per cent confidence.
The threshold is a fixed score, so the rates hold only while the scores for each class stay as in the test set. A new scanner, fraud pattern or patient mix shifts the scores and moves the false-positive rate with no change to the model, so the analysis must be repeated on labelled operational samples, which can be slow or costly. Labels matter too: where only flagged cases are reviewed, the rate for unflagged cases is unknown unless a random sample of them is labelled.
A single rate averages over the whole test set. It can meet its limit overall while exceeding it for a group of people or an operating condition (night-time images, one hospital site) that is rare in the test data. Rates need reporting for each group and condition the claim covers, each with enough cases to be informative, which can multiply the labelled data required.

Resources

Research Papers

Basic principles of ROC analysis

Charles E. Metz•Oct 1, 1978

An introduction to ROC analysis

Tom Fawcett•Jun 1, 2006

Software Packages

scikit-learn

scikit-learn: machine learning in Python. Contribute to scikit-learn/scikit-learn development by creating an account on GitHub.

Tutorials

Post-tuning the decision threshold for cost-sensitive learning — scikit-learn

Once a classifier is trained, the output of the predict method outputs class label predictions corresponding to a thresholding of either the decision_function or the predict_proba output. For a bin...

Documentations

Receiver operating characteristic (ROC) (Metrics and scoring: quantifying the quality of predictions) — scikit-learn user guide

Which scoring function should I use?: Before we take a closer look into the details of the many scores and evaluation metrics, we want to give some guidance, inspired by statistical decision theory...

Related Techniques

Name	Description	Assurance Goals
Area Under Precision-Recall Curve	Area Under Precision-Recall Curve (AUPRC) summarises how well a classifier finds the cases that matter when those cases are rare. Precision is the share of positive predictions that are correct, and recall is the share of actual positives that are found. Plotting one against the other at every classification threshold gives a curve, and AUPRC is the area under it, from 0 to 1. Neither precision nor recall uses true negatives, so a large majority class does not flatter the score as it can flatter accuracy and AUC-ROC. A model with no skill scores about the proportion of positives in the data, so the score is only meaningful next to that baseline.	Reliability
Bootstrap Confidence Intervals for Evaluation Metrics	Bootstrap confidence intervals put a range around a model's measured performance, showing how much the score could differ had another test set of the same size been drawn from the same population. The labelled held-out evaluation set is resampled with replacement thousands of times, each resample holding as many cases as the original so that some cases appear more than once and others not at all, and the metric (an error rate, the area under a curve, a translation score) is recomputed on each resample from the model's stored predictions, with no retraining. The spread of these recomputed values stands in for the sampling distribution of the metric: the percentile interval takes its 2.5th and 97.5th percentiles for a 95 per cent interval, and the bias-corrected and accelerated (BCa) interval shifts those percentiles to correct for bias and skew in that distribution. The statistic can also be the difference between two models scored on the same cases, recomputed on each shared resample, which shows whether an observed improvement is larger than the noise from test-set sampling. The result is the metric with lower and upper bounds at a stated confidence level, so a requirement such as a maximum false-positive rate can be checked against the bound rather than the point estimate.	Reliability
Continuous Ranked Probability Score	The continuous ranked probability score (CRPS) measures the quality of a probabilistic forecast of a numerical quantity, such as tomorrow's temperature or next week's demand, by scoring the whole predicted distribution, not just a single best guess, against the value later observed. For each case it compares the forecast's cumulative distribution (the predicted probability that the outcome will fall below each possible value) with a step that jumps from 0 to 1 at the observed value, squares the gap at every value and adds these up, which is the same as adding up the Brier score of the yes-or-no forecasts 'the outcome will be below this value' over every possible threshold. For a forecast given as an ensemble or a sample it can be computed as the average distance between the members and the observation minus half the average distance between pairs of members, closed forms exist for common distributions such as the Gaussian, and for a single-value forecast it reduces to the absolute error, so probabilistic and deterministic forecasts can be compared directly. The score is proper: a forecaster expects the best score by reporting the distribution they actually believe, so it cannot be improved on average by making forecasts narrower or wider than warranted, and it is lowest when the forecast distribution matches how outcomes actually vary. Scores are averaged over many cases, are in the units of the forecast quantity with lower values better, and are often reported as a skill score relative to a reference forecast such as climatology (the long-run historical distribution), giving evidence for claims about the accuracy of a forecaster's probabilities by variable, location and lead time.	Reliability