Area Under Precision-Recall Curve

Description

Area Under Precision-Recall Curve (AUPRC) summarises how well a classifier finds the cases that matter when those cases are rare. Precision is the share of positive predictions that are correct, and recall is the share of actual positives that are found. Plotting one against the other at every classification threshold gives a curve, and AUPRC is the area under it, from 0 to 1. Neither precision nor recall uses true negatives, so a large majority class does not flatter the score as it can flatter accuracy and AUC-ROC. A model with no skill scores about the proportion of positives in the data, so the score is only meaningful next to that baseline.

Example Use Cases

Reliability

A payments team compares candidate fraud models on a held-out month of transactions in which about 1 in 500 is fraudulent. It ranks them by AUPRC against the 0.2% no-skill baseline, then reads the best model's curve to choose a threshold that keeps false alarms within what the investigation team can review.

A hospital evaluates a model that screens for a rare condition affecting about 1 in 200 patients. ROC AUC can look high even when most flagged patients are healthy, so the team reports AUPRC against the 0.5% baseline and reads the precision at the recall clinicians require, to decide whether the follow-up workload is manageable.

A manufacturer compares two defect-detection models for a production line where about 1 in 1,000 parts is faulty. It uses AUPRC on a labelled validation batch to choose between them, so that a model is not preferred simply for labelling nearly every good part correctly.

Limitations

AUPRC depends on the share of positives in the data, so scores from datasets or groups with different proportions cannot be compared directly, and no universal pass mark can be set. Each score has to be read against its own no-skill baseline, which is about the positive proportion.
It treats one class as the positive class and ignores true negatives, so it says nothing about how reliably the model clears the majority class. That matters where certifying a case as negative or safe is as important as catching positives.
A single area averages over every threshold, including ones no deployment would use, and can hide poor precision at the recall the application needs. Choosing the operating point needs a look at that part of the curve and the cost of each kind of error.
The area can be computed in different ways. Trapezoidal linear interpolation between points gives an overly optimistic value, whereas step-wise average precision does not, so scores from different libraries or settings may not match.
Few positive test cases make precision-recall estimates unstable, so a high score from a small sample may not hold on a larger, representative evaluation set.

Resources

Research Papers

The relationship between Precision-Recall and ROC curves

Jesse Davis and Mark Goadrich•Jan 1, 2006

Receiver Operator Characteristic (ROC) curves are commonly used to present results for binary decision problems in machine learning. However, when dealing with highly skewed datasets, Precision-Recall (PR) curves give a more informative picture of an algorithm's performance. We show that a deep connection exists between ROC space and PR space, such that a curve dominates in ROC space if and only if it dominates in PR space. A corollary is the notion of an achievable PR curve, which has properties much like the convex hull in ROC space; we show an efficient algorithm for computing this curve. Finally, we also note differences in the two types of curves are significant for algorithm design. For example, in PR space it is incorrect to linearly interpolate between points. Furthermore, algorithms that optimize the area under the ROC curve are not guaranteed to optimize the area under the PR curve.

Software Packages

scikit-learn

scikit-learn: machine learning in Python. Contribute to scikit-learn/scikit-learn development by creating an account on GitHub.

Tutorials

scikit-learn Precision-Recall

Scikit-learn Developers•Jan 1, 2007

Documentations

Precision, recall and F-measures (Metrics and scoring: quantifying the quality of predictions)

Which scoring function should I use?: Before we take a closer look into the details of the many scores and evaluation metrics, we want to give some guidance, inspired by statistical decision theory...

Related Techniques

Name	Description	Assurance Goals
Cross-validation	Cross-validation estimates how well a model will perform on data it has not seen, and how much that estimate depends on the particular data used. The dataset is divided into k parts called folds, the model is trained on all but one fold and scored on the held-out fold, and this repeats until every fold has served as the held-out set once. The scores are averaged to give a performance estimate, and their spread gives a rough indication of how much the result depends on which data the model was trained on. Common variants are stratified cross-validation, where each fold keeps the overall class proportions, and leave-one-out, where each fold is a single example. A large gap between training scores and held-out scores points to overfitting, where a model has learned quirks of its training data that do not carry over to new data.	Reliability
Continuous Ranked Probability Score	The continuous ranked probability score (CRPS) measures the quality of a probabilistic forecast of a numerical quantity, such as tomorrow's temperature or next week's demand, by scoring the whole predicted distribution, not just a single best guess, against the value later observed. For each case it compares the forecast's cumulative distribution (the predicted probability that the outcome will fall below each possible value) with a step that jumps from 0 to 1 at the observed value, squares the gap at every value and adds these up, which is the same as adding up the Brier score of the yes-or-no forecasts 'the outcome will be below this value' over every possible threshold. For a forecast given as an ensemble or a sample it can be computed as the average distance between the members and the observation minus half the average distance between pairs of members, closed forms exist for common distributions such as the Gaussian, and for a single-value forecast it reduces to the absolute error, so probabilistic and deterministic forecasts can be compared directly. The score is proper: a forecaster expects the best score by reporting the distribution they actually believe, so it cannot be improved on average by making forecasts narrower or wider than warranted, and it is lowest when the forecast distribution matches how outcomes actually vary. Scores are averaged over many cases, are in the units of the forecast quantity with lower values better, and are often reported as a skill score relative to a reference forecast such as climatology (the long-run historical distribution), giving evidence for claims about the accuracy of a forecaster's probabilities by variable, location and lead time.	Reliability
Bootstrap Confidence Intervals for Evaluation Metrics	Bootstrap confidence intervals put a range around a model's measured performance, showing how much the score could differ had another test set of the same size been drawn from the same population. The labelled held-out evaluation set is resampled with replacement thousands of times, each resample holding as many cases as the original so that some cases appear more than once and others not at all, and the metric (an error rate, the area under a curve, a translation score) is recomputed on each resample from the model's stored predictions, with no retraining. The spread of these recomputed values stands in for the sampling distribution of the metric: the percentile interval takes its 2.5th and 97.5th percentiles for a 95 per cent interval, and the bias-corrected and accelerated (BCa) interval shifts those percentiles to correct for bias and skew in that distribution. The statistic can also be the difference between two models scored on the same cases, recomputed on each shared resample, which shows whether an observed improvement is larger than the noise from test-set sampling. The result is the metric with lower and upper bounds at a stated confidence level, so a requirement such as a maximum false-positive rate can be checked against the bound rather than the point estimate.	Reliability