Quantile Regression

Description

Quantile regression predicts chosen percentiles of an outcome, such as the 10th, 50th and 90th, rather than only its average. In the classical method a separate model is fitted for each percentile, using a loss that penalises under-prediction and over-prediction unequally. For example, instead of predicting an average house price of £300,000, it can predict that a house of a given size and location has a 10% chance of selling below £250,000 and a 90% chance of selling below £380,000. Comparing the fitted percentiles shows how features affect different parts of the outcome, such as size mattering more for expensive homes than for cheap ones. A low and a high percentile together give a prediction interval (a range expected to contain a new outcome) without assuming the errors are normally distributed, though how often the interval actually contains new outcomes must be checked on held-out data.

Example Use Cases

Reliability

Predicting patient recovery times after surgery by fitting the 10th, 50th and 90th percentiles, so clinicians can say 'most patients recover within 2 to 4 weeks, but one in ten takes longer than 8' when planning follow-up care and bed use.

A grid operator extends its wind power forecast with 10th and 90th percentile forecasts, fitted on past forecast errors and weather conditions, and sizes reserve capacity to the 90th percentile of the next-day shortfall rather than to an average error.

A flood-forecasting agency fits high percentiles of river level on rainfall and soil moisture, giving an upper bound that does not assume symmetric errors, and uses it to decide when to issue warnings for extreme events.

Limitations

Cost grows with the number of quantile levels fitted when, as in the classical method, each level is a separate optimisation, which adds up on large datasets or with complex models.
May produce crossing quantiles without proper constraints, where predicted 90th percentile values are lower than 50th percentile values, creating logically inconsistent and unusable prediction intervals.
Extreme quantiles (for example the 5th or 95th) rest on few observations and, with heavy-tailed outcomes, are noisy and unstable. With small datasets they cannot be estimated reliably, because too few points lie in the tails.
Gives no guarantee that an interval built from two quantiles contains the stated share of new outcomes. Coverage can fall short if the model form is wrong or the data shift, so check it on held-out data before relying on it.
In its classical form each conditional quantile is assumed to be linear in the features. If the true relationship is curved or has interactions, the fitted quantiles are biased, and flexible models that avoid this lose the readable coefficients.

Resources

Research Papers

Regression Quantiles

Roger Koenker and Gilbert Bassett•Jan 1, 1978

Software Packages

statsmodels

Jun 12, 2011

Statsmodels: statistical modeling and econometrics in Python

Tutorials

Quantile regression

Documentations

statsmodels.regression.quantile_regression.QuantReg

Related Techniques

Name	Description	Assurance Goals
Deep Ensembles	Deep ensembles estimate how uncertain a neural network's predictions are by training several copies of the same network independently and comparing what they predict. Each copy starts from different random initial weights, sees the training data in a different random order, and is trained to output a full predictive distribution, such as a probability for each class or a predicted value with its spread, rather than a single answer. The ensemble's prediction is the average of its members. Where members agree the prediction is usually more reliable, though agreement alone is no guarantee, and where they disagree the model is showing uncertainty about which answer is right, known as epistemic or model uncertainty. In the original study, ensembles of about five networks were better calibrated than single networks, meaning their stated confidence matched how often they were right, and expressed higher uncertainty on inputs from classes absent from the training data.	Reliability Transparency
Monte Carlo Dropout	Monte Carlo dropout estimates how uncertain a neural network is about each prediction. Dropout randomly switches off some of the network's units and is normally used only in training, but here it stays on at prediction time. Each forward pass (one run of an input through the network) therefore uses a different random pattern of units, and the input is run through typically tens to a hundred times. The average gives the prediction, and the spread across runs is read as epistemic uncertainty, meaning uncertainty from the model's limited knowledge rather than noise in the data. The method approximates Bayesian inference, but the spread is an estimate whose reliability should be checked against held-out errors.	Reliability
Bootstrap Ensembles	Bootstrap ensembles estimate how much a model's predictions depend on the particular sample of data it was trained on. The training set is resampled with replacement many times, each resample (a bootstrap replicate) holding as many cases as the original so that some cases are repeated and others left out; one model is trained on each replicate with the same settings, and every new input is passed through all of the models. The spread of the models' predictions for an input, reported as a standard deviation or as a range between percentiles, estimates the uncertainty in the model's prediction for that input, and inputs where the models disagree are those the training data constrain least. The spread covers uncertainty in the fitted model only, not the random variation in individual outcomes, so an interval meant to contain a case's actual outcome must add an estimate of that noise, for example from the models' errors on held-out data. Putting a confidence interval around a measured performance score is a different procedure, which resamples the evaluation set and retrains nothing; see Bootstrap Confidence Intervals for Evaluation Metrics.	Reliability Transparency Fairness