Coefficient Magnitudes (in Linear Models)

Description

Coefficient Magnitudes explains a linear model by reading its fitted coefficients. Each coefficient shows how much the prediction changes when its feature rises by one unit while every other feature stays fixed, and its sign gives the direction. Because a coefficient's size depends on the units of its feature, the features are first put on a common scale, for example by standardising them or by multiplying each coefficient by its feature's standard deviation. The features with the largest scaled coefficients are then read as having the strongest influence on the model's predictions. This gives a global view of the model without needing a separate explanation method.

Example Use Cases

Explainability

An estate agency's data team standardises the inputs of a linear house-price model, then ranks the absolute coefficients to show valuers which property attributes move the estimate most. Valuers check that the signs match their experience, for example that more floor area raises the price, before the model guides asking prices.

A public health analyst fits a logistic regression to predict emergency readmission and reports standardised coefficients to clinicians, who see which patient characteristics push the predicted risk up or down. The analyst refits on resamples first, to check that the large coefficients keep their sign.

Transparency

A lender's model-risk team documents a linear credit-scoring model for its regulator by publishing the standardised coefficient of every input. The reviewer can read off the direction and relative weight of each input without a separate explanation tool, and can query any input that carries more weight than policy allows.

Limitations

Raw magnitudes have output-per-input units and cannot be ranked across differently scaled variables. Express comparable input changes explicitly; changing metres to centimetres makes the corresponding coefficient smaller for the same predictions.
Coefficients concern the encoded model inputs while other inputs are held fixed. If the model includes transformed variables or interactions, inspecting one coefficient does not give the whole effect of changing an original variable.
Correlated predictors can have unstable or shared coefficients even when predictions change little. Scaling does not remove this problem, and regularisation can change which correlated feature receives a large weight.
Each coefficient is conditional on the other features in the model. A feature can have a negative coefficient while rising with the outcome on its own, so a coefficient read in isolation can mislead about the feature's plain relationship to the outcome.
A coefficient is a conditional association in the fitted model, not a causal effect. Its magnitude alone also does not give a particular applicant's contribution, which depends on that applicant's feature values.

Resources

Research Papers

The Elements of Statistical Learning

Trevor Hastie, Robert Tibshirani, and Jerome Friedman•Jan 1, 2009

How to Standardize Regression Coefficients

Johan Bring•Aug 1, 1994

In many regression studies, there is an ambition to compare the relative importance of different variables. One measure frequently used is standardized regression coefficients. The present article reveals an inconsistency in the definition of the standardized coefficients and demonstrates that if they were correctly defined, a comparison between standardized coefficients would be similar to comparing t values.

Software Packages

statsmodels

Jun 12, 2011

Statsmodels: statistical modeling and econometrics in Python

scikit-learn

scikit-learn: machine learning in Python. Contribute to scikit-learn/scikit-learn development by creating an account on GitHub.

Tutorials

Common pitfalls in the interpretation of coefficients of linear models — scikit-learn

In linear models, the target value is modeled as a linear combination of the features (see the Linear Models User Guide section for a description of a set of linear models available in scikit-learn...

Linear Regression — Interpretable Machine Learning (Molnar)

Documentations

scikit-learn Linear Models

Scikit-learn Developers•Jan 1, 2007

Related Techniques

Name	Description	Assurance Goals
Mean Decrease Impurity	Mean Decrease Impurity (MDI) scores how much each feature contributes to a tree-based model, such as a random forest, by looking at how the model's decision trees split the training data. Each split sends cases into two groups, and a good split makes each group more uniform in its outcome, which is a drop in impurity, measured with a score such as Gini impurity. For each feature, MDI adds up the drops from every split that uses it, weights each by the share of training cases reaching that split, and averages over all trees. The scores are usually scaled to sum to one, so they show relative importance. Because they come from the training data, they describe how the model was built, not how well each feature predicts on new data.	Explainability
Generalized Additive Models	A generalised additive model (GAM) builds its prediction by adding a baseline and one smooth curve for each input feature. It extends linear and logistic regression by letting each feature's effect bend, so the model can follow patterns such as risk that climbs steeply after a certain age. Each curve is learned from the data with a smoothing method, such as splines (flexible curves built from joined polynomial pieces). Plotting the curves shows how each feature moves the prediction across its range, so the model can be read directly without a separate explanation method. Because the curves are added together, the model does not capture the joint effect of two features unless a combined term is added by hand.	Transparency Explainability
Ridge Regression Surrogates	A ridge regression surrogate approximates a complex model with a linear equation fitted to the complex model's own predictions. The complex model is run on a set of inputs, and ridge regression, a linear model with a penalty that keeps coefficients stable when features are correlated, is fitted to reproduce its outputs. Each coefficient then shows roughly how far a feature moves the complex model's output, averaged over those inputs. Fidelity, often reported as R-squared (the share of the model's variation the surrogate reproduces on held-out inputs), shows how far the coefficients can be trusted. They describe the complex model, not the real world.	Explainability Transparency