RuleFit
Description
RuleFit builds a predictive model that a person can read: a short list of if-then rules plus a few ordinary feature weights. It first trains an ensemble of decision trees (many small trees whose outputs are combined) and turns each branch into a candidate rule, such as 'IF age > 50 AND income < £30k'. It then fits a linear model over those rules and the original features using the lasso, a penalty that sets many weights to exactly zero, so only the most useful terms remain. The surviving rules and weights can be read directly, and each term has an importance score, overall and for single predictions. Fitted to real outcomes, the result can replace an opaque model; fitted to an opaque model's predictions, it is a surrogate that approximates that model, and its fit should then be checked.
Example Use Cases
Explainability
A telecoms analytics team fits RuleFit to the predictions of its gradient-boosted churn model rather than to the churn labels, to get a short list of rules that approximates the model. Marketing staff read the top rules, and the team reports how closely the rules reproduce the model's predictions before anyone relies on them.
A lender replaces an opaque default-risk model with a RuleFit model fitted to repayment outcomes. For one declined application, the importance scores show which few rules and income or debt terms drove its score, and the validation team checks them against credit policy.
Transparency
A hospital team builds an admission-risk model as a RuleFit model fitted directly to patient outcomes, so that clinicians and the ethics committee can read the whole model, a few rules such as 'IF systolic_bp > 140 AND diabetes = true' plus a handful of weights. The team counts the active rules first and keeps the model only if it is short enough to read.
Limitations
- The lasso can leave many rules active, and a model with hundreds of active terms is no longer readable. Rules also approximate smooth linear trends poorly and need many pieces to do so. Count the non-zero terms before calling the model interpretable.
- Rules overlap and are often correlated, so a rule's weight cannot be read on its own. 'Temperature above 15' is also true wherever 'temperature above 10' is. With correlated inputs, rules can also suggest interactions that are not in the real relationship.
- Tree size sets how long rules can be and which interactions can appear. Shallow trees give short, readable rules but miss high-order interactions, and deep trees capture them at the cost of long rules. No single depth suits every problem.
- Fitted to a black-box model's predictions, RuleFit only approximates that model. Without a fidelity check, such as agreement on held-out inputs, its rules describe the surrogate and may not describe the model they are meant to explain.
Resources
Research Papers
Predictive learning via rule ensembles
General regression and classification models are constructed as linear combinations of simple rules derived from the data. Each rule consists of a conjunction of a small number of simple statements concerning the values of individual input variables. These rule ensembles are shown to produce predictive accuracy comparable to the best methods. However, their principal advantage lies in interpretation. Because of its simple form, each rule is easy to understand, as is its influence on individual predictions, selected subsets of predictions, or globally over the entire space of joint input variable values. Similarly, the degree of relevance of the respective input variables can be assessed globally, locally in different regions of the input space, or at individual prediction points. Techniques are presented for automatically identifying those variables that are involved in interactions with other variables, the strength and degree of those interactions, as well as the identities of the other variables with which they interact. Graphical representations are used to visualize both main and interaction effects.