Bias Self-Assessment¶
The following self-assessment is designed to help you and your team identify, evaluate, and manage the risks associated with a variety of biases that can occur before and throughout your research project workflow. Diligent and end-to-end bias-awareness is important. Not only can it help you to improve the accuracy and robustness of your results, but it can also assist you in recognising and appropriately communicating the limitations of your model and its outputs.
A short definition and summary of each of the biases is included, and can be revealed by clicking on the ‘click to reveal’ toggle. In addition, there is a set of questions (or, deliberative prompts) for each bias. These are meant to help you evaluate the extent to which the potential bias is relevant for your data, analysis and research methods. Such questions should help to key you in to how best to manage the potential impact of the bias on your project and its results.
The biases are grouped according to the stage of the project workflow where they are likely to have the biggest impact, and where your project team can act to reflect upon and take actions to minimise the negative effects of the respective bias. In general, the earlier these biases are considered the better. As such, you are urged to consider all of the biases initially and then return to individual biases at the relevant stage.
In addition to the use of the hypothesis service that is enabled on all of the pages there is also a form that can be downloaded and used to record the steps that you have taken to identify, evaluate, and mitigate bias throughout your project.
Source Material¶
The following references were used in the process of selecting and revising this list of biases:
Suresh, H., & Guttag, J. V. (2020). A Framework for Understanding Unintended Consequences of Machine Learning. ArXiv:1901.10002 [Cs, Stat]. http://arxiv.org/abs/1901.10002 [SG20]
Rajkomar, A., Hardt, M., Howell, M. D., Corrado, G., & Chin, M. H. (2018). Ensuring Fairness in Machine Learning to Advance Health Equity. Annals of Internal Medicine, 169(12), 866. https://doi.org/10.7326/M18-1990 [RHH+18]
Oxford Centre for Evidence Based Medicine. (2020). Catalogue of Bias. https://catalogofbias.org
Kliegr, T., Bahník, Š., & Fürnkranz, J. (2020). A review of possible effects of cognitive biases on interpretation of rule-based machine learning models. ArXiv:1804.02969 [Cs, Stat]. http://arxiv.org/abs/1804.02969 [KBahnikFurnkranz20]