Developing Trustworthy Assurance—Argument Patterns for fairness and Explainability¶

Co-designing argument patterns¶

While the assurance methodology is a tool in its own right, there is a missing component that was highlighted in Chapter 2: argument patterns.

You may recall that argument patterns are reusable structures that serve as starting templates for building assurance cases. They identify the types of claims (or, the sets of reasons) that need to be established to justify the associated top-level normative goal. And, in doing so, they set useful constraints on both the deliberative process and evidence-generating and evidence-gathering exercises. We say more about the evidential generation and selection process towards the end of this section.¹

Before this, we present and explain two argument patterns for use in the assurance of DMHTs. The first is for assurance cases that address and justify the fairness of a DMHT; the second is for cases that address and justify the explainability of systems.

Fairness¶

In the context of data-driven technologies, a core attribute of fairness is the equal distribution of risk and benefit across all groups of affected users. For instance, a technology that was highly accurate for users aged between 18-30 but became decreasingly accurate for older individuals would be unfair to those from higher age brackets.

The differentiation between risk and benefit leads to a subsequent distinction between corresponding duties or obligations (e.g. legal duties). Where there is a duty to ensure that a particular group of users are not exposed to disproportionate risks of harm, this can set up a so-called “negative duty” (e.g. a duty for a developer to not discriminate). However, negative duties often set only the minimal ethical standards. In other words, one can build a non-discriminatory service that does not harm anyone, but similarly benefits no one or benefits only a small portion of users.

Therefore, corresponding “positive duties” also exist to promote beneficial outcomes, sometimes focusing on those who are most disadvantaged—a so-called “prioritarian” approach to ethics (i.e. prioritising those who need the most support). A good example of this duality is the Public Sector Equality Duty, which sets the following objectives for all public authorities:

• eliminate discrimination, harassment, victimisation and any other conduct that is prohibited by or under the Equality Act 2010;
• advance equality of opportunity between persons who share a relevant protected characteristic and persons who do not share it;
• foster good relations between persons who share a relevant protected characteristic and persons who do not share it.

Notice that these objectives can be seen as having both negative and positive aspects to them (e.g. ‘eliminate discrimination’ as ‘opposed to advance equality’).²

In the context of mental healthcare, where stigmatisation prevents many vulnerable individuals from accessing care and discrimination exacerbates symptoms for already marginalised or minoritised groups, both types of duty are absolutely crucial. However, acting upon positive duties is not without its challenges.

In our workshops, for instance, a key concern that was emphasised was the degree to which the delineation of "good" or "desirable" mental health outcomes, for the purpose of evaluating the impacts of a system, could adequately take into account the subjective nature of mental health and well-being. For instance, while there may be widespread agreement about what constitutes undesirable outcomes (e.g. chronic stress, suicide), the range of positive outcomes for mental health (or well-being) are varied and multitudinous (to echo back to Whitman’s quote that started this report), and the experience and process of recovery can also mean many different things to people³. A failure to account for such issues can introduce a further source of bias into the design of a system (e.g. how the problem it seeks to address is formulated) which in turn could lead to unfair outcomes that only benefit a small group of users with aligned goals.

The pattern that has been developed, through participation of stakeholders and users, attempts to address both negative and positive duties, while also making room for core attributes such as user autonomy. In the context of mental healthcare, this inclusion of autonomy can be problematic in the most severe cases where it is simply not viable (e.g. severe forms of psychosis). However, recall that a pattern is a starting point or scaffold; it is not a checklist of jointly sufficient claims and supporting evidence. Therefore, if a particular type of claim is inappropriate due to contextual factors that are determined during project scoping or stakeholder engagement, it can be adjusted as necessary.

Fairness Pattern¶

Figure 5.1: A pattern for designing, developing, and deploying fair digital mental health technologies ( click image to expand; download here)

The goal claim in this argument pattern addresses distributional concepts of fairness (e.g. whether harms and benefits are shared equally), while also acknowledging broader conceptions of social justice (e.g. representational harms to marginalised groups).

Unlike traditional safety cases, which often include System Description and Context of Use placeholders, this pattern also includes a Stakeholder component to emphasise the importance of engagement. Here, the term 'stakeholder' should be treated in as inclusive a manner as possible, and not only the direct users of the technology.

The goal is broken into four higher-level property claims and their respective sub-claims, which we group according to the following core attributes of the Fairness principle as specified and operationalised in the context of digital mental healthcare:

• Argument over bias mitigation
• Argument over non-exclusion
• Argument over non-discrimination
• Argument over equitable impact

Argument over bias mitigation¶

This argument emphasises identification, evaluation, and mitigation activities. As there are too many biases to incorporate into a single pattern, this argument instead draws attention to transparent and accountable methods that allow stakeholders to determine if the set of biases that have been addressed are sufficient to address their concerns. This argument is supported by additional tools, such as a bias self-assessment tool that outlines social, statistical, and cognitive biases that can affect the lifecycle of a machine learning or AI system project.⁵

Argument over non-exclusion¶

This argument sets out another negative duty to consider wider social impacts of digital technologies, and prompts developers to ensure they are not contributing to growing socioeconomic inequalities (i.e. the digital divide) by overlooking important social determinants (e.g. education, poverty). In some instances, this may require nothing more than ensuring that UI/UX design choices do not exclude those who have additional accessibility requirements. However, the argument also sets up a duty to consider how a system being developed for use within a public healthcare system, for example, does not create an unsustainable multi-tiered approach, where some users are excluded from accessing a better performing technological service and forced to use comparatively inferior options.

Argument over non-discrimination¶

This argument addresses the better known obligations, such as ensuring that members of protected groups are not discriminated against. Much of the FairML literature, which has provided useful categorisations of formal criteria (i.e. independence, sufficiency, and separation) and the respective (statistical) notions of individual and group level fairness (e.g. demographic parity, equalised opportunities, counterfactual fairness) are relevant here. And, indeed, a requirement to explain the choice of any statistical measures used during the development of a predictive model (e.g. classifier) is included. Again, there are useful tools and taxonomies that offer further guidance on these decisions. However, our pattern also urges project teams to reflect upon other patterns of discrimination, marginalisation, and minoritisation, which can exacerbate mental health issues, but which fall beyond protected characteristics that lie outside the scope of the Equality Act 2010 (e.g. housing, employment, social support). Doing so will typically require engagement of domain experts where such expertise does not exist within teams.

Argument over equitable impact¶

Finally, this argument references positive duties that matter in the context of digital mental healthcare specifically. Key to this is the consideration of ethical values such as autonomy and self-determination, and the prioritarian weighting that was mentioned previously. However, there is also an aspect of our Sustainability principle that trickles into this argument. This is important in the context of digital mental healthcare because of associated risks that a) positive effects diminish over time (e.g. behavioural nudges or habit forming techniques that become ineffective over time)⁶ and b) negative impacts worsen and compound (e.g. prolonged use of social media worsening anxiety or depression)⁷. Studies have already criticised the evidence base of mental health apps and services⁸, especially for a lack of reliable longitudinal evidence, so drawing attention to sustainable impacts and setting up requirements for continuous monitoring is vital to maintain trust and also ensure that specific users are not locked in to services or technologies that degrade in quality or efficacy over time (e.g. apps that start by offering free services to leverage network effects only to force subscriptions at a later date).

Explainability¶

Much like fairness, explainability has become a popular and thriving area of research (e.g. so-called XAI). And, also like fairness, it is a normative goal that encompasses a range of significant concerns and salient ethical values.

The most obvious conceptual distinction is between interpretability as a core component of explainability. Whereas the former is to often treated as the ability for developers or users to understand the inner workings of algorithms (or inability in the case of complex, non-linear techniques), the latter refers to an interpersonal ability to communicate knowledge in a manner that is accessible to those who may be asking questions about a system (e.g. patients asking for explanations from a clinician). Although statistical techniques help significantly in the case of the former, they are more limited in their ability to support the latter where a more diverse range of users are likely.

Explainability Pattern¶

Figure 5.2: A pattern for designing, developing, and deploying explainable digital mental health technologies ( click image to expand, download here)

In previous guidance¹⁶, we have distinguished between two sub-categories of explanations:

1. Process-based explanations of AI systems are about demonstrating that you have followed good governance processes and best practices throughout your design and use.
2. Outcome-based explanations of AI systems are about clarifying the results of a specific decision. They involve explaining the reasoning behind a particular algorithmically-generated outcome in plain, easily understandable, and everyday language.

These categories are reflected in our pattern, where they form 'intermediate arguments' that help refine the goal claim and also serve as scaffolding for the main arguments.

As with the fairness pattern, placeholders for System Description and Context of Use and Stakeholder are also included.

The intermediate arguments are then broken into four higher-level property claims and their respective sub-claims, which we group according to the following core attributes of the Explainability principle as specified and operationalised in the context of digital mental healthcare:

• Argument over transparency and accountability
• Argument over responsible project governance
• Argument over informed and autonomous decision-making
• Argument over sustainable impact

Argument over transparency and accountability¶

This argument addresses the processes and mechanisms that have been undertaken throughout the project lifecycle to establish sufficient forms of transparency and accountability. This includes documentation relevant to the identification of responsible project members, as well as choices made about data (e.g. why certain data types were included or excluded). Importantly, this argument also recommends the inclusion of a statement about sources of funding and conflicts of interest, which was an important matter for trustworthiness that arose during our engagement with participants with lived experience of DMHTs.

Argument over responsible project governance¶

This argument is more comprehensive than the others, and so is further split into three sub-arguments:

1. Sub-argument over meaningful engagement: here, meaningful engagement can be seen to include participation in decisions about the formulation of the problem that a DMHT is expected to address, as well as issues of data usage—both of which affect later stages of the project lifecycle and the final behaviour of the deployed system.
2. Sub-argument over interpretability: sufficient levels of accuracy and the potential trade-off with interpretability can require high levels of technical and data literacy. Therefore, this argument focuses on the requisite information that is needed to support explainability (recall earlier distinction between interpretability and explainability).
3. Sub-argument over accessible communication: the previous sub-argument feeds into this sub-argument, which focuses on how 'accessible' forms of communication will achieved, and the challenges of communicating probabilistic information. Ultimately, this sub-argument will depend on decisions reached and evidence obtained through consultation with intended users, as well as on the basis of knowledge about any time-constraints presented by the context of use (e.g. urgency in high-risk care environments).

Argument over informed and autonomous decision-making¶

The core attribute motivating this argument is shared with our fairness pattern. Here, the argument emphasises the importance of explanations that refer to the observed behaviours or outcomes of the system. For instance, one of the claims is intended to ensure that explanations are "sufficiently expressive", without overwhelming the user with unnecessary or overly-complex information. This will depend on the intended user and context of use. However, to supplement this claim, emphasis is also placed on the ability for user to challenge outcomes, rather than just having them explained without an option to contest.

Argument over sustainable impact¶

The final argument also follows the theme of the fairness pattern, but rather than addressing equitable impact, it focuses on sustainable impact¹⁵. This is important because explanations are sometimes used as a means to justify why a specific norm was transgressed (e.g. why you were late). However, over time, if the same explanation is provided without a change to the offending behaviour, the explanation loses its validity. A similar risk is present in the automated delivery of explanations by algorithmic systems. For example, if an AI chatbot continues to offer the same inaccurate and irrelevant explanations, it is likely to lose the trust of a user. Therefore, assessments about the impact of explainable AI need to account for longer-term dynamics to ensure that the relevant systems are sustainable over time.

The inclusion of clinical safety and efficacy should not suggest that these goals are not significant in their own right. In fact, we would advocate for a separate assurance case (and corresponding pattern) on these goals specifically. Instead, reference is simply made to the need to ensure that some form of explanation is provided to stakeholders.

Evidential Considerations¶

Neither of the patterns above include prescriptions about specific evidential artefacts that could be used to ground the assurance case. There are two reasons for this intentional omission:

1. Prescribing specific forms of evidence is too difficult outside of highly constrained contexts where there are clear details about a) the intended use context, b) the type of ML/AI technique being used, and c) the intended users and target audience.
2. Developers and regulators should be free to determine the appropriate forms of evidence, based on developing best practices and standards, many of which do not exist at present.

However, there are a couple of general remarks that can be made, as well as some suggestions for further resources.

First, as we have argued elsewhere¹⁷, the generation, evaluation, and selection of evidence can be guided by the following considerations:

1. Is the evidential artefact/claim relevant to the parent claim?
2. Is the evidential artefact/claim (or set of artefacts/claims) sufficient to justify the parent claim?
3. Is there sufficient probative value in the overall assurance case to justify the top-level normative goal?

Outside of the regulatory considerations already mentioned above, there have been several developments in recent years to help organisations address these considerations. Some examples (among many) include:

• Model Cards for Model Reporting: templates for model documentation, which include ethical considerations alongside statistical information to support reuse.
• Responsible AI Licensing: licenses that help developers restrict the use of their AI technology in order to prevent irresponsible and harmful applications.
• Data Hazards: a set of labels that enable project members to make decisions about the risks of data-driven technologies using a shared vocabulary
• Assurance of Machine Learning in Autonomous Systems (AMLAS): a methodology for assuring the safety of ML systems, with systematic means for evaluating processes such as model testing or verification.
• Algorithmic Transparency Standard: a template for organisations to use when choosing to publish information about how they are using algorithmic systems to aid decision-making.

Typically, these tools, methods, or templates exist to help organisations and project teams address a specific challenge (e.g. licensing, model documentation). The framework and methodology we have presented in this report is designed to work alongside these tools, but it also goes further by helping teams organise them according to a particular goal or objective (e.g. fairness). As such, our framework and methodology is broader in scope and offers a systematic means for choosing when to use specific tools throughout a project's lifecycle and how to bring the documented evidence together to create a trustworthy and justifiable assurance case.