Presenting Trustworthy Assurance—A Framework and Methodology¶

Designing, Developing, and Deploying Trustworthy Digital Mental Health Technologies¶

Designing, developing, and deploying an AI system is not a one-person task!¹ The stages and activities that comprise a typical AI project lifecycle involve a wide-ranging set of skills and capabilities. These skills are encapsulated within a variety of roles, including 'project commissioner', 'product manager', 'data protection officer', 'data scientist', 'system architect' and 'software engineer'. And, these roles are interwoven such that they create an irreducible and collective responsibility that spans the entire project lifecycle, and may span multiple teams and organisations.

Figure 2.1 presents a simplified model of a typical research or innovation lifecycle for a data science or AI project lifecycle, to help gain a purchase on these interweaving roles, skills, and responsibilities².

The model represents three over-arching stages of (project) design, (model) development, and (system) deployment. For each stage, there are corresponding activities, detailed in Table 2.1. The project lifecycle is depicted as a circular process to highlight the fact that responsibility is ongoing and does not end once a system has been implemented or put into deployment. Rather, responsible (and trustworthy) approaches to research and innovation require consideration of how a technological system may need to be monitored and updated once in production, and removed and replaced once it reaches the end of its lifecycle.

Figure 2.1: a model of a typical project lifecycle for a data-driven technology, detailing the overarching stages of design, development, and deployment and their constitutive activities (reprinted from Burr and Leslie, 2022)

Table 2.1: Project Lifecycle Activities

To see how this model can help us understand the interwoven nature of responsibility, consider the following example. An organisation wants to implement a speech recognition algorithm within a service they are developing for online counselling. However, there is no one in the organisation with the relevant expertise to collect data and train a model from scratch. Therefore, they choose to procure a pre-trained model from another company. This means that a significant portion of the project lifecycle—from Data Extraction or Procurement to Model Documentation—will have been carried out by a separate organisation.

Although the specifics of the relationship between the two organisations will complicate forms of responsibility, such as legal duties or obligations, this need not concern us here. Instead, we can focus on how the initial organisation who has chosen to undertake the project (e.g. the product owner or commissioner) can use the project lifecycle model to a) identify and analyse their own responsibility and how it intersects with the responsibilities of others, and b) how this necessitates a process of trustworthy communication and assurance.

Reflective and Anticipatory Deliberation¶

At the start of a project, while activities such as planning and initial evaluation of feasibility are being conducted, the project lifecycle model can be used to structure reflective and anticipatory processes of deliberation among the project team. For instance, the team could use the model to identify and evaluate potential actions and decisions that are likely to emerge during specific activities, such as which data types may be required and whether stakeholders or users will consent to these data being collected and analysed (a reflective and anticipatory exercise). As this example suggests, the project team may carry out the preliminary deliberation, but additional stakeholders will need to be engaged to thoroughly evaluate the ethical, legal, and social permissibility and acceptability of the project.

Consider another example. A team of developers working for a commercial organisation have identified a risk associated with an AI system they have developed, which they claim is able to detect emotions. They have been approached by a healthcare provider who wish to procure and implement their system into a video consultation service to help their counsellors better understand the emotional and behavioural responses of their patients during an initial assessment. However, the developers did not evaluate their model (during Model Testing or Validation) using a dataset that is representative of the patient population intended by the healthcare provider that has approached them (i.e. individuals that are likely to be suffering from a mental health issue). Therefore, the developers are unable to make any claims about the generalisability of their model to this new population. Moreover, neither the developers nor the healthcare provider have engaged the relevant stakeholder groups during Project Planning to determine if this would be an acceptable use for their system. As such, additional activities would need to be carried out to determine the full scope of the risks and possible harms that could arise from the use of this technology. This would likely require the procuring organisation (i.e. the healthcare provider) to set clear requirements for what forms of evidence would be required from the developers (e.g., at Model Documentation), and to determine clear boundaries and thresholds for whether the project should proceed.

As this example illustrates, the project lifecycle structure can help support forms of reflective and anticipatory deliberation that help instantiate a responsible ecosystem of research and innovation. And, it can also help identify points in the lifecycle where structured and transparent communication between teams and organisations may be crucial.

In addition, there is a further purpose for the project lifecycle model that will become clearer in the next sections: the identification of actions and decisions that generate forms of evidence that provide justificatory support for trustworthy assurance.

What is Trustworthy Assurance?¶

Trustworthy assurance is a procedure for developing a structured argument, which provides reviewable (and contestable) assurance that a set of claims about the ethical properties of a data-driven technology are warranted given the available evidence.

This definition captures three important and interlocking components of trustworthy assurance:

1. A structured argument comprising linked claims and evidence that collectively justify a top-level goal
2. A procedure for developing an assurance case, which represents the argument either formally and/or visually
3. Agreed upon standards for reviewing and evaluating the argument

Figure 2.2: a schematic showing the three interlocking components that support trustworthy assurance.

Argument¶

Trustworthy assurance is a form of argument-based assurance. It uses a structured type of documented argumentation, known as an assurance case, as the primary means for providing assurance that a goal has been obtained, based on the claims and evidence presented.

There are three basic elements of an argument:

1. A claim about the goal to be established (supported by descriptions of the system and the context in which the system is intended to operate)
2. A set of property claims about the project or system that collectively specify and operationalise the goal
3. A set of evidential claims that jointly establish the validity of the property claims

Box 2.1 Elements of a Trustworthy Assurance Case

The following table provides summary information about the typical elements of a trustworthy assurance case. NB: the colours do not mean anything. They are purely used as a visual aid to help differentiate the elements.

Element Name	Description	Icon
Goal Claim	A claim about an ethical goal of the DMHT, which the assurance case attempts to justify has been established on the basis of the evidence and argument provided.
System Description	A short description about the DMHT, including any central algorithmic techniques.
Context Description	A short description about the intended context of use for the DMHT, including the users of the system (e.g. healthcare professionals).
Property Claim	A claim about how the ethical goal has been implemented or opertaionalised, which references a property of the system or project, e.g. an action that was undertaken during the model's development.
Evidential Claim	A specific claim about some evidence, which serves to establish the validity of the higher level property claim.
Evidential Artefact	A description of the evidence referred to by the above evidential claim, including a link to the relevant document where available.

The following figure constitutes a simple (but incomplete) argument, showing the relationship between the three central elements.

Figure 2.3: a portion of a simplified assurance case on respect for privacy

Here, the goal that is being established relates to the project team's ambition to 'respect user privacy'. They argue that this is achieved by adherence to data minimisation principles—a claim about a property of the project's governance and the system's design. Evidence of this adherence is also provided.

While useful as an illustration, this example is too simple to constitute a full-fledged assurance case because it reduces the concept of 'respect for user privacy' to a single principle (i.e., data minimisation). Although this claim may be relevant, on its own it is insufficient. We can, for example, consider an app that collects no personal data but still violates reasonable expectations of privacy by routinely notifying and disturbing users.

A more detailed procedure is required, therefore, to help project teams identify the set of property claims that a) specify and operationalise the top-level goal and b) collectively justify the goal.

Procedure¶

The procedure advocated for trustworthy assurance is the anticipatory and deliberative exercise introduced above, which incorporates inclusive and accessible forms of stakeholder engagement.

By using the project lifecycle model as a scaffold for anticipatory reflection and stakeholder-informed deliberation, project teams are able to answer the following questions:

1. Which claims (that may emerge from participatory deliberation) are necessary and sufficient to specify and justify the top-level goal?
2. How do these claims relate to one another?
3. What evidence is required to demonstrate the validity of the claims being made?

Consider the following example. A company wishes to develop an assurance case that shows how their system, which uses a ML algorithm to predict whether users of an online betting platform are "problem gamblers", can generate results that are explainable to their users. They decide this is an important ethical goal for an assurance case, because they want to be able to provide accessible forms of communication to any user that they contact on the basis of their algorithmically-generated prediction.

They start by formulating the following goal statement, which sets out an ambitious objective to achieve:

"An explanation of how our system predicts whether a user is a "problem gambler" can be provided to all users of our platform."

Next, they consider which potential actions or decisions taken throughout the stages of the project lifecycle could be relevant to the specification and justification of this goal. For example, they flag that results from a series of planned workshops to be carried out with representative users during their Project Planning activities may be relevant.

Following the delivery of these workshops, it turns out that plain language explanations are preferable to detailed explanations of the algorithmic techniques, but that users were more trusting of these explanations when they knew they had been independently validated by a professional auditor. This result influences which machine learning algorithm the team go on to select during the Model Selection and Training activities, and which features they report on during Model Documentation for the independent audit.

Once the team have reflected on all the stages of the project lifecycle, and carried out the corresponding activities, they recognise that there are two broad sets of claims. One set of claims are about the design choices made during the project, which support accessible explanations for users. The second set are about the interpretability of the system, which are relevant for professional auditing and assessment of the system. Categorising the claims in this manner helps the company determine what evidence will be needed for each claim, and how best to structure the argument. At this point, most of the evidence has already been generated as a byproduct of the team's work, so this stage is primarily a matter of collection, curation, and communication (through documenting an assurance case).

The following figure summarises the points made in the above example.

Figure 2.4: a portion of the assurance case for this hypothetical project.

The final component of trustworthy assurance relates to standards.

Standards¶

Standards support the development and refinement of best practices and codes of conduct. There are standards for measurement (e.g., universal scales), procedures (e.g., manufacturing), and assessments (e.g., risk and impact assessment), and much more. Here, we are interested in standards as they pertain to evidence and claims.

Evidential standards can refer to both the identification and evaluation of evidence.

Standards for identifying evidence are common in areas such as law where rules exist to determine what constitutes relevant, material, and admissible evidence.

Similarly, standards for evaluating the quality of evidence are well-established in domains such as scientific research, where various procedures or methods are held to produce reliable forms of evidence (e.g. peer review or randomised controlled trials), and in risk assessment and management (e.g. standardised guidelines on risk management for systems and software engineering).⁶

As distinct communities of practice develop and emerge within digital mental healthcare, we would expect standards and best practices to evolve to help with both the identification and evaluation of evidence. Subsequently, this would help support the development of trustworthy assurance in domains such as digital mental healthcare because specific types of claims or evidence would be recognised as more reliable forms of evidence-based assurance.

It should be noted, however, that regulators and developers are not starting with a blank slate. There are many relevant standards that exist today, and new standards are emerging to support the procedure of constructing a trustworthy assurance case.⁷ We will consider some of these standards in Chapter 3.

Argument Patterns¶

Claims as Reasons¶

Trustworthy assurance is a process of giving and justifying claims about choices made during the design, development, and deployment of DMHTs. These claims can be viewed as a series of reasons for why a particular decision was made.

To see why, let’s assume that an organisation is in the process of procuring an AI-enabled chatbot to provide therapeutic support to service members returning from deployment.⁸ As this technology is new and relatively untested, the organisation has a series of questions for the developers.

1. "Why should we license this digital system instead of investing in traditional forms of talk therapy?"
2. "Why have you chosen a female avatar as your virtual assistant?"
3. "How did you measure and validate the clinical efficacy of the system for different subgroups to ensure that it is fair?"

In answering these three questions, the developers would be giving reasons (supported by evidence) for their actions—reasons that would need to be accepted by the procuring organisation to be relevant and justifiable. This perspective on claims emphasises one of their most vital roles as publicly contestable reasons. That is, whether a claim or set of interrelated claims are valid in the context of an assurance case is, in part, conditional on whether they are accepted as reasonable justifications by those who are tasked with evaluating the assurance case⁹.

Let’s look at another example. Consider the following section of an assurance case for the aforementioned chatbot, which the developers have produced for the procuring organisation:

Figure 2.5: a portion of an assurance case for a chatbot¹⁰

As we have discussed, the organisation responsible for evaluating the trustworthiness of the AI system has to determine whether the evidential claim (EC1) is a reasonable choice to justify its parent claim (C2). They may, for instance, argue that EC1 is a reasonable claim, but nevertheless argue that it is insufficient on its own to justify the claim that the chatbot is an "accessible" alternative to human-led therapy. Alternatively, they may claim that it is not reasonable on the grounds that the ratings given by the service members are not relevant to establishing whether the chatbot is an “accessible form of therapy” but merely that the avatar is “friendly and realistic”. This example highlights a potential challenge associated with the development of assurance cases: determining what constitutes relevant, sufficient, and reasonable claims.

In the context of safety assurance, a large body of guidance has been established to help developers assess what claims they will need to establish and justify, and a key part of this guidance is the development of argument patterns.

What are argument patterns?¶

Argument patterns are starting templates for building assurance cases. They identify the types of claims (or, the sets of reasons) that need to be established to justify the associated top-level normative goal. Figure 2.6 shows an example argument pattern for the technical goal of interpretability.¹¹

Figure 2.6: a pattern for an interpretability case (reprinted from Ward and Habli, 2020)

The pattern depicted in Figure 2.6 shows a template for an assurance case that serves to justify the following top-level goal:

"The {ML Model} is sufficiently {interpretable} in the intended {context}".

Here, the curly brackets serve as placeholders for specific variables that are properly established when a full assurance case is developed. A notable contribution of this pattern is the identification of three essential aspects of interpretability:

1. Right Method: The right interpretability methods are implemented, i.e. the correct information is faithfully being explained.
2. Right Context:
   • Time: Interpretations produced at the appropriate times.
   • Setting: Interpretations are available in the right setting.
   • Audience: Interpretations produced for the right audience.
3. Right Format: The interpretability methods are presented in the right format for the audience.

These three essential aspects subsequently serve to delineate the more detailed argument and evidential claims at the lowest levels.

Argument patterns, such as the one above, are helpful for the following reasons:

• They provide a consistent and systematic approach for the reflective and deliberative activities carried out across a project’s lifecycle.
• They speed up the process of developing assurance cases.
• They provide reusable structures that, if used widely throughout a domain, could establish best practices.

But where do they come from?

Generalisable Patterns¶

In the case of the argument pattern from Figure 2.6, this pattern was proposed by the authors as a means to address a gap in the safety assurance of ML systems. As experts in their domain, and as a peer-reviewed contribution, this is a valid source for an argument pattern.

However, an alternative (though not entirely disconnected) means for achieving generalisable structures and patterns is through participatory engagement from stakeholders and affected users, perhaps building sample assurance cases and then extracting common themes. This is the method that we have explored in the current project on Digital Mental Healthcare and subsequently propose as a procedure for Trustworthy Assurance.

Much like ML algorithms, humans have remarkably effective (but biased) pattern recognition capabilities, some of which underpin our assessment and internalisation of ethical, legal, and social norms. As an example, James W. Nickel says of human rights:

"We can think of the emergence of a human right as the coming together of the recognition of a problem; the belief that the problem, is very severe; and optimism about the possibility of addressing it through social and political action at national and international levels." ¹²

Similarly, we can think of ethical and social norms as the shared recognition and subsequent externalisation of beliefs and attitudes towards events as diverse as acceptable etiquette during a dinner party through to permissible forms of punishment for various transgressions.

This understanding of the emergence of norms is crucial to ensuring the relevance, sufficiency, and reasonableness of evidence, and the legitimacy of corresponding trustworthy argument patterns.

In terms of the emergence of argument patterns, the three elements that we have explored already are, again, important: the top-level normative goal, the property claims, and the evidential support. Let's take each of these in turn.

The phrase 'trustworthy assurance' creates a wide scope for top-level goals that may be deemed relevant to establishing trust (e.g. sustainable digital platforms, accountable methods of data governance, fair classifiers, and explainable decision support systems). As trustworthy assurance cases are developed for data-driven technologies, it is likely that we will see certain goals emphasised (and re-emphasised) over others. In turn, these normative goals will orient other projects and help cultivate best practices. In related work, we have proposed a series of ethical principles that have been developed to provide actionable insights and safeguards on responsible research and innovation in data science and AI.¹³ They are known as the SAFE-D principles:

• Sustainability
• Accountability
• Fairness
• Explainability
• Data (Quality, Integrity, Protection and Privacy)

These principles have been refined and validated in a wide range of domains, and were originally based on a broad understanding of the typical harms and benefits associated with data-driven technology (e.g. starting from the felt injustices or needs of users and stakeholders, and developing principles to reflect these challenges). Therefore, unlike alternative frameworks they are tailored to the specific needs and challenges of responsible, trustworthy, and ethical data science and AI, rather than, say, importing or revising existing frameworks such as biomedical ethics.¹⁴

However, the SAFE-D principles were designed to be domain-neutral starting points. That is, we did not presume that these principles would capture the ethical, social, and legal values that are dominant in digital mental healthcare. Instead, the present project undertook a process of exploratory engagement and participatory design to explore which ethical values and principles were relevant to the specific context of trustworthy digital mental healthcare, and whether specific SAFE-D principles captured these. We will return to this point in Chapter 4 where we analyse our findings from the project's workshops.

Turning now to the property claims and supporting evidence, as assurance cases are communicated for specific goals we will likely see sets of property claims and supporting evidence used more than others as justifiable and accepted reasons for establishing the respective goal. For instance, as developers focus on goals like 'accountability', core attributes of the system and project are likely to be emphasised as relevant targets (e.g. constructing traceable data pipelines, establishing mechanisms to support auditing processes, ensuring accessible documentation).

Returning to the SAFE-D principles once more, we have previously developed a set of core attributes for each of the principles, which a) identify the types of properties that need to be established in a project or a system to ensure the relevant goal is obtained and b) the stages of the project lifecycle where actions can be taken to implement the respective property. Table 2.2 shows an example of the core attributes for 'sustainability'.

Table 2.2: a summary of the core attributes for the principle 'sustainability'

Again, we are not proposing that these principles and core attributes should be adopted in digital mental healthcare as the respective goals, claims, and evidence. However, they could provide a starting point for the refinement of domain-specific principles while argument patterns emerge and become crystallised.¹⁵

Figure 2.7 offers a simple graphic to help visualise this process as it relates to trustworthy assurance.

Figure 2.7: process of consensus formation for ethical principles as constraints on trustworthy assurance

These preliminary remarks about trustworthy assurance serve as a foundation for understanding and contextualising our project's research and the recommendations we derive from our findings.