Why AI assurance will continue to fail unless we start to think holistically - Sam Farrow
AI Assurance can significantly benefit enterprises and society by helping to unlock AI's potential. However, current approaches often lack a holistic view, failing to develop justified trust.
What is the purpose of AI Assurance?
AI Assurance aims to achieve justified trust, address information asymmetries, and effectively measure, evaluate, and communicate evidence, each presenting unique challenges
Achieving Justified Trust
Trust alone is not sufficient for assurance – justification must be provided to allow for stakeholders to have confidence that the system will act in the way it was intended to. Providing sufficient evidence for stakeholders to trust the system's intended behaviour is challenging for complex systems.
Addressing Information Asymmetries
Describing a system's behaviour can reveal gaps or asymmetries in understanding and evidence. By building an assurance case argument, you aim to address these asymmetries to a sufficient extent. However, whilst being able to say that “I know more” is easy, saying “I know enough” is entirely non-trivial.
Measurement, Evaluation, and Communication of Evidence
Evidence is crucial for assurance; however, collecting, evaluating, and communicating it to diverse stakeholders is challenging, especially when presenting limitations.
Why is assurance hard?
Assurance is challenging in any context, with complexities such as:
The diversity of skillsets required which may span technical, ethical, legal and other domain-specific areas
The need for clear, informed and indisputable accountability
The difficulty in perceiving all of the risks and evidence involved when evaluating any sufficiently complex system
What makes AI special?
Adding onto the difficulties of any form of assurance, AI also brings additional challenges. These can broadly be categorised into two areas:
Structural and regulatory gaps (lack of established expertise, geopolitical and security implications, need to capture both development and usage information, challenging market control mechanisms, rapidly evolving and diversifying regulatory landscape)
Technical gaps (rapid pace of model change and capability change, high “minimum viable” level of AI system complexity, “black-box” challenges with model mechanistic explainability, non-validity of conventional test procedures, additional cybersecurity vulnerabilities around both models and data chains, lack of key technical skillsets and investment)
Why a 'one size fits all' approach to AI Assurance fails?
There are a number of proposed approaches to AI Assurance (such as the UK Introduction to AI Assurance, the EU AI Act, ISO 42001, and the NIST AI RMF), but many (whilst providing some benefit) fail to grasp the scale of the challenge for high-stakes AI systems. The limitations of these current methods cluster around four broad themes:
1. Translation and Communication
One key issue with existing methods centres around translation; particularly the ability to translate between high-level, principles-based frameworks and the concrete, technical realities of real-world AI applications. While principles such as fairness, transparency, and ethicality are widely endorsed, organisation often struggle to interpret what these abstract ideals actually require in practice. Further, there is frequently a disconnect between these broad principles and the more established, lower-level assessment techniques used in narrower domains like security or privacy making it difficult to align assurance methods across levels. This results in teams being left with a patchwork of practices that may address some areas of technical risk but fail to link back to demonstrated assurance against endorsed principles.
2. Adequacy
A key question in assurance is “have I done enough to be confident?”. Many of the current risk-based approaches quite effectively produce lots of different risks, but when do we stop? How do we know that further analysis is unlikely to identify an additional risk which would pose a threat to the safe operation of the system? Conversely, how do we prevent ourselves from falling into the ‘paralysis by analysis’ trap by conducting endless iterations of risk-based analysis.
3. Effectiveness
With AI Assurance being an emerging field, there are concerns over the effectiveness of applying a standardised framework of controls. These approaches assume that predefined sets of controls or checklists can adequately cover all risks, without considering the unique characteristics of each model, dataset, or use case. Whilst this seems like an attractive, one-size-fits-all approach, it fails to account for the complexity, and rapid evolution of modern AI systems.
4. Siloed thinking
A final problematic feature of current approaches is siloed thinking. Many standard frameworks and approaches to AI Assurance are shaped by existing organisational structures, leading to assessments that mirror the expertise and focus of individual teams. This often results in ‘risk silos’, where certain types of risks are well-explored while others are overlooked. Such fragmentation prevents a holistic system assessment and obscures cross-cutting risks where, for example, a security breach might introduce bias, transforming a technical failure into an ethics or fairness concern.
What are the characteristics of an effective AI Assurance approach?
What, then, are the fundamental characteristics of an effective AI assurance approach? We propose that there are five key features that are essential to success:
1. Argument-based: the approach must focus on the communication of an assurance argument, with appropriate evidence and justification.
2. Proportionate: the approach must be flexible to allow for the tailoring of effort towards areas of least stakeholder risk appetite and greatest system risk.
3. Holistic: the approach should be focussed on whole-system, whole-lifecycle risk, including robust change management as models, systems, and contexts change and drift.
4. Vertical: the approach must deliver clear and actionable information from low level evidence to high level claims. The argument is clear and traceable, throughout the levels of system abstraction.
5. Robust: the approach provides clear evidence and justification that senior responsible owners can use to defend their decision-making in high-stakes applications
Approaches that do not include these key features are, in Synoptix’s view, unlikely to be able to provide comprehensive AI Assurance. Synoptix emphasises the importance of looking to high-assurance engineering industries, such as aerospace and defence, where these challenges have been prevalent for decades, to provide strong examples of effective assurance cases.
Why AI assurance will continue to fail unless we start to think holistically - Sam Farrow
AI Assurance can significantly benefit enterprises and society by helping to unlock AI's potential. However, current approaches often lack a holistic view, failing to develop justified trust.
What is the purpose of AI Assurance?
AI Assurance aims to achieve justified trust, address information asymmetries, and effectively measure, evaluate, and communicate evidence, each presenting unique challenges
Achieving Justified Trust
Trust alone is not sufficient for assurance – justification must be provided to allow for stakeholders to have confidence that the system will act in the way it was intended to. Providing sufficient evidence for stakeholders to trust the system's intended behaviour is challenging for complex systems.
Addressing Information Asymmetries
Describing a system's behaviour can reveal gaps or asymmetries in understanding and evidence. By building an assurance case argument, you aim to address these asymmetries to a sufficient extent. However, whilst being able to say that “I know more” is easy, saying “I know enough” is entirely non-trivial.
Measurement, Evaluation, and Communication of Evidence
Evidence is crucial for assurance; however, collecting, evaluating, and communicating it to diverse stakeholders is challenging, especially when presenting limitations.
Why is assurance hard?
Assurance is challenging in any context, with complexities such as:
The diversity of skillsets required which may span technical, ethical, legal and other domain-specific areas
The need for clear, informed and indisputable accountability
The difficulty in perceiving all of the risks and evidence involved when evaluating any sufficiently complex system
What makes AI special?
Adding onto the difficulties of any form of assurance, AI also brings additional challenges. These can broadly be categorised into two areas:
Structural and regulatory gaps (lack of established expertise, geopolitical and security implications, need to capture both development and usage information, challenging market control mechanisms, rapidly evolving and diversifying regulatory landscape)
Technical gaps (rapid pace of model change and capability change, high “minimum viable” level of AI system complexity, “black-box” challenges with model mechanistic explainability, non-validity of conventional test procedures, additional cybersecurity vulnerabilities around both models and data chains, lack of key technical skillsets and investment)
Why a 'one size fits all' approach to AI Assurance fails?
There are a number of proposed approaches to AI Assurance (such as the UK Introduction to AI Assurance, the EU AI Act, ISO 42001, and the NIST AI RMF), but many (whilst providing some benefit) fail to grasp the scale of the challenge for high-stakes AI systems. The limitations of these current methods cluster around four broad themes:
1. Translation and Communication
One key issue with existing methods centres around translation; particularly the ability to translate between high-level, principles-based frameworks and the concrete, technical realities of real-world AI applications. While principles such as fairness, transparency, and ethicality are widely endorsed, organisation often struggle to interpret what these abstract ideals actually require in practice. Further, there is frequently a disconnect between these broad principles and the more established, lower-level assessment techniques used in narrower domains like security or privacy making it difficult to align assurance methods across levels. This results in teams being left with a patchwork of practices that may address some areas of technical risk but fail to link back to demonstrated assurance against endorsed principles.
2. Adequacy
A key question in assurance is “have I done enough to be confident?”. Many of the current risk-based approaches quite effectively produce lots of different risks, but when do we stop? How do we know that further analysis is unlikely to identify an additional risk which would pose a threat to the safe operation of the system? Conversely, how do we prevent ourselves from falling into the ‘paralysis by analysis’ trap by conducting endless iterations of risk-based analysis.
3. Effectiveness
With AI Assurance being an emerging field, there are concerns over the effectiveness of applying a standardised framework of controls. These approaches assume that predefined sets of controls or checklists can adequately cover all risks, without considering the unique characteristics of each model, dataset, or use case. Whilst this seems like an attractive, one-size-fits-all approach, it fails to account for the complexity, and rapid evolution of modern AI systems.
4. Siloed thinking
A final problematic feature of current approaches is siloed thinking. Many standard frameworks and approaches to AI Assurance are shaped by existing organisational structures, leading to assessments that mirror the expertise and focus of individual teams. This often results in ‘risk silos’, where certain types of risks are well-explored while others are overlooked. Such fragmentation prevents a holistic system assessment and obscures cross-cutting risks where, for example, a security breach might introduce bias, transforming a technical failure into an ethics or fairness concern.
What are the characteristics of an effective AI Assurance approach?
What, then, are the fundamental characteristics of an effective AI assurance approach? We propose that there are five key features that are essential to success:
1. Argument-based: the approach must focus on the communication of an assurance argument, with appropriate evidence and justification.
2. Proportionate: the approach must be flexible to allow for the tailoring of effort towards areas of least stakeholder risk appetite and greatest system risk.
3. Holistic: the approach should be focussed on whole-system, whole-lifecycle risk, including robust change management as models, systems, and contexts change and drift.
4. Vertical: the approach must deliver clear and actionable information from low level evidence to high level claims. The argument is clear and traceable, throughout the levels of system abstraction.
5. Robust: the approach provides clear evidence and justification that senior responsible owners can use to defend their decision-making in high-stakes applications
Approaches that do not include these key features are, in Synoptix’s view, unlikely to be able to provide comprehensive AI Assurance. Synoptix emphasises the importance of looking to high-assurance engineering industries, such as aerospace and defence, where these challenges have been prevalent for decades, to provide strong examples of effective assurance cases.
Why AI assurance will continue to fail unless we start to think holistically - Sam Farrow
AI Assurance can significantly benefit enterprises and society by helping to unlock AI's potential. However, current approaches often lack a holistic view, failing to develop justified trust.
What is the purpose of AI Assurance?
AI Assurance aims to achieve justified trust, address information asymmetries, and effectively measure, evaluate, and communicate evidence, each presenting unique challenges
Achieving Justified Trust
Trust alone is not sufficient for assurance – justification must be provided to allow for stakeholders to have confidence that the system will act in the way it was intended to. Providing sufficient evidence for stakeholders to trust the system's intended behaviour is challenging for complex systems.
Addressing Information Asymmetries
Describing a system's behaviour can reveal gaps or asymmetries in understanding and evidence. By building an assurance case argument, you aim to address these asymmetries to a sufficient extent. However, whilst being able to say that “I know more” is easy, saying “I know enough” is entirely non-trivial.
Measurement, Evaluation, and Communication of Evidence
Evidence is crucial for assurance; however, collecting, evaluating, and communicating it to diverse stakeholders is challenging, especially when presenting limitations.
Why is assurance hard?
Assurance is challenging in any context, with complexities such as:
The diversity of skillsets required which may span technical, ethical, legal and other domain-specific areas
The need for clear, informed and indisputable accountability
The difficulty in perceiving all of the risks and evidence involved when evaluating any sufficiently complex system
What makes AI special?
Adding onto the difficulties of any form of assurance, AI also brings additional challenges. These can broadly be categorised into two areas:
Structural and regulatory gaps (lack of established expertise, geopolitical and security implications, need to capture both development and usage information, challenging market control mechanisms, rapidly evolving and diversifying regulatory landscape)
Technical gaps (rapid pace of model change and capability change, high “minimum viable” level of AI system complexity, “black-box” challenges with model mechanistic explainability, non-validity of conventional test procedures, additional cybersecurity vulnerabilities around both models and data chains, lack of key technical skillsets and investment)
Why a 'one size fits all' approach to AI Assurance fails?
There are a number of proposed approaches to AI Assurance (such as the UK Introduction to AI Assurance, the EU AI Act, ISO 42001, and the NIST AI RMF), but many (whilst providing some benefit) fail to grasp the scale of the challenge for high-stakes AI systems. The limitations of these current methods cluster around four broad themes:
1. Translation and Communication
One key issue with existing methods centres around translation; particularly the ability to translate between high-level, principles-based frameworks and the concrete, technical realities of real-world AI applications. While principles such as fairness, transparency, and ethicality are widely endorsed, organisation often struggle to interpret what these abstract ideals actually require in practice. Further, there is frequently a disconnect between these broad principles and the more established, lower-level assessment techniques used in narrower domains like security or privacy making it difficult to align assurance methods across levels. This results in teams being left with a patchwork of practices that may address some areas of technical risk but fail to link back to demonstrated assurance against endorsed principles.
2. Adequacy
A key question in assurance is “have I done enough to be confident?”. Many of the current risk-based approaches quite effectively produce lots of different risks, but when do we stop? How do we know that further analysis is unlikely to identify an additional risk which would pose a threat to the safe operation of the system? Conversely, how do we prevent ourselves from falling into the ‘paralysis by analysis’ trap by conducting endless iterations of risk-based analysis.
3. Effectiveness
With AI Assurance being an emerging field, there are concerns over the effectiveness of applying a standardised framework of controls. These approaches assume that predefined sets of controls or checklists can adequately cover all risks, without considering the unique characteristics of each model, dataset, or use case. Whilst this seems like an attractive, one-size-fits-all approach, it fails to account for the complexity, and rapid evolution of modern AI systems.
4. Siloed thinking
A final problematic feature of current approaches is siloed thinking. Many standard frameworks and approaches to AI Assurance are shaped by existing organisational structures, leading to assessments that mirror the expertise and focus of individual teams. This often results in ‘risk silos’, where certain types of risks are well-explored while others are overlooked. Such fragmentation prevents a holistic system assessment and obscures cross-cutting risks where, for example, a security breach might introduce bias, transforming a technical failure into an ethics or fairness concern.
What are the characteristics of an effective AI Assurance approach?
What, then, are the fundamental characteristics of an effective AI assurance approach? We propose that there are five key features that are essential to success:
1. Argument-based: the approach must focus on the communication of an assurance argument, with appropriate evidence and justification.
2. Proportionate: the approach must be flexible to allow for the tailoring of effort towards areas of least stakeholder risk appetite and greatest system risk.
3. Holistic: the approach should be focussed on whole-system, whole-lifecycle risk, including robust change management as models, systems, and contexts change and drift.
4. Vertical: the approach must deliver clear and actionable information from low level evidence to high level claims. The argument is clear and traceable, throughout the levels of system abstraction.
5. Robust: the approach provides clear evidence and justification that senior responsible owners can use to defend their decision-making in high-stakes applications
Approaches that do not include these key features are, in Synoptix’s view, unlikely to be able to provide comprehensive AI Assurance. Synoptix emphasises the importance of looking to high-assurance engineering industries, such as aerospace and defence, where these challenges have been prevalent for decades, to provide strong examples of effective assurance cases.
