Written by Vincent Bryce
Diagram of a goalline technology system. Licensed under CC-BY-SA 3.0 (attribution: https://commons.wikimedia.org/wiki/User:Maxxl2)
The idea that there should be a “human in the loop” for important decisions has become an axiomatic assumption of many commentators on responsibility and digital technology, and a rallying cry for human-centred AI narratives.
But what are the motivations and values of humans with roles in technology-enabled decision making? Does their involvement enhance, or reduce a system’s inherent fairness? And how we do get not just humans, but “humanity in the loop”?
Recent media coverage has highlighted how seemingly-highly-automated systems may include human interventions, and how these can misalign the system with broader stakeholder perceptions of fairness.
Unexpected items in the bagging area
The concept of the “Video Assistant Referee”, or VAR, has increased in prominence in the world of football, with the promise of using technology and automation as a seemingly objective method to judge game-deciding decisions, such as whether match balls have crossed the goal line. However, behind the VAR sits a human referee, and when a decision seems to go against what may be perceived by stakeholders as fair and impartial standards – the role of this ‘human in the loop’ is called into question, and attention is drawn to the possibility of bias.
The Post Office Horizon scandal illustrates how the ability of management operators to intervene in ostensibly automatic processes poses fundamental questions for the claimed fairness of decisions, based on data from the underlying system. The saga also illustrates the potentially life-changing impacts of technology-enabled decisions on individuals, and the need for greater reflection on the limits of technology and management responsibility for consequent decisions.
The world of Human Resources provides examples of where purportedly scientific methodologies, brought in with the promise of reducing individual fiat, can introduce biases of a different kind which may be less obvious, but lead to decisions which are just as unfair. The cases of Amazon’s algorithmic talent management , and HireVue’s automated video interview based recruitment, show us not just the risk that machine learning systems can incorporate a range of biases inherent in the data used by developers to train them, but may go beyond this in creating a veneer of objectivity over management decisions which remain inherently subjective.
Visualising hidden bias
The ’hidden layers’ of ‘development’ involved in the development of machine learning based systems – in both the technical sense, and in the sense of the underlying design decisions which contribute to how they operate – are increasingly complex, can be hard to understand, and are often wrapped in glamourous sales pitches as to the transformative effects of generative AI and data science.
One way to understand the potential for hidden bias, and the inherent decisions of developers which shape these systems, is in reviewing the results of AI-generated imagery. Depending on the platform, searches for ‘football player’ will typically yield male characters, reflecting the inherent bias in the data generated by society on a given topic. As a result, the ability to promote a system in a way that returns appropriate results may be highly dependent on the operator of the system, and in turn on the perceptions of fairness in the audience(s) they are working with.
While mitigating this bias is an area of very active academic and industry exploration, recent challenges experienced by Google’s Gemini project indicate it is a difficult issue to address (and that addressing may involve confronting societal stakeholders about current, and historical inequality).
Images generated by Stable Diffusion v1-5 using ClipDrop API in response to the prompt “A portrait photo of a person playing soccer” and “A portrait photo of a person cleaning”. © Washington Post 2023
Frameworks for understanding and managing technology-assisted decision making
The mechanisms for surfacing and managing the biases involved in automated decisions are many and varied. The paradoxes and trade-offs involved have been the subject of academic debate since the 1980s, and remain so to this day.
To understand the extent to which seemingly-automated decisions may be subject to human bias requires us to consider a system’s day to day operation, and its origins.
In terms of day-to-day operation, it is helpful to see automation not as an either-or, but as a spectrum from full automation to full human control. Taking the example of an AI-generated poster, as a creative responsible for producing it, I can take a range of approaches:
- I can draw the poster myself (Full manual control)
- I can create a workflow which automatically generates and sends a poster from an AI image generation site when I receive a request (Fully automated)
- I can use the workflow, but all outputs require my approval before they are sent (“Human-in-the-loop”)
- The workflow will operate continuously if I don’t intervene, but I have visibility of the process and can step in to adjust if I decide to (“Human-on-the-loop”)
- I draw posters manually, but can selectively use AI for specific aspects or tasks (“AI-in-the-loop”).
Different approaches to automation. © Vincent Bryce 2024
These intermediate options define the space for ‘augmented intelligence’ and its associated narratives. They have been positioned as one way to manage broader societal risks in relation to, for example, technological unemployment.
However, as well as being tricky to separate in practice, all of these options retain human involvement. This results in systems influenced by the biases and values of individuals overseeing decisions. We might suggest that if judgements are applied to automated decisions which are at odds with the stated purpose of a system or values of an organisation, the net result may be ‘diminutive’ rather than augmentative – in other words, that the examples above might be described as examples of “Diminished intelligence” rather than “Augmented intelligence”.
The scientific literature offers a number of frameworks at a high level for aligning systems with the broader values and expectations of society, as well as increasingly a range of detailed tools for machine-learning based development.
Value-sensitive design is an engineering-focused methodology which aims to elicit, clarify, and purposefully apply values from the earliest stages of design decision making.
‘Responsible innovation’ offers a broad framework which encourages developers to anticipate and reflect on the impacts of their technologies, through stakeholder engagement processes and deliberative action to align with societal expectations. The alignment this involves may be either global (for example to ensure that a technology aligns with global standards or expressions of values such as the UN Sustainable Development Goals or European Charter of Human Rights), or local (for example to align a development to emerging local laws, or through dialogue with specific stakeholder groups). The resulting process enables a range of thematic aspects to be incorporated into development, which can include gender equality, “open science” principles, and governance considerations.
These methodologies have led to the development of further approaches, including the concept of “responsibility by design” which aims to incorporate a broad spectrum of responsibilities considerations into the earliest stages of development. There is an increasing realisation in the case of digital technologies that this requires awareness of supply chain and ecosystem considerations.
While typically applied to scientific and technology innovation processes, digital technologies increasingly enable design decisions by end users, and are configured into broader systems. For this reason, these approaches are highly relevant to those developing systems within organisations, as well as those developing innovative software or carrying out research.
Practical steps for designing systems for a responsible digital future
- Engage affected stakeholders at the early stages of system design, including end users to understand how they are likely to use the system in practice
- Carry out reflective processes at an early stage to ensure clarity on requirements, compatibility of the proposed system with stakeholder expectations, and clarity on the values to be applied to the system’s development
- Use ‘stage gating’ to allow for review and reflection at key points prior to implementation
- Design in governance and reporting processes to monitor system operation, in particular where fairness is dependent on human decision-making
- Ensure the boundaries for human and automated decision-making are clear, easily explainable, and consistent with your privacy policies
- Ensure any individuals who will have decision-making responsibilities in the system are supported by clear guidance, training, and governance/monitoring processes
- Consider a broad range of perspectives on the work you are carrying out – take steps to open up decisions for broader stakeholder scrutiny and input.