A police officer from Derbyshire in the UK is under criminal investigation over their use of AI. Initial reports said there was:
“alleged use of AI systems by an officer to create evidential material in a number of cases“
Later reports provided a bit more detail, saying that the officer gave:
“biased prompts to an AI chatbot” to “generate paperwork slanted towards outcomes that the police wanted”.
The type of behaviour alleged in this case is not new and is not always alleged.
But this is not just another story of police misbehaviour or AI misuse, it’s a reminder that AI tools in public services need to be designed for accountability, not just productivity or efficiency.
Why accountability exists
Significant miscarriages of justice over the years have caused innocent people to be jailed, guilty people to go free, and victims to be let down. Two UK police forces have been found to be institutionally racist. This behaviour harmed the people they serve and reduced trust in the police and the broader justice system.
Legislation, like PACE and CPIA, along with professional standards for police officers, have been introduced and strengthened over the years. These types of legislation and standards are not perfect – nothing is – but they exist to make criminal justice processes fairer, reduce opportunities for abuse, and increase the chance of accountability when standards are not met.
There are similar pieces of governance across both the broader justice system, for example for barristers, and other types of services. Particularly public services where both the power imbalance between individuals and the state and the price of failure can be enormous.
Most recently, the UK’s Hillsborough Law (or, more precisely, the Public Office (Accountability) Bill) is expected to introduce a statutory duty of candour for public authorities and officials. This aims to ensure that organisations are open, transparent and accountable when things go wrong.
As the UK’s public services face ever-increasing pressure to redesign public services with AI, it’s important that the goals of this kind of governance continue to be met.
Designing AI tools for accountability
This is not only the responsibility of the professionals who use these AI tools, it is also the responsibility of the people who deliver them
So, as well as understanding their users and the purpose of their service, teams need to understand the surrounding governance and design their tools to support, rather than undermine, those safeguards.
For digital teams that means working closely with operational delivery teams to design both AI tools and working processes together. Designing tools that make it easier for users to meet their professional responsibilities. Ensuring there is meaningful and appropriate human sign-off on important decisions. Versioning models and designs, and recording which versions were used for which case. Putting in place ongoing monitoring to recognise opportunities for improvement and correct any issues that may be caused by the tool.
It will also mean working with teams and regulators that enforce those professional practices. Tools and processes should automatically record enough evidence of how outputs were produced and decisions were made to allow people to verify that professional standards were followed and identify when they were not. Those records need to be tamper-evident, so that they can be reliably used in courts as evidence to either hold people accountable or to demonstrate that tools and people worked and acted appropriately.
Regulating systemic risks
Regulators can help delivery teams by setting clear expectations and providing practical guidance for how to design new AI tools and monitor their usage, but they also need to address the new systemic risks that AI tools introduce.
We already recognise that individual professionals, like police officers, can be biased or act improperly. AI introduces new systemic risks, such as the biases that can be inherent in AI models, flaws in AI tool design, or biases and organisational incentives in the people and organisations who deliver AI tools.
AI tools can be deployed across entire organisations, so flaws in their design or implementation can affect many more cases than the actions of a single individual. One lesson from the UK’s Post Office scandal is that when poor technology is deployed at scale and misused by organisations then many people can be significantly harmed.
We need mechanisms to detect similar failures early and hold organisations accountable when they occur.
Making institutions more accountable
Because humans can behave in ways that harm others, societies developed governance mechanisms to hold professionals and institutions accountable for their actions.
The success of AI in public services should ultimately be judged not just by efficiency gains, but also by whether it makes those institutions more accountable to the people they serve.
