Tag: Trust

Designing AI for accountability in public services

2026-07-17 / peterkwells / 0 Comments

A police officer from Derbyshire in the UK is under criminal investigation over their use of AI. Initial reports said there was:

“alleged use of AI systems by an officer to create evidential material in a number of cases“

Later reports provided a bit more detail, saying that the officer gave:

“biased prompts to an AI chatbot” to “generate paperwork slanted towards outcomes that the police wanted”.

The type of behaviour alleged in this case is not new and is not always alleged.

But this is not just another story of police misbehaviour or AI misuse, it’s a reminder that AI tools in public services need to be designed for accountability, not just productivity or efficiency.

Why accountability exists

Significant miscarriages of justice over the years have caused innocent people to be jailed, guilty people to go free, and victims to be let down. Two UK police forces have been found to be institutionally racist. This behaviour harmed the people they serve and reduced trust in the police and the broader justice system.

Legislation, like PACE and CPIA, along with professional standards for police officers, have been introduced and strengthened over the years. These types of legislation and standards are not perfect – nothing is – but they exist to make criminal justice processes fairer, reduce opportunities for abuse, and increase the chance of accountability when standards are not met.

There are similar pieces of governance across both the broader justice system, for example for barristers, and other types of services. Particularly public services where both the power imbalance between individuals and the state and the price of failure can be enormous.

Most recently, the UK’s Hillsborough Law (or, more precisely, the Public Office (Accountability) Bill) is expected to introduce a statutory duty of candour for public authorities and officials. This aims to ensure that organisations are open, transparent and accountable when things go wrong.

As the UK’s public services face ever-increasing pressure to redesign public services with AI, it’s important that the goals of this kind of governance continue to be met.

Designing AI tools for accountability

This is not only the responsibility of the professionals who use these AI tools, it is also the responsibility of the people who deliver them

So, as well as understanding their users and the purpose of their service, teams need to understand the surrounding governance and design their tools to support, rather than undermine, those safeguards.

For digital teams that means working closely with operational delivery teams to design both AI tools and working processes together. Designing tools that make it easier for users to meet their professional responsibilities. Ensuring there is meaningful and appropriate human sign-off on important decisions. Versioning models and designs, and recording which versions were used for which case. Putting in place ongoing monitoring to recognise opportunities for improvement and correct any issues that may be caused by the tool.

It will also mean working with teams and regulators that enforce those professional practices. Tools and processes should automatically record enough evidence of how outputs were produced and decisions were made to allow people to verify that professional standards were followed and identify when they were not. Those records need to be tamper-evident, so that they can be reliably used in courts as evidence to either hold people accountable or to demonstrate that tools and people worked and acted appropriately.

Regulating systemic risks

Regulators can help delivery teams by setting clear expectations and providing practical guidance for how to design new AI tools and monitor their usage, but they also need to address the new systemic risks that AI tools introduce.

We already recognise that individual professionals, like police officers, can be biased or act improperly. AI introduces new systemic risks, such as the biases that can be inherent in AI models, flaws in AI tool design, or biases and organisational incentives in the people and organisations who deliver AI tools.

AI tools can be deployed across entire organisations, so flaws in their design or implementation can affect many more cases than the actions of a single individual. One lesson from the UK’s Post Office scandal is that when poor technology is deployed at scale and misused by organisations then many people can be significantly harmed.

We need mechanisms to detect similar failures early and hold organisations accountable when they occur.

Making institutions more accountable

Because humans can behave in ways that harm others, societies developed governance mechanisms to hold professionals and institutions accountable for their actions.

The success of AI in public services should ultimately be judged not just by efficiency gains, but also by whether it makes those institutions more accountable to the people they serve.

The data wasteland is polluted

2019-02-10 / peterkwells / 0 Comments

At the Open Data Institute we use a theory of change. It is one of the tools that we use internally to help us make decisions and externally to explain to people what we do and how we do it.

Our theory of change describes the farmland, oilfield and wasteland futures and helps us try to steer between the extremes of the oilfield and wasteland futures to get to the farmland.

The wasteland future emerges when there are unaddressed fears arising from legitimate concerns — such as who has access to data and how it might be used.

We frequently talk through the theory of change to explain what we do and how we do it. We try to provide pauses in the conversation to get other people to give their opinions. It helps people to think and learn for themselves. It helps us learn too. We hear what other people think happens in the wasteland future. How they think people and organisations will react to their fears being unaddressed.

Most of us the people we talk with think that the wasteland future has a lack of data. They realise that with a lack of trust then many people and organisations will reduce how much data they share. They imagine people refusing to use services because they don’t trust them, and that organisations similarly refuse to share data because they fear being punished. They think the data stops flowing.

These tips—based on insurers using social media data to set premiums—are stunningly dystopian. pic.twitter.com/uceFFIThyO

— Karen Levy (@karen_ec_levy) February 1, 2019

A smaller group of people realise the wasteland is more complex and weird. People’s behaviour will change in many different ways. Humans are fun like that.

Some people might post inaccurate data. Perhaps you will post fake claims of jogging exploits to social media if it is the only way to get a fair life insurance deal. Other people will hide in the data. Maybe we will give our children common names so they are hard to identify or so they appear to be from an ethnic group that is not discriminated against.

Similarly businesses will feel the need to create fake data. Organisations that fear that their supply chain data is being captured and used unfairly by their competitors might start to create ever more complex corporate structures to hide the data. Obviously reducing the chance of this unfair behaviour will also make it harder for regulators and civil society to know if a business is acting fairly.

I’m sure that even if you hadn’t thought of them at first you can now think of many more things that happen in the wasteland future.

You can see some of this future now. There are already people and organiastion hiding in the flows of data. Some of those people need and deserve help to hide because they have a genuine fear of harm, perhaps due to their political beliefs, ethnicity or sexuality. Equally there are others who are trying to evade fair scrutiny, for example tax dodgers and other criminals, and organisations providing services to help them do so. But if we increasingly fear harm then more people will want and need these services and, inevitably, they will become ever cheaper and used by more of us.

As this behaviour becomes widespread we will see data that is massively biased and misleading. People and organisations that use data-enabled services to tackle global challenges such as global warming, to price a life insurance premium in a way that doesn’t unfairly discriminate, or to decide whether or not to take a job will struggle. That would not be good for any of us.

Navigating the a route between the wasteland future and a different future where we get more economic and social value from data will not be easy. There will always be some people who need to pollute and hide in data to protect themselves from harm, we need to allow that to happen. Understanding and addressing people’s fears is not only a technical challenge, it is also a social and political one. To retain trust we need businesses and governments to adapt to people’s ever-changing expectations in a range of cultural contexts.

An increasing fear of how data is used will not simply stop people using services or sharing data, it will change peoples behaviour in a range of ways. If that happens we can expect data to be increasingly poor quality, biased and misleading. And that pollution will make data less useful to help people, communities and organisations make decisions that hold the potential to improve all of our lives. Some of that potential is false — the use of data required is too scary and people do not want or need it — but that is why it is important to understand and address the concerns we can if societies are to navigate towards the farmland.

You can read more about the ODI’s strategy and theory of change on our site.