Month: October 2023

The Post Office scandal and the law about computer evidence

The Post Office scandal is one of the most outrageous miscarriages of justice in recent UK history.

For over a decade people were wrongly accused of defrauding the Post Office. Many were prosecuted. Many others will have paid money to the Post Office to stop the accusations. Eighty three convictions have already been overturned, with more likely to come. There were jail sentences, deaths, divorces, bankruptcies and suicides before justice started to be restored. A public inquiry is under way.

One thing the scandal shows is that the legal presumption that computers are operating correctly risks causing a lot more harm.

The building on the left of this picture used to be a combined sweet shop and post office, it was run by my grandparents. Fortunately they retired before the Horizon system was implemented. Image © Google.

Some of the problems were caused by computers

The root cause of the scandal is often described as being bad technology. Horizon, a system developed and operated for the Post Office by Fujitsu, was clearly not up to the job. It incorrectly reported that subpostmasters owed the Post Office money. But stories from subpostmasters, journalists, court cases and the ongoing inquiry show a broader set of failures.

The failures spanned a full stack of user interfaces, technology, organisations and public policy. Image by Projects by IF.

Security researchers at University College London have done a neat summary of many of the technical issues with the Horizon system that were identified by a judge in 2019.

Nick Wallis’s book, The Great Post Office Scandal, also contains evidence of other issues. Sometimes Horizon’s touchscreen interface was too slow to keep up with the speed that people could enter orders, while the training and tools provided to subpostmasters were inadequate.

But there were organisational failures too

Employees of the Post Office and Fujitsu falsely told victims, politicians, and courts that the IT system was working correctly. Both organisations had evidence of its ongoing failures.

To make things worse the Post Office was in the rare position of having the legal power to both investigate and bring private prosecutions. There appears to have been little oversight of how the Post Office was using those powers by the Post Office’s sole shareholder, the UK government.

A report to the public inquiry that was published last week said that the policies and training provided to Post Office ‘s investigation and prosecution teams was not adequate to ensure that investigations and prosecutions were fair, auditable, and in accordance with the interests of justice.

And the law is an ass

And, for the icing on the cake, the Post Office’s prosecutions relied on a piece of law that says that courts should presume that computers are operating correctly. It is for defendants to demonstrate that a computer is operating incorrectly.

As the Post Office case shows it is incredibly difficult for defendants to do that when some organisations are as broken and incapable of being honest as the Post Office and Fujitsu. The evidence that they held of system failures was not shared with defendants or the courts.

The UK government has said that it has no plans to review this piece of law. As the technology industry is currently moving towards more probabilistic forms of technology, like the current wave of AI foundation models, that will be an increasingly strange position to take.

This is a law that will not just harm people in extreme cases, like the Post Office, but in many other cases too.

But if the law is an ass, then the law is an ass. There need to be campaigns to review and change the law, but in the meantime we can also try and make the law less of an ass.

A nice picture of an ass that I found on the web.

Making the law less of an ass

Over the years most people that design, build and operate computer systems, products and services have realised that we need to mitigate the effects of computer faults.

But there’s less recognition, and less guidance, on how to reduce the chance of law, like the presumption that computers are reliable, being misused to harm people. The intent of this law was not to falsely jail people, but that is the outcome it has led to.

So, if you are responsible for designing, building or operating a computer system then you need to think about how and when to provide evidence that that system does have faults and flaws.

That means doing the basic things like collecting and storing evidence about problems with the computer, its user interfaces, and the surrounding organisational processes. The kind of things that most organisations do.

But it also means being prepared to do what can be a hard thing. To disclose that evidence, even when an organisation’s leadership – like the Post Office’s – decide not to.

Because a fair justice system that does not falsely prosecute and jail people is one of the most important foundations of society. And, as our society increasingly runs on computers, how the justice system handles evidence about computers is going to become ever more important to all our lives.

Why the public sector needs trust to innovate

The public sector is facing pressure to innovate in response to changing needs, financial pressures, and emerging technologies. To innovate it needs to be trusted. The public sector can earn and maintain trust by aligning data and technology use with democratic values.

The pressure to innovate is coming from multiple directions.

Read more

Part 1….Towards a market of trustworthy AI foundation models

Foundation models, such as OpenAI’s GPT-4 and Google’s LaMDA, underpin many recent AI services. These models — which include generative artificial intelligence and large language models- make services like OpenAI’s chatGPT and Google’s Bard possible.

The UK Competition and Markets Authority (CMA) is carrying out an initial review of AI foundation models, focusing on potential competition and consumer protection considerations.

There are many possible intervention points where the introduction of new measures could reduce consumer harm. In this response from Projects by IF to the CMA we argue that the most effective measures will be to require model providers to deliver trustworthy foundation models, including supporting documentation and resources, that are designed to help product teams build better services.

IF’s Responsible Technology by Design framework

Read more

Want to reduce harmful design? Make good design easy.

People are rightfully concerned about being manipulated online. One of the causes is deceptive, or harmful, design practices. Two of the UK’s regulators have published a report on harmful design practices. This makes it even more important for organisations to act, but it is not enough. Regulators and service providers need to make it easier to do good design. Here’s where to get started.

Read more

The report says that cookie banners like this are harmful, and probably illegal, under both data protection and competition law.

© 2023

Theme by Anders NorenUp ↑