The Post Office scandal is one of the most outrageous miscarriages of justice in recent UK history.
For over a decade people were wrongly accused of defrauding the Post Office. Many were prosecuted. Many others will have paid money to the Post Office to stop the accusations. Eighty three convictions have already been overturned, with more likely to come. There were jail sentences, deaths, divorces, bankruptcies and suicides before justice started to be restored. A public inquiry is under way.
One thing the scandal shows is that the legal presumption that computers are operating correctly risks causing a lot more harm.
The building on the left of this picture used to be a combined sweet shop and post office, it was run by my grandparents. Fortunately they retired before the Horizon system was implemented. Image © Google.
Some of the problems were caused by computers
The root cause of the scandal is often described as being bad technology. Horizon, a system developed and operated for the Post Office by Fujitsu, was clearly not up to the job. It incorrectly reported that subpostmasters owed the Post Office money. But stories from subpostmasters, journalists, court cases and the ongoing inquiry show a broader set of failures.
The failures spanned a full stack of user interfaces, technology, organisations and public policy. Image by Projects by IF.
Security researchers at University College London have done a neat summary of many of the technical issues with the Horizon system that were identified by a judge in 2019.
Nick Wallis’s book, The Great Post Office Scandal, also contains evidence of other issues. Sometimes Horizon’s touchscreen interface was too slow to keep up with the speed that people could enter orders, while the training and tools provided to subpostmasters were inadequate.
But there were organisational failures too
Employees of the Post Office and Fujitsu falsely told victims, politicians, and courts that the IT system was working correctly. Both organisations had evidence of its ongoing failures.
To make things worse the Post Office was in the rare position of having the legal power to both investigate and bring private prosecutions. There appears to have been little oversight of how the Post Office was using those powers by the Post Office’s sole shareholder, the UK government.
A report to the public inquiry that was published last week said that the policies and training provided to Post Office ‘s investigation and prosecution teams was not adequate to ensure that investigations and prosecutions were fair, auditable, and in accordance with the interests of justice.
And the law is an ass
And, for the icing on the cake, the Post Office’s prosecutions relied on a piece of law that says that courts should presume that computers are operating correctly. It is for defendants to demonstrate that a computer is operating incorrectly.
As the Post Office case shows it is incredibly difficult for defendants to do that when some organisations are as broken and incapable of being honest as the Post Office and Fujitsu. The evidence that they held of system failures was not shared with defendants or the courts.
The UK government has said that it has no plans to review this piece of law. As the technology industry is currently moving towards more probabilistic forms of technology, like the current wave of AI foundation models, that will be an increasingly strange position to take.
This is a law that will not just harm people in extreme cases, like the Post Office, but in many other cases too.
But if the law is an ass, then the law is an ass. There need to be campaigns to review and change the law, but in the meantime we can also try and make the law less of an ass.
A nice picture of an ass that I found on the web.
Making the law less of an ass
Over the years most people that design, build and operate computer systems, products and services have realised that we need to mitigate the effects of computer faults.
But there’s less recognition, and less guidance, on how to reduce the chance of law, like the presumption that computers are reliable, being misused to harm people. The intent of this law was not to falsely jail people, but that is the outcome it has led to.
So, if you are responsible for designing, building or operating a computer system then you need to think about how and when to provide evidence that that system does have faults and flaws.
That means doing the basic things like collecting and storing evidence about problems with the computer, its user interfaces, and the surrounding organisational processes. The kind of things that most organisations do.
But it also means being prepared to do what can be a hard thing. To disclose that evidence, even when an organisation’s leadership – like the Post Office’s – decide not to.
Because a fair justice system that does not falsely prosecute and jail people is one of the most important foundations of society. And, as our society increasingly runs on computers, how the justice system handles evidence about computers is going to become ever more important to all our lives.
