In its 2024 manifesto the UK Labour Party promised to build a National Data Library that would:

“bring together existing research programmes and help deliver data-driven public services, whilst maintaining strong safeguards and ensuring all of the public benefit.”

Great, there’s a lot to do on data in the UK. But, unfortunately the manifesto commitment is a pretty broad scope and the term ‘library’ can be confusing. It is not surprising that people are finding it hard to agree what to do next.

Perhaps the National Data Library should look like the warehouse from Indiana Jones? 

Here’s a suggestion for what the “National Data Library” should be and how it could get started.

The National Data Library should help public sector teams deliver trustworthy data services by providing guidance, reusable platforms / technical components, and growing communities of practice.

It should start with practical work to build and improve the government’s existing data services. This will help it learn what is needed to deliver trustworthy data services across the public sector.

The visions and priorities for the library are broad, that is making it hard to define

At a political level the description in the manifesto might seem to make sense, but when you poke at it the sentence starts to fall apart.

The concepts of bringing together “existing research programmes” and helping “deliver data-driven public services” are different.

Public services tend to need access to reference data, information about rules, and to collect and process data from and about individuals and businesses that use the service. They are shaped by democratic debate and a range of legislation including data protection and administrative law. There are many existing public service-focussed data initiatives inside the government.

Research programmes tend to need access to large datasets from one or more organisations. They have oversight, such as research ethics committees, and researchers are likely to be accredited. A research programme is a lot more likely to seek informed consent, for both participation in the research and data use, than a public service. There are many existing research programmes running in universities, philanthropic and public sector organisations.

The original proposal for a UK National Data Library, from centre-right think tank Onward, said that it should be “a centralised, secure platform to collate high-quality data for scientists and [AI] start-ups”. While data for scientists might mean the same thing as data for research programmes, data for AI start-ups certainly does not. So, that is another set of needs to understand, prioritise and design for.

Finally, the Labour government places an emphasis on mission-led government with a set of initiatives that cut across government departments, other parts of the public sector and wider society.

These missions will rely on data to understand problems, make change, and report progress to the public (perhaps as official statistics) and senior politicians. To deliver on their missions teams are also likely to need to use data in ways that create impact that differ from people’s classic conception of a public service. So that is another set of things to think about.

An old project that looked at ways to use data to create impact

And the library will prioritise between all of these different areas and their possible use cases  while maintaining strong safeguards and ensuring all of the public benefit?

It does not surprise me that no one has yet managed to work out what the national data library should be. 

And the idea of a data library has been leading people in confusing directions

Meanwhile there’s another problem. The term ‘library’ seems to be confusing things.

Like most people who does policy – and other things – I can take a strange joy from exploring definitions and meanings, but the term ‘library’ seems to be proving unhelpful. People seem to be thinking of it as a single object or thing that contains all the data.

Box-and-wire diagrams are in fashion this year, aren’t they?

A central data portal with a catalogue for all of the data seems to be a popular idea.

Yet a lesson we have learnt is that a single, big portal will not meet people’s varying needs when publishing data, searching for it, or making use of it. A broad scope like the one set out in the Labour manifesto needs lots of catalogues, portals and other things to ensure that data gets to people who need it and are allowed to use it.

A less popular idea, but an idea that is still visible in policy circles, is a single technology platform – such as the one described by Onward. This would be a platform where all of the data is accessible with common governance, technology and standards.

Unfortunately a single platform for data would be a great target for hackers, stifle innovation, and change democratic accountability in ways that are hard to predict. 

Government is not one organisation. It is thousands of organisations with varying goals, that fulfill their democratic lines of accountability in different ways. This means that different governance, technology and standards will often be appropriate. The world of using health data for medical research is pretty different to the world of using data to improve local authority services for planning applications.

And, just like a central portal, a central platform would not be able to meet the wide range of needs of data users.

Given the focus on the word ‘library‘ I’m even expecting someone to be daft enough to suggest library cards, perhaps with fines for people who don’t ‘return’ a particular piece of data on schedule…

The UK needs an approach that works for many different contexts and that builds on the work that is already being done by teams across the country.

There is not much discussion of the need to improve the government’s existing data services

Lots of data services already exist across the UK that might fall into the National Data Library’s broad scope.

Research programmes like Genomics England, Biobank, OPENSafely, the Justice Data Lab, Research Data Scotland and the ONS Integrated Data Service. Public service initiatives like CDDO’s data marketplace, the MHCLG Planning Data platform, the Information Gateway for digital verification services, or Democracy Club. Data for innovation by startups like DfE’s Content Store, and Geovation.

Each of these services has stakeholders with different behaviours, needs, motivations and skills. They also have delivery teams with different capabilities, strengths, and weaknesses. In the different services data might use different standards, because they meet different needs. There are multiple legal and governance frameworks. Data protection is not the only law at play here.

Some of these existing services are great, some are heading in the right direction, some don’t seem to understand their stakeholders, and some simply don’t exist even when they should.

Rather than building a single thing, the National Data Library should make it easier for teams across the public sector to deliver data services like these.

The National Data Library needs to help people deliver trustworthy data services

Despite the variations between these services there will be some shared problems and lessons that have been learnt and shared for how they can be tackled. This is an area where the National Data Library could usefully focus. 

Here are some ideas for the support capabilities that could be provided:

  • guidebooks and manuals for how to design trustworthy data services and their associated governance and oversight mechanisms. Organisations can then use these core practices or adapt and build on them in their own context.
  • a shared library of user research to make it easier for data service teams to understand the range of people impacted by their work such as researchers, data analysts, policymakers, and members of the public.
  • a design system and patterns, the data design patterns that currently exist tend to have been developed outside the public sector. 
  • reusable, tested and well-maintained components, for example platforms – or simpler technical libraries – for publishing data or information about research projects, tools that make it easy to create data portals, verify data against standards, or understand data bias and its potential discriminatory effects.
  • a data linking service, guidance, public engagement, transparency and approval mechanisms for linking data across government and/or non-government research infrastructures. Perhaps London shows a way?
  • data transparency and control services, that empower individuals, communities and regulators to understand and control how data is used at times and places that are relevant to them.
  • a National Data Academy that provides training courses and coaching in data skills
  • funding, to support experiments, pilots and discoveries in under-resourced public sector organisations.
  • communities of practice to maintain and improve all of the above. Some of these communities already exist, both formally and informally, but many need more support. Communities of practice could usefully exist in teams building data services and in the groups of people – like researchers and start-ups – that use them.
Design patterns are a way that digital team teams communicate repeatable solutions to common problems. These images are from  IF’s design pattern catalogue.

To stress. These are just ideas.

Other people might have better ideas. A team that helped people deliver trustworthy data services would have to learn what was needed by:

  • doing the practical work to build/improve some data services themselves,
  • working with existing teams to understand their challenges,
  • and listening to various viewpoints from outside the public sector.

This was roughly how GDS got going in improving digital services.

A data library team would also need to be careful of the overlap with existing things that help with the goal of delivering trustworthy data services. For example the ICO’s regulatory guidance on data protection, HRA’s advice on health research, or even the OSR regulatory guidance and ONS guidebook’s on statistics. 

But these are not insurmountable challenges and, from my own work across government, I’m confident that there are many needs for support that are not being met and many opportunities to tackle the problems together.

The existing data service teams across government can then use these new support capabilities to help them build trustworthy data services better, cheaper and faster.

Getting things started

It is important that one of things a national data library team that helps people deliver trustworthy services starts with is some practical work to help public sector organisations improve or build some data services. This would deliver some early impact, create momentum, and generate learnings and capabilities in that core team.

In selecting these data services it will need to look for some variety. This will help illuminate the problem / opportunity space.

The final decisions for where to start should be based on government priorities but – from my own knowledge about common problems / opportunities – perhaps it could include:

  • a data service that openly provides access to authoritative, non-personal data held by the public sector that could be widely used by public services and startups, for example address data
  • a data service that provides authorised organisations and researchers with secure access to attributes about individuals, for example their age or eligibility for benefits
  • a platform that helps research programmes publish accessible information about research projects throughout their lifecycle so that individuals and communities can understand how they are impacted by research activities
  • a platform that helps research programmes publish accessible information about the data they hold and types of research they support so that researchers can find what they need more easily
  • a service that makes it easier for multiple local authorities to publish data locally and then aggregates it for use nationally, for example information about elections or places where clean energy infrastructure could be built

By doing this work the national data library team can start to develop the much-needed guides, components and communities of practice that can deliver more trustworthy data services across the public sector.

The National Data Library should help people deliver trustworthy data services

The commitment behind the National Data Library provides an opportunity to improve how the UK public sector, researchers and startups use public sector data, but this opportunity will not be realised if the UK ends up in endless abstract policy debates, workshops and roundtables or – even worse – building big new central portals and technical platforms that everyone is told to use.

Building capabilities that support existing teams to deliver more trustworthy data services across the public sector will be far more impactful and start to deliver the change that is needed.

Want to read more stuff?

If you’re interested in the National Data Library then here are some links I found useful when forming my views:

And thanks to Ellie, Steve, Andy and others who I bounced around these ideas with as I was writing them up. All mistakes and idiocies are always my own.