Tag: Open Data (Page 2 of 3)

Cat data is complex, and that’s ok

2017-03-12 / / 0 Comments

Last year I openly published data about some of the cats that work for the UK government. I ended up giving a talk about it. When publishing the data and giving the talk I skipped over the potential data protection and privacy issues.

Why are you talking about my data?

Some of those potential issues came up again recently when our family cat, Bugsy, was being transferred to our new home. I was nervous about the cat arriving safe and on time. A friend asked:

can’t you publish some data showing the cat on his journey?

Such a short and simple question. This is my long and complex answer. Most of my friends are patient people.

This post might sound like it is going to be whimsical —ok, there will be some cat whimsy…— but there is a serious point. Publishing and thinking about cat data helped me think and talk about other data things with more people.

Thinking and talking about data protection, ownership and control for cat data will have the same effect. It is pretty important that more people know how complex they are.

This cat data deserves data protection

Different countries have their own data protection and privacy laws. Personal data can be hard to define but at the Open Data Institute we encourage people to look at relevant legislation and start by simply saying:

Data from which a person can be identified is personal data.

If data can be combined with other information to identify a person, that data will still be personal data.

If there is personal data in a dataset then we should consider relevant data protection legislation and the univeral human right of privacy.

At this point I expect that lots of people reading this post will be thinking that a cat is not a person so neither the personal data definition or human rights do not apply.

This is true but, like other animals, cats do have rights. Some people argue that pets are becoming people, in a legal sense, and that animals deserve democratic representation. Perhaps cats do not have data protection rights today but if that might change in the future then perhaps I need to worry about it today.

A cat called Paddington chasing its own tail. Picture by Bill Abbot, CC-BY-SA.

Whilst this would be a fascinating topic to explore unfortunately, to paraphrase a recent article by Luciano Floridi on the rights of robots and artificial intelligence, I’m in danger of chasing my own tail when I should be focussing on the current opportunities and challenges with data that affect people. People like me. Our cat wasn’t moving home in a few year’s time, he was moving now; and I was nervous.

There is a simple reason why I need to think about data protection if I was to publish this cat data. Whether cats realise it or not, their data can refer to people. My cat lives in the same house as me. If you knew the destination of its journey then you would know where I live. If you knew the date when it was being transferred to a new home then you might be able to guess that my old or new home is empty. Etcetera.

So if I was to publish data about Bugsy’s journey I would need to think about the impact on privacy using a methodology like the one provided by the UK’s Information Commissioner’s Office (ICO) before I published the data.

Ownership of cat data is complex

I occasionally hear people saying that defining a legal right to personal data ownership will make this process easy. My privacy, my data, my choice. I doubt my cat cares about human laws but, according to the law, I own him. So I might legally own data about my cat and would have the legal right to choose to publish it. Unfortunately data ownership is not that simple and nor is cat data.

How is my cat’s identity defined? Some cats have microchips, and Edinburgh University have even given a library card to a cat so it can prove its identity and demonstrate its entitlement to borrow books, but our cat just has a phone number on its collar. Is that sufficient?

Defining legal ownership of cats in data seems simple.

Meanwhile Bugsy is a family cat. He is owned by me and my wife. It might look like that joint ownership can easily be defined in data, but the world is more complex than my simple model. How is my identity and that of my wife defined? How would we verify our identities to say that we are allowed to track our cat on his journey? Identity management is hard.

And once we get past those issues I might find that my wife disagrees on how the cat’s data can be used. We both own and live at the same house that the cat is being transferred to. The data refers to both of us. My wife might think my nervousness is utterly ridiculous and not worth risking our privacy for. There have been several legal disputes over the ownership of pets. I don’t think it would calm my cat moving nerves if I was to take my wife to court over ownership of cat data.

Meanwhile we’re still missing something quite important. The cat isn’t travelling alone on his journey. He is being transported by an employee of a company. What about that company’s potential rights to own the data produced by their service? What about the cat transporter’s privacy?

Controlling cat data

At this point, when answering that simple question from a friend about publishing data about Bugsy’s journey to make me feel less nervous, I started to talk more about consent.

Data protection isn’t just for the online world. We also need to think about the offline world and the billions of people who don’t use computers.

Giving people choice and ongoing control over how you use their data is becoming more important. It’s one of Tim Berners-Lees three challenges for the web. Some trading blocks, like the EU, and individual nations, like the UK, have decided that it is necessary to put in place new legislation that strengthen people’s rights over data. Consent is not always necessary but the ICO recently published some draft guidance on consent under that new legislation which I could use to help publish cat data.

My wife knows quite a bit about data so could give informed consent which I could record. I could also ask the cat transporter and their employer if they were willing to consent. To be clear I would want to give the cat transporter the choice of saying no. A world where people who transport cats have less privacy than other people does not sound a sensible world.

Unfortunately given the impending journey I did not have time to think about or research the cat transporter’s needs and skills. The ICO’s guidance says that I can assume that “adults have the capacity to consent unless you have reason to believe the contrary”, and I knew how to be open about how I planned to use the data, but without more research I would not know how to design something so that the cat transporter could choose whether to consent, or not. I might mistakenly assume that an online only service was good enough, despite a large proportion of the UK population having no access to the internet or insufficient skills to use it. The cat transporter could be one of those people.

And all I would have achieved by this point was possibly gaining consent. I would not have given the cat transporter control over the data about their journey. With that control they could reuse the data for another purpose, such as reclaiming their petrol costs or seeing what cat data tells us about people moving house around the country. My wife, the cat transporter, their employer and I all had rights to the cat data and should all be able to have some control over its use.

Sometimes you need to keep things simple

At this point my wife and friend both firmly interrupted me and told me I was not being utterly ridiculous but being completely and utterly ridiculous. I was trying to design a perfect solution that would work for many cats and purposes, rather than keeping things simple and starting with a solution for a particular problem. My nervousness about our cat.

My wife rang the cat transportation company and asked them to text us a couple of times during the journey. They agreed, of course. Sensible wife.

Data is complex, and that’s ok

Now you might read all of this and ask:

if we have to think through all of this complexity everytime we’re thinking of publishing data how will we ever build anything?

The team at the Open Data Institute, where I work, do the hard work to try and make data as simple and easy as possible so more organisations can get data to people who need it.

That requires us to work on lots of things including how to publish data; how people will search for it; the skills they need; how to use it in organisations, large and small, or whole sectors; and how to get data to benefit everyone. Lots of other people do similar things.

But sometimes I wonder if we and other people can make it sound too easy.

So when we’re encouraging more people to do wonderful things with data then as well as the brilliant possibilities we also talk about the challenges using both real examples and whimsical ones like the ones I faced with my cat data. Whimsical tales sometimes help convey simple messages.

We can build a better future with data but we need to solve problems and be realistic about the complexity if we are to build one that works for people. Data is complex, and that’s ok.

Make data great again

2016-12-29 / / 0 Comments

Data is becoming increasingly important to our societies. We live in an age of data abundance and, without many of us realising, data has become a new type of infrastructure and a critical one at that. The age of data abundance has led to brilliant new services and can help our societies tackle challenges such as climate change and population growth, but it also creates risks to privacy and concentrations of power.

Societies need to be able to debate what this age of data abundance means for them. People need to make decisions about the relationship between individuals, communities, societies and data. We need to pick a future vision for our relationship with data and then make steps towards it. Many governments and societies are having this debate now.

In my job I put forward the Open Data Institute’s position on those decisions while also trying to encourage a more public debate. I want a debate because I, and the lovely people I work with, want the decision to be made by societies around the world.

To make this debate as broad and informed as possible, I need what I say to be understandable by as many people as possible. I try to use plain language and frequently test new language and concepts to see if they are understandable. Sometimes I test things through tweets or blogs, like this one, at other times by talking with people from differing backgrounds and perspectives.

By testing, listening and learning I have made some of the language more accessible but I’ve also realised that something was more important than I first thought: politics. Both my politics and that of others.

Let me try and explain.

Choices about data

Sometimes people say they want to help people make better choices about data. I did that a few times in this blog about an open future for data.

I was talking about the ideas in that blog with a left-wing British politican who stopped me mid-sentence and asked if I was a Blairite nowadays. No, I replied. “Then why are you using the language of Blair’s choice agenda?”, they asked.

image copyright the BBC. Taken from a blog stating that the comedy show Yes (Prime) Minister, was the most cunning political propaganda ever conceived

Further testing of the language caused another person to recoil and suggest that if I kept talking about choices I might be accused of being a secret Thatcherite pushing the theory of public choice. Hmm….

I’d used the word ‘choice’ because I thought it was plain language but it was clear that the decision risked putting in place a political barrier for some of the other ideas in the blog. This is a problem.

Data is political

When thinking about and debating technology and data with other technologists it can be easy to fall into a trap of thinking that every decision can be based on empirical evidence, that there is a single right answer and that we can make that right answer a reality by designing and building the right technology. This is nonsense.

In our debates about data we need to decide issues of access, ownership, regulation and the relationship between citizens and the state. These are political decisions.

Whilst we might have individual opinions about data we need a state and legal system to help put decisions into practice. States will allow technologists to innovate and try things out but there comes a time when existing legislation will be more strongly applied or new legislation will be put in place as society’s needs change. This happened and continues to happen with road traffic, it will happen with data.

By broadening the debate we are helping that decision to be made democratically. Democracy might have seemed under strain in some countries in 2016 but as Churchill said:

Indeed it has been said that democracy is the worst form of Government except for all those other forms that have been tried from time to time

To put it more simply politics and democracy is important and data, as with most things, is political.

Words already carry political meaning

The “white heat of technology” makes me think of Harold Wilson and the 1960s UK Labour party. Because of my political history I have positive feelings about the phrase despite the speech being followed by the scrapping of several high-profile technology projects. Image copyright PA.

Words are a tool political people use to reach our hearts. Sometimes those words are a catchy slogan. At other times it’s a frame: a guiding metaphor or image for a political argument.

Political slogans and language are designed to appeal to a group of people, build on existing beliefs and make them choose a particular path.

Some words carry a particular meaning in the present because they have been used in a political context in the past. Marx said it more poetically:

The tradition of all dead generations weighs like a nightmare on the brains of the living.

The word “choice” resonated amongst some people involved in British politics that I spoke to because of those traditions and their political history. It will have bought back nightmares for some and heavenly dreams for others.

Data is not about left or right wing politics

In economic terms each of these cakes is rivalrous: only one person can eat them. Cake is not like data, multiple people can use data at the same time. Picture of cake by Hani AlYousif, CC BY-NC-ND 2.0

Our societies and political systems are used to making political decisions about many types of resources, for example oil or water, but data has different qualities to the physical resources that are embedded in our political systems, debates and legislation.

To give two regularly used examples: data is non-rivalrous, unlike a piece of cake many people can use data at the same time, and data benefits from network effects, it becomes more valuable as more people use and maintain it.

These differences are one of the reasons the team at the Open Data Institute talk about data as analogous to roads:

Data is infrastructure. Just like roads. Roads help us navigate to a location. Data helps us make a decision.

The “data is roads” analogy breaks people out of the traditional mindset. It helps open their minds to thinking differently.

I think that, as with the web, these different qualities mean that a closed-open axis is a more useful way of thinking than the traditional left and right-wing political axis.

But it will be harder to get people to think about the decisions along that closed-open axis if our words and ideas cause them to think of old left and right wing political battles.

Take back control of data

Data has many other different qualities to other resources. One that is becoming increasingly evident and important is that data is sometimes about identifiable people, sometimes it isn’t and sometimes it’s a bit complicated.

Much of the current debate about data is dominated by personal data: the stuff which is about identifiable people. Many people believe that there is an asymmetry of power and privacy as data about us is controlled by governments and corporations.

Tav Kotka, the Estonian Chief Information Officer, at MyData 2016 in Helsinki. Watch the full video.

Tav Kotka, the Chief Information Officer of Estonia, recently gave a talk in which he broached the idea of adding a fifth freedom to the EU’s existing four freedoms for free movement of goods, workers, services and capital. The talk was mostly about personal data and the concept of personal data stores that could allow individuals to control how data about them is used.

Whilst I agree that more personal control over personal data is important the talk bought up memories of Margaret Thatcher and my teenage political nightmares. The talk did not mention society’s need to access and use that data. Taking back control of data by giving control to individuals misses out the challenges of digital inclusion and the role of other important parts of society like families, communities and nations. Different levels of control, rights and responsibilities are likely to need to given to these different groups. To give just one example vital medical research and national statistics need to use large amounts of personal data, this can’t be neglected or left solely to the decisions of individuals.

But, as I realised, this time I was the one allowing my political history to do the interpretation for me and I was the one who wasn’t listening to the underlying argument. Tav Kotka was using language that built on his political history while talking in English to a Finnish audience. Even though I work for a global organisation my initial reaction was from a UK perspective. My bad.

The political debate about data is happening now

The EU is currently discussing complex concepts such as data control and data ownership through the free flow of data initiative. Major geopolitical organisations, like the EU, can have a large impact on countries outside their membership, the UK government has committed to following current EU data protection regulation after it exits the EU. That EU debate involves politicians from multiple countries, each with their own rich histories and perspectives. There are many other debates in countries around the world.

If you want to help build a great future for data then as well as building new services you may want to get involved in either this or other multinational, national and local debates.

But if you do, remember to think about politics: both other people’s politics and your own. That way you will be best placed to help people think about the decisions not in terms of traditional left and right-wing politics but instead in terms more suited to the different challenges and possibilities of data.

Words from leasehold and commonhold reform APPG

2016-12-15 / / 0 Comments

Approximate words spoken at the meeting of the the UK Parliament’s All Party Parliamentary Group (APPG) on residential leasehold and commonhold. The meeting was chaired by Jim Fitzpatrick MP and Sir Peter Bottomley MP. There were 60–70 people in the room: MPs, Peers, conveyancing firms, big homebuilding companies and people suffering under bad leasehold terms.

Yes it’s 900 years away but why should anyone produce or sign a contract that commits them to spend this? (source: Telegraph)

I spoke after Patrick Collinson from the Guardian, who has written extensively about leaseholds in England and Wales and the issues some leaseholds cause for people; Bob Bessell of Retirement Security; and Phillip Rainey QC a specialist in property litigation and expert in leaseholds.

Phillip discussed various policy options to tackle the challenges. The options includes banning ground rents or limiting how much they could increase in value and many other subtle tweaks.

I then had 5 minutes.

Hello, thank you for inviting me. I’m from the Open Data Institute (ODI). You may not have heard of us. (murmers of agreement)

We were founded 4 years ago by Sir Tim Berners-Lee, the inventor the web, and Sir Nigel Shadbolt. Our CEO is Jeni Tennison, she apologises for not being here. So do I as I’ve ended up creating an all-male panel. That’s bad.

We are global. We connect, enable and inspire people to innovate with data. Or “to get stuff done that make things better by being more open” as I sometimes say.

I am not a housing or leasehold specialist, my job is to get data to people who need it. Leasehold Knowledge Partnership are part of our current UK startup programme. They’ve been helping us understand the problems in leasing, we’ve been helping them understand whether more data can help.

At the ODI we think of data as a new form of infrastructure. It has become essential infrastructure without us realising it.

Like most physical infrastructure – for example roads – data creates most value when it is as open as possible while respecting privacy.

When data is open and available for anyone to use it is easier for people to use it to make decisions and solve problems.

Take leaseholds. Let’s imagine if more information was open while respecting the privacy of homeowners.

  • People expect easy access to data in the web age. Many homebuyers use sites like RightMove and Zoopla as they look for a home. Opening up leasehold data would enable those services to help people make an informed decision. For example they could compare terms with other properties, leasehold or not, in the area and see what’s reasonable. Some of the cases Patrick mentioned happened because people lacked information when buying a home.

  • Conveyancers and estate agents would have access to more data too. They could get things done faster and give better advice to homebuyers.
  • Researchers would be able to model the market; help people understand how it is working and suggest improvements
  • Legislators would be able to get better information about problems, where legislation is needed or where soft power could be used to influence things
  • With better access to data government could test a policy idea, like the ones Phillip suggested, in a region before deciding whether to roll it out nationally

Much of this data is available but it is locked away. In government offices, in the offices of house building firms, in law firms or in contracts held by leaseholders and freeholders.

Some of our big public registries and institutions – things like the Land Registry, Ordnance Survey, the Met Office — were created to make this type of information available to people who need it but it feels like they haven’t adapted to changing times and 21st century needs.

Getting this data open can take time and cost money. Not that much, technology can be cheaper than some people might tell you. But getting the data open and using it to change markets, like leasehold, can also affect business models. That’s usually more significant.

We need to support those organisations to change their business models; move to a future where we have data infrastructure that is as open as possible while respecting privacy; and help meet society’s 21st century needs. That might mean they also need to help open up data held outside government.

In closing I’d ask both the members of the APPG and all of the leasehold experts in the room to think about the power of the web, what people expect in the modern age and how the tools and techniques of the web and data can help build a better housing market. One that can reduce the number of cases like those that Patrick Collinson has written about over the last few months.

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — -

After the various speeches questions were asked by people in the room. The questions were from a more diverse group of people than the the all-male panel (grr!).

I was asked whether there was enough data available for someone in Ellesmere Port to get a reasonable view on whether their leasehold flat will be worthless in 10 years time. I’m checking that today.

Someone else raised the issue of freehold management companies surprising people with unnecessary administration fees — for example £250 for a simple bit of paperwork that is necessary if the homeowner wants to sell their home. That’s an issue my wife and I are well aware of having just sold our leasehold flat in London. We plan to blog on how data helped and where some data was missing.

Someone else asked whether we knew if the problem with leaseholds was bigger than in the 1970s. The answer from the panel was a bit vague but Phillip Rainey raised an important point. He said that the problem was getting worse because lawyers were producing new tighter leasehold clauses that benefitted the freeholder. He said that lawyers used the web to share these new clauses so they were all getting better in a way that made the situation worse for leaseholders.

You see technology can be used for good and bad and — as a very wise person once said — knowledge is power.

To help level out power imbalances we need to share the knowledge and the skills to use it with everyone.

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — -

After these questions the event was closed by Peter Bottomley who discussed next week’s leasehold reform debate in Parliament and how he intends to name names.

{Update 22 December: the Hansard transcript of the debate is now up}

Seven maps that show the anatomy of America’s vast infrastructure and one blank map

2016-12-03 / / 0 Comments


The Washington Post had an article the other day on six maps that show the anatomy of America’s vast infrastructure: the electric grid; bridges; pipelines; railroads; airports; and ports and inland waterways. The article has beautiful pictures of these big, important things that make it possible for society to work for as many people as it does.

All of the maps were created using data from OpenStreetMap. OpenStreetMap is brilliant. A map of the world that is collaboratively maintained and free for people to use. OpenStreetMap is also part of a new type of infrastructure, one made of data. That data infrastructure also underpins our society in the same way that other more visible bits of infrastructure do.

Data helps engineers understand where physical infrastructure is needed, what capacity is required and how to build it safely. Data, like maps or journey planners, helps people discover and use infrastructure. It does many more things too, even if some may seem a little weird.

Without data infrastructure, and without it being so easy to use, then the Washington Post might not have printed those beautiful pictures; engineers wouldn’t find it as easy to plan and build physical infrastructure; and people wouldn’t find it as easy to use that infrastructure.

A seventh map

A map of open address data for the USA courtesy of openaddresses.io

As well as the six maps that the Washington Post chose they could have used this one from openaddresses.io. Every dot is an address.

It’s a bit patchier than the other maps that the Washington Post showed as some USA address data is not openly available. Either the data doesn’t exist or it us kept behind pay walls which makes it hard to use. This is a problem. Everything happens somewhere and addresses help us locate all of those somewheres wherever they are in the world. This data is vital infrastructure and must be freely available for anyone to use.

Luckily data infrastructure is a lot cheaper and quicker to build than roads and waterways. The US government recognises the benefits of making this data available and is working to do it.

A blank map

A map of open address data for the UK courtesy of openaddresses.io

In the title of this post I promised a blank map. It is not quite blank but there are no dots.

Address data for the UK is not openly available, it is locked behind paywalls. It is as if there were toll roads all over our road infrastructure. Just as fewer people would use roads if they had to pay a toll every few miles, fewer people use address data because of the paywalls. In both cases there is less social and economic impact.

Meanwhile the UK’s address data is not collaboratively maintained, like OpenStreetMap, and the quality suffers as a result. People who move into new build houses often discover that their address is missing from the lists stored in computers. They can’t order a pizza, a sofa or even register to vote. People know the address exists, it is the computers that don’t.

A couple of years ago I worked with a team of people trying to fix this. We failed. A team in the UK government are now trying to open up UK address data, I hope they succeed.

Data gets overlooked, even when a journalist is using it

Data infrastructure is part of the government’s responsibility in the same way as the other forms of infrastructure that the Washington Post wrote about. They are all vital infrastructure that underpins our society. They should be both protected and made widely available in exactly the same way.

Much of our data infrastructure is patchy or difficult to use. Things like maps, records of land ownership, ompany information, where and how we can vote.

Data infrastructure should also form part of the public debate alongside other forms of infrastructure. The danger is that data is misunderstood and overlooked, even when a journalist is using it to draw some beautiful pictures.

Automated cars and data

2016-11-17 / / 0 Comments

Everyone’s talking about automated cars and how they will make it cheaper and easier for us to get from place to place. As well as helping us travel they will change our cities by freeing up space, save lives by reducing the number of driving accidents and lead to the loss of millions of driving jobs with the associated impact on people and communities.

If you’re reading this I bet you’ve heard this talk. If you live in one of the test areas in America, UK, China and etcetera you may even have seen trials. There are skeptics, but I think that people will be able to gradually build ever safer and more automated cars. Once they do many people will choose to use them. Change is coming.

More Autonomous Cars Coming to Public Roads in 2016 Copyright © 2016 ENGINEERING.com

Making it easier and cheaper to move around, changing cities, saving lives, removing a type of job are complex things. There are many more secondary effects. Our policymakers need to consider the risks and benefits to help us get to a better society that includes automated cars and benefits everyone.

But I’m not seeing enough discussion of one important aspect of automated cars: data, and how security, privacy and openness can increase its impact.

Automated cars collect a lot of data

As well as transporting people and parcels automated cars will collect vast amounts of data. A human driver needs to look around to see street signs, the weather or cyclists. Similarly automated cars will need to collect data to make driving decisions.

http://dataconomy.com/how-data-science-is-driving-the-driverless-car/

Automated cars collect a lot of data. A PhD student recently calculated that a modern car already generates 25Gb of data an hour. In 2013 it was reported that Google’s automated car generates 750Mb of data a second. Earlier this year Comma.ai, a company that was working on automated cars released 80Gb of data generated during 7 1/4 hours of driving.

This data includes such things as the car location, maps and video footage of the surrounding area, information about nearby traffic, accidents, weather information, the route of the car and information about any passengers or parcels that that are in the car.

That’s a lot of data, how do we get most value from it?

Security and privacy

The security of this data clearly needs to be considered. We need to protect the data collected by the car and the data that the car needs to be able to get to do its job. Car hacking is a real risk whilst an automated car is likely to be more dependent on access to data than a car driven by a human. Data is already an under-recognised piece of critical national infrastructure, automated cars will only increase the need to strengthen it.

Silly Wired. Nexar, like any camera, isn’t just collecting your data it’s also collecting data about other people.

Privacy will also be an important consideration. If automated cars mishandle personal data about the people travelling in them or the people and things seen by their video cameras then some people will be damaged while other people may lose trust and choose not to use the cars.

Some of these issues will be explored by smartphone apps, like Nexar, that use the smartphone’s camera and microphone to collect data about car drivers, passengers, pedestrians and other cars.

But automated cars will collect far more data than a smartphone camera.

Automated cars will use data collected by other cars and people

Automated car manufacturers and policymakers should be thinking about security by design, privacy by design and how openness can help build the trust that will be needed to get the most impact from automated cars. Open can help in other ways too.

The data collected by cars is needed for them to do their job but automated cars will also use data provided by other things and people.

Automated cars won’t be like a starting character in the Civilization games. They’ll be able to see the full map. Civilization made by Firaxis Games, image from VentureBeat

An automated car will not wake up in a factory, blearily blink its headlights and then discover the world like a video game player constantly surprised by new things. The car will have a reasonably accurate map of the world, will get weather data (what sensible car would choose to drive into a hailstorm that might damage its paintwork?) and be able to share data with other cars.

Just as we hear of traffic jams from other people via radio alerts or smartphone apps like Waze, the people designing and building automated cars have planned for them to be able to share news about traffic congestion or improvements to their basic maps. Those improvements are vital because map data, just like any other data, is not always 100% accurate. Things change. An automated Google car driving down a street might discover that a road is blocked off, by sharing this with other Google cars it can make Google’s service more efficient.

This all sounds like good use of data, but it’s not good enough. We can and should do better.

Data should be as open as possible while respecting privacy

Werner Herzog’s first automated car looked a lot like a boat.

Sebastian Thrun of Stanford says in Werner Herzog’s new documentary “Whenever a self-driving car makes a mistake, automatically all the other cars know about it, including future unborn cars.”. But isn’t the only way that all self driving cars will ‘automatically’ know of all other mistakes is if the data that describes those mistakes is available beyond just the automated cars of one manufacturer?

At the Open Data Institute we think that we get the most value from data when it as open as possible while respecting privacy.

The team at OpenStreetMap’s 2016 April Fool’s spoof was a plan to launch their own automated car. They said: “our self-driving car breaks new ground by automatically correcting OpenStreetMap data based on your driving behaviour”. The story was a spoof but this bit — regardless of whether it’s OpenStreetMap or another mapping organisation/community relevant to a particular country or city — is one of the ways that cars can share data with each other and with other people.

Mapping is a shared problem. All cars, automated or not, will benefit from better maps. As will pedestrians, cyclists, local authorities planning new infrastructure investments, etcetera. Collaboratively maintaining open mapping data between all of these people can reduce costs and improve quality. Facebook are happy to collaboratively maintain open mapping data as they recognise the value in this approach. Automated car manufacturers, mapping organisations and policymakers should be too.

Reducing accidents is another shared problem. The machine learning algorithms that will drive automated cars will learn faster and more accurately from more data. Sharing detailed data about the conditions in place when an accident occurred will save lives.

People will ask an automated car to drop them off at an address. That address may not be in the current list of addresses — perhaps it’s a new flat? — so the person may teach the automated car where it is. The address could then be sent to an open address register as a potential improvement to the data. The next automated car will know about it but addresses are vital for many other things from pizza delivery to an ambulance. We should be maintaining addresses as efficiently and openly as possible. Collaborative maintenance helps with that and openness means that anyone can use it.

There will be many other types of data collected by the car that when opened up in this way will improve transport services, save lives and make things better in other sectors.

Live weather conditions (something that the lovely folk at TransportAPI are working on). Air quality. Congestion data. Aggregated movement of people around a city. Etcetera.

This impact of opening up this data will be felt not just in better automated car services but in other services and sectors that use the same datasets. Automated car manufacturers are in the transport business, not the mapping or air quality business. Publishing the data openly will help them tackle shared problems and increase the impact of the data. Everyone benefits from better and more open data.

Automated car data should be secure, private and open by design

The Open Data Institute’s data spectrum. The most important things about data is who can access and use it. Mapping an automated car’s data against this data spectrum would be very interesting.

The transport sector has long been a leader in open data. The countries and organisations that have taken the lead in opening up this data have benefitted both from better services for people and through the creation of innovative new services like GoogleMaps and companies like CityMapper, Transport API and ITOWorld that create jobs and help get the data used.

As that seemingly inevitable next wave of change occurs with the rollout of automated cars that will improve transport, free up space, save lives by reducing accidents and impact jobs let’s make sure we don’t forget about the data infrastructure that is necessary for those cars do their job and can create so much value for the rest of our society.

Making that data infrastructure secure, private and open by design will benefit everyone.

If you want to chat about the thoughts in this blog then tweet or mail me.




Hacker Noon is how hackers start their afternoons. We’re a part of the @AMI family. We are now accepting submissions and happy to discuss advertising & sponsorship opportunities.

If you enjoyed this story, we recommend reading our latest tech stories and trending tech stories. Until next time, don’t take the realities of the world for granted!


Words for the launch of the APPG on data analytics

2016-11-02 / / 0 Comments

These are the approximate words I said at the launch of the new All-Party Parliamentary Group (APPG) on data analytics on 31 October. An APPG brings together representatives from different political parties from both the House of Commons and House of Lords to pursue a particular topic or interest. Daniel Zeichner MP’s speech from the launch is also online. Other speakers were from TfL, Experian, CompareTheMarket and the Institute for Environmental Analytics. In person I wandered off topic a bit based on audience reactions but I promise that there were no cat jokes.

Hi, thank you to everyone who’s come along and for inviting us to speak. I work at the Open Data Institute, or ODI as it’s more commonly known. The ODI’s mission is to connect, equip and inspire people around the world to innovate with data.

It is based in London but the network is global. We have nodes and members on six continents and in every nation of the UK. We do research, train people, advise them, introduce them to people with similar interests, give them simple tools to help them publish and use data, incubate startups and encourage thinking on fundamental issues such as data infrastructure and how to use personal data in a way that creates trust. We do this with large businesses, startups, charities and governments. We are a global voice for the better use of data to deliver social, environmental and economic impact.

The ODI is a not-for-profit and was founded five years ago by Tim Berners-Lee and Nigel Shadbolt. Both of them are at the yearly ODI summit which takes place at the British Film Institute tomorrow.

Bringing people together to solve common problems

The ODI team at the 2015 summit. Don’t let anyone convince you that diversity in tech is impossible, it’s not. Image by Paul Clarke, CC-BY-SA.

The summit is kind of unique, as is the ODI. It brings together large corporates with charities and startups; people interested in global development and democracy with people interested in the latest smart cities and transport trends; people from local government, national government and reps from global institutions. The attendees and speakers come from around the world. They all believe that openness and data can benefit them and everyone else too. (you can watch a stream of many of the summit sessions)

Which brings me to this all-party parliamentary group on data analytics. I’m a big fan of democracy and I’m also a big fan of things that bring together people from different backgrounds such as elected representatives and peers from across the political spectrum to find common points of interest, or problems, where people can work together to get things done and make things better. It’s the type of approach we use to help bring together large sectors like banking and agriculture, another one will be announced tomorrow. I won’t spoil the surprise. (it was sports)

An age of data abundance

We are in an age of data abundance with billions more people and devices coming online. It’s ever cheaper to collect, use and publish data. A web of data is evolving that sits alongside and behind the web of documents which changed our lives when Tim Berners-Lee invented the web 20-odd years ago. Our experience from the last 5 years is that that data will create most value when it is as open as possible while respecting privacy: an open future. But the future is uncertain.

We need to work together to shape an open future because whilst the current wave of technology change has bought many benefits it also carries many risks. Privacy risks, monopoly risks, democratic risks. We need to overcome those risks and project a positive message to get to a good future.

Tim famously said “this is for everyone” when tweeting about the world wide web from the launch of the London Olympics in 2012. The type of open thinking that Tim showed when he gave away the web is going to be necessary if we are going to realise the brilliant potential of this new web of data to benefit everyone.

And that open thinking is what we hope to see from this all-party parliamentary group. As well as the rest of us we need government and legislators to play an active part in making this happen. Government can lead by example.

Data for everyone

We can benefit everyone if we build data infrastructure (vital reference datasets like maps, lists of local authorities and addresses, and tools, processes, policy, legislation, organisations) which is reliable, adaptable, trustworthy, and as open as possible. Open in the sense of culture as well as open data.

We need to provide data skills for citizens, business and policymakers, with policymakers using data both for evidence and as a tool to achieve their policy ends.

And we need to encourage open innovation. A bridge between academic research, public, private and third sectors, and a thriving startup ecosystem where new ideas and approaches can grow. Innovation that solves problems.

We describe this as the open future. A future where we’ve understood and tackled those risks, made data as open as possible and created benefits for citizens, businesses and government. Data for everyone.

There were questions

After we talked the audience asked questions covering a whole range of topics from data in manufacturing and engineering; trust in use of data; public sector reform; EU proposals for copyright and how that impacted on organisations holding data; and whether people should be paid when their data is used. A wide range, as you’d expect from something that connects together and underpins sectors across the economy.

The last two questions I found particularly interesting. Both of them seemed to come from applying models from the real world to something, data, which has different qualities. Data is non-rivalrous, it benefits from network effects, etcetera. That’s why the economics of data are different from other things and still being researched. The questions also seemed to come from an implicit assumption that we could use the concept of ownership in the physical sense of the word. We need to be careful in how we use the language of ownership to address questions about data. Physical world metaphors don’t readily fit the data world. And even our understandings and expectations of ownership in the physical world aren’t as simple as they seem. This blog from Ellen Broad is a good read and what I channeled in my response. I hope the APPG thinks about those questions and the concept of ‘data ownership’ deeply. Its members will be part of shaping the legislative environment that will help us get to that open future.

An open city is a better city

2016-09-27 / / 0 Comments

Approximate words from a talk at the Holyrood Connect: Data Forum in September 2016. Approximate as I tend to ad-lib in person as I see shocked, or occasionally, pleased faces in front of me. I also had a bad cold so ad-libbed even more than normal. The slides are also available online.

— — — –

Hi, I’m Peter. I do some stuff at the Open Data Institute (ODI). I’m here to talk about how an open city is a better city.

First some background and a couple of concepts: the data spectrum and data infrastructure. Then some current examples of data analytics in cities, and their limitations, followed by some UK examples of people building more open cities with more benefits. I’ll end up with some principles to help get you started and a bit about what’s coming in the future. Ok, background:


Background

The ODI was founded four years ago by people like Tim Berners-Lee and Nigel Shadbolt. It is headquartered in the UK but its team works around the world. There are currently 29 nodes in 18 countries. In the UK that includes places like Aberdeen, Leeds, Belfast, Devon, Bristol and Cardiff.

The ODI’s mission is to connect, equip and inspire people around the world to innovate with data. We believe in knowledge for everyone. We help the public sector, third sector, academia and businesses to get more impact from data. Last week there were research fellows in the office from Madrid and Singapore debating and sharing ideas about geospatial data and privacy, crowdsourcing and smart cities. In the last few weeks the HQ team have been doing stuff in the UK, in Malaysia, New York, Mexico and Tanzania.

Concepts

The ODI works across the data spectrum. Some of us worry about personal health records being “made open”. Some confuse commercial and personal data, or mix up “big data” with “open data”. To unpack data’s challenges and its benefits, we need to be precise about what these things mean. They should be clear and familiar to everyone, so we can all have informed conversations about how we use them, how they affect us and how we plan for the future. And it doesn’t have to be complicated. It can be simple. In one image. Whether big, medium or small, whether state, commercial or personal, the important thing about data is how it is licensed and who can use it. Closed so that it can only be used within one organisation, shared can only be used by some organisations (because of rules or price restrictions), or open data that can be used by anyone for any purpose.


The ODI works to improve data infrastructure. Data has become vital infrastructure over the last few years. It underpins transparency, accountability, public services, business innovation and civil society. Data such as statistics, maps and real-time sensor readings help us to make decisions, build services and gain insight. Data infrastructure will only become more vital as our populations grow and our economies and societies become ever more reliant on getting value from data.

I often hear people say that data is the new fuel or that it’s oil for the digital revolution. Daft analogies. Data doesn’t get burnt up when we use it, we can use it again and again and again. It doesn’t get extracted from the ground: unless it’s geological data. The analogy we use for data infrastructure is roads. Roads help us navigate to a location. Data helps us make a decision. Roads have signs and maps to tell us how to use them. So does data, well hopefully.

Lots of cities are improving data infrastructure

Now back to the theme of cities and data. Cities and local authorities around the world are using and improving data infrastructure. It may not feel like it sometimes, but they are.

Many public sector organisations are developing skills and creating more impact by using their own data to make better decisions. Whether it be where to spend money on social care, what time to pick up the bins or how to design a local authority website so that it’s easy to use. In each case the organisation is having to learn how to gather data, analyse it and use it to make a better decision.

These are all activities in the closed part of the data spectrum.

Half-spectrum doesn’t give you all the value.

We’re also seeing more and more public sector organisations work together and share data to make better decisions. Down in Manchester local authorities are sharing data to help vulnerable children. In London local authorities are sharing and analysing data to look for unlicensed houses of multiple occupancy, they can be unsafe places to live. This type of big data analytics takes inspiration from places like Chicago which has been using data about graffiti tags to tackle gang violence, or New York City and Amsterdam which have analysed data from across the city to work out what characteristics were the best indicators for fire and help prevent it.

These activities take place in the closed and shared part of the data spectrum.

All the data and all the open

But let’s go back a bit. When I talked about data infrastructure I said it underpins transparency, accountability, public services, business innovation and civil society.

All of the previous examples are about public services. The rest of the benefits of data infrastructure missing. There’s some business innovation — for example from data analytics companies selling into the public sector — but only a portion.

Why is that ? Let’s look again at the full data spectrum. We’re missing public data and open data.

At the ODI we say that cities, their businesses and their citizens get most impact from a data infrastructure that is as open as possible while respecting privacy. There’s lots of research showing this and there’s also practical examples. I’ll cover some in a bit.

It’s true you know.

The reasons that open data infrastructure creates most impact is due to the qualities of data. For example, it benefits from network effects. Data becomes more useful and creates more value as more people use and maintain it.

When you work openly and use as much open data as possible then more people can work together to solve problems, make decisions, find insights and build services. You benefit from network effects. You can build a better city. One that benefits everyone.

This is particularly true if you combine all the data — closed, shared and open — with all the open. Open culture. Open source. Open government. Open standards. Open innovation. Etcetera.

There’s lots of examples, here are some

Let’s take a few examples showing some different aspects.

First, Bath and Strava, the cycling app. Strava users cycling around Bath can choose to share their closed personal data with a community group called Bath:Hacked. That group preserve privacy, analyse the data and are working with the council to use it to improve cycling routes. Interestingly there’s anecdotal evidence that people are cycling and using the app more because they can see that the data they collect benefits the city and themselves. Win win. Meanwhile Bath:Hacked are sharing what they’re doing online.

As a coffee drinker I am unsurprised by the decline in tea-drinking in Britain (source: Defra, ODI and Kiln)

There are two reasons for that. First, by opening up the knowledge for everyone other people can use it and other people can tell Bath how they are using it. People can learn with each other. Second, openness about how organisations secure and manage personal data builds trust. It can improve quality too. take Defra who recently did a privacy impact assessment in the open, with people outside the organisation commenting, before releasing diaries showing the diet habits of 150,000 households. They worked out by debating with their community that some of this data which would otherwise have all been kept closed could be made open for anyone to use. Transparency and open debate about personal data can make things better.

Another example, I was talking to someone from Devon council last week. They published a map of places where people could get help. Unfortunately the map was wrong. Because both the data and the source code were open a friendly person could fix it for them and send them the corrected version. Problem fixed within a few hours. Thank you friendly person.


Another. In places like Manchester and Leeds people from the public sector, private sector and civil society are working to build a low-cost open infrastructure for the internet of things. They’re helping each other using each other’s skills and experience as needed. On the infrastructure people will be able to build and deploy sensors to monitor air quality or the height of a river and anyone will be able to use the data to decide whether to place a new school near a road or a set of new houses by a river, whether to buy a house or whether to evacuate a house as the waters are rising…

These things cost money but they don’t need to cost the big money that so many projects with technology do. The cost of software, hardware and hence data is falling dramatically. You can now build an air quality sensor for less than £100, you can get a LIDAR sensor — a device that can measure distance using lasers — that used to cost tens of thousands of pounds for a few hundred pounds. (That’s part of the reason we’re hearing about automated cars so much. They need those sensors too). As much as possible of the data from that infrastructure will be open, that’s the culture of the community. That will allow other people to use it too for only the cost of allowing people to use the data that has already been collected. The infrastructure is designed for open.

And to continue the theme of culture. In Aberdeen the team in the council run hackathons open to anyone and learn innovative techniques from civil society businesses to help the council deliver other services. Those hackathons will also help with the Scottish government’s digital skills initiative that I was reading about on the train yesterday. An initiative that could also be supported by the new work that the Open Government Partnership are starting with the Scottish government.

Back to Leeds. The city council has funded ODI Leeds to act as a neutral space outside the council that can be used to convene businesses, academia, civil society and the public sector to understand and define problems; share data to explore ideas and then open the data as much as possible to allow people to build solutions. Those solutions could be built by new startups or established businesses. Arup, the global construction firm, use similar open innovation techniques working with startups to help improve how they build stuff. It’s like the data analytics examples we saw earlier but it uses the full spectrum.


In each of these cases we can see people from multiple sectors sector working together to solve common problems as openly as possible. In the process new businesses are built, there’s transparency and accountability, civil society are engaged, and there’s better public services too. All of the things our data infrastructure supports.

There’s countless more examples across the world for those who look.

How do I build open data infrastructure?

But, I often hear people ask, how do I do this?

As you may have realised from these examples data infrastructure is not only about data. Data infrastructure includes datasets; the technology, training and processes that makes them useable; policies and regulation such as those for data sharing and protection; and the organisations and people that collect, maintain and use data. We can all see that the datasets may be from anywhere in the data spectrum. But the more open the data infrastructure, the more value it will create as more people can use it.

Principles to help people build better data infrastructure.

Based on the ODI’s own work and research on what works and what doesn’t at city, national and globally we’ve published some principles to help other people build better data infrastructure.

The first and last principles are key. Design for open and encourage open innovation.

Based on our experience we believe we need a number of things to work together to create the space for open innovation to happen: strategy, policy, training, technology, research, a tech community, and engagement. With that engagement you’re looking to build a receptive internal customer (for example a councillor in a city), a responsive tech community and an engaged civic community willing to work with you. With open innovation the best answers can come from anywhere. You just need to get started and have the courage to try.

Anyway, I hope that was interesting, and useful, but before I go I want to leave with you another thought as to why getting to grips with open and data is so important.

The web of data is coming.


Over the last 25 years we’ve all been building the web of documents. Billions of webpages linked together. It’s fabulous. But the billions of people, sensors and services that are connected to the web and the internet produce, publish and use data. A web of data is now evolving that sits alongside and behind the web of documents.

That might seem like a challenging thing and something we can’t control but I would encourage everyone to see it as an opportunity. By getting to grips with your data infrastructure and making it as open as possible you will be positioning your city and the businesses and citizens that live in it to thrive in that future. That sounds like a pretty important mission to be cracking on with. It’s about building for the open future.

An open city is a better city.

There’s countless other examples to demonstrate why an open city is better and to help you understand how to grow your city in a way that works for your problems and your challenges. But, as a start, I’d encourage all of you to pick a problem and get started. Work together with your businesses and citizens to solve that problem and start building that open city and make things better for everyone.

Open addresses: will the address wars ever end?

2016-09-09 / / 1 Comment

This is the (rough) text of a talk I gave at the British Computer Society (BCS) Location Information Specialist Group’s 3rd annual addressing update seminar in August 2016. There were more jokes in person. And some Pikachu. The slides for my talk are also online as are those for Ant Beck’s talk.

Hi, I’m Peter. I do some stuff at the Open Data Institute (ODI). The ODI was founded three years ago. It’s mission is to connect, equip and inspire people around the world to innovate with data. Its headquarters are in the UK but it works around the world.

I’m here to talk about open addresses in the UK. To understand the tale it’s useful to start off with a (shortened) bit of history.

Ancient history…

Addresses and other types of geospatial data were early targets for open data releases. They are vital datasets that make it possible to build many, many services and products. Way back in 2006 Charles Arthur and Michael Cross wrote in the Guardian to ask the UK government to “give us back our crown jewels”. They pointed out the complex arrangements for maintaining address data and how the data was sold to fund those complex arrangements. They even pointed out the issues it generated for the 2001 census.

In 2009 the UK government announced that Tim Berners-Lee, one of the ODI’s founders, was going to help it open up data and in 2010 government said that postcodes and address data were going to be early releases. Victory!

Some of the tales from 2013

But it was a pyrrhic victory. Whilst government released many thousands of datasets the promised address data was not one of them. In 2013 the Royal Mail was privatised along with its rights to help create and sell that address data. The complex arrangements that were pointed out in 2006 just got more complex. And, in the meantime, another census happened with the inevitable, and costly, need to build another new address list.

The open data community was rightly sad, and probably got a bit angry. They knew how important that data was. They kept working to make things better. They didn’t just tweet, they organised.

More recent history…

In 2014 the Cabinet Office’s release of data fund provided some money to the ODI to explore whether it was possible to rebuild the UK’s address list and publish it as open data. The ODI pulled together lots of people who work with addresses to share and debate ideas.

The homepage of Open Addresses

This led to the launch of Open Addresses UK. I was one of the team working for Open Addresses. We worked as openly as possible with regular blogs and open source code.

We explored the benefits of better address data for the UK. We found that we could help fix problems such as the months it can take before new addresses are added to computer systems across the country. Months during which someone might not be able to order a pizza, get home insurance or register to vote. We looked at the economic evidence from case studies of other countries, such as Denmark, that have released address data as open data. If the success of Denmark scaled in proportion to the population of the country then the UK could expect to see an extra £110 million a year of social and economic value. Value that we don’t get at the moment because paid data creates less economic value than open data.

We looked at funding models. We started off with £383k of funding from the Cabinet Office. We got some extra funding from BCS (thank you). We knew that we would need to be able to show people what our services would look like before we could start bringing in funding from the users of address services.

From talking with potential users of those services we learnt about the challenges of address entry on many websites. User research supported our theory that moving to free-format address entry would both make life easier for many people and lead to better quality address data going into organisations. We built a working demo of that service.

We knew we needed to gather address data. Following on from the discovery phase we built a model that would allow any organisation or individual to contribute their own address data; that would allow anyone to add large sets of open data containing addresses if they followed guidelines and confirmed that they were legally allowed to publish that address data as open data; and put in place a takedown policy to investigate and remove any infringing data. For the legally minded, we were set up to host the data. This was important. In the past people had been threatened with legal action by the Royal Mail over address data and the hosting model provided a defence.

Unfortunately we hit a snag.

Digital cholera makes me sad.

We learned that one of the largest open data sets held by government was tainted by what we called ‘digital cholera’. It contained third party rights that government was not authorised to licence as open data. This was no good. We wanted to publish address data that was safe to use.

We didn’t want to spend the limited grant funding on more and more legal advice or court battles (sorry lawyers…). So we concentrated on other approaches.

We used clean open data sets and statistical techniques to multiply the address data we already had. For example, “if house number 1 exists and house number 5 exists then house number 3 probably exists”.

We started developing a collaborative maintenance model. People could use our address services to both improve their own services and improve the address data that everyone was using. The model would enable us to learn and publish new address information (such as alternative addresses — like Rose Cottage rather than 8 Acacia Avenue and new addresses) as people started to use them. This would increase the speed of publishing new information and improve data quality. By crowdsourcing data through APIs the data would get better as more people used it.

The team recognised that these new ways of collecting address data would impact on confidence. So, we started developing a model that would allow the platform to declare a level of confidence in each address. The model allowed for different levels of trust based on how frequently we’d seen an address, who reported it, and how long ago they’d reported it. Data users could use the APIs to determine confidence and choose whether to trust an address for their particular use case.

But all this time the clock was ticking. There was limited funding. From the beginning we knew that we were testing two hypotheses.

Two hypotheses. Both are true.

Unfortunately we discovered that both hypotheses were true. We could build much better address services using modern approaches, but the intellectual property issues would keep hindering us.

A report was published: to share the lessons of what worked, and what didn’t. As you’ll see in the report even with all of our mitigations against intellectual property violations in place, Open Addresses was only able to find one insurer who would provide it with cover for defence against Intellectual Property infringement claims. The insurers were too concerned that the Royal Mail would take legal action to protect their revenues from address data.

A blog was published about the shades of grey in open data. And then Open Addresses went to sleep.

Someone else would have to take up the challenge of opening up address data and making things better for everyone.

Meanwhile…

While Open Addresses was happening so were other things. Lots of things. I’m obviously interested in the data ones.

The ODI was thinking about who owned our data infrastructure. Data is infrastructure to a modern society. Just like roads. Roads help us navigate to a location. Data helps us make a decision.

Spot the infrastructure in this excellent picture by Paul Downey.

The government was also working on its policy of government-as-a-platform. Companies House were opening up their data and putting it on the web. The Land Registry described itself as a steel thread that we could all build on.

Things started to come together with the description of registers as authoritative list that we could all trust. We could all build things on top of government’s open registers.

Registers are data infrastructure. An important part of data infrastructure is geospatial data, like addresses.

Now

In the 2016 budget it was announced that government had allocated £5m to explore options to open up address data.

It is important to understand that this is about exploring options. As Open Addresses had learnt UK addresses are pretty complex. We have centuries of legacy to deal with.

Matt Hancock, who was the Minister for the Cabinet Office when the announcement was made, likened it to the ‘US administration (decision) to allow GPS data to be made freely available for civilian use in the 1980s, which he said had “kick-started a multi-billion dollar proliferation of digital goods and services”’.

He got the importance of this data being open. Not that surprising when you know that his parents ran a company that built “software that allows you to type your postcode into the internet and bring up your address”.

Government is building a common language about addresses.

Government is exploring the options as openly as possible. They are sharing their research into topics such as the need and complexity of address matching. and the need for a common language for addresses. They are trialling technology approaches, you can see the source code for yourself: it’s open. And this all forms part of the bigger picture of building registers as infrastructure for the government-as-a-platform strategy. In fact just this week government announced an early version of an authoritative register for English local authorities.

Whilst not all of the work is in the open (remember, the arrangements for UK address data are complex commercially and legally) it is clear that many government organisations — such as the Cabinet Office, Ordnance Survey, BEIS and Treasury — are working together to explore the options and business case for an open register. Good ☺

Will the address wars ever end?

All of the above is what I said in the talk at the BCS addressing update seminar. At the end the audience debated some of the issues raised. The legal issues seemed to confuse some people — derived database rights are tricky. Eventually I was asked the most important question: will this new UK government initiative to create an open address register succeed?

The honest answer is “I don’t know” but I do trust the people working on it. They are good and there is clear political will to get this problem sorted. With good people and political support it’s possible to do hard things. I choose to be optimistic. I think they’ll succeed. Good ☺

The web of data is coming.

It is important for the UK that they do. We need to build for the future web of data.

Other countries recognise the value of data infrastructure that is as open as possible. The USA, Australia and France have all recently made strong moves to get their address data open.

Data infrastructure is a competitive advantage in the 21st century. We need to move on from old licensing and funding models that don’t make the best use of the qualities of the web and data.

Let’s build better data infrastructure that makes things better for everyone.




Hacker Noon is how hackers start their afternoons. We’re a part of the @AMI family. We are now accepting submissions and happy to discuss advertising & sponsorship opportunities.

If you enjoyed this story, we recommend reading our latest tech stories and trending tech stories. Until next time, don’t take the realities of the world for granted!


Gov cats

2016-08-23 / / 0 Comments

In recent years the UK government has got into the habit of announcing that it has employed cats. Downing Street, the Foreign Office and the Treasury all have cats whilst the Cabinet Office are about to appoint one. An unusual habit for a government but, I suppose, life should be full of strangeness.

One afternoon I was feeling simultaneously bored and whimsical, a risky combination, so I spent 10 minutes building a UK gov cat register — a list of these cats — which I published on the web.

the cat register

The cat register is open data. Anyone can use it for any purpose. It is also open for contributions. Anyone can suggest changes and help improve it. Some people have done so already.

This week I created a dashboard for the cat register. That should have been relatively simple too but it took a little longer. Some of my skills are a bit rusty.

the cat dashboard

A list of cats that work for the UK government might seem like a silly joke – it was 🙂 – but it also gave me a chance to use, and give feedback on, some new tools developed by the Open Data Institute (ODI)’s Labs team.

Here’s what I did. It might help others publish some open data or build a dashboard. If you read it all you’ll also learn who Schrödinger’s gov cats are…

How I built cat register

I started off by pulling together some of the available data: names; the department the cats worked in; the dates when they started (or ended) their work; and social media accounts. Yes, UK government cats have social media accounts: both official and unofficial. The data was gathered into a spreadsheet application and saved as a CSV file.

I will shamefully admit that I did not think too much about the needs of potential users of the data. After all, this was a whimsical experiment which users would be able to help maintain if they wanted to be whimsical too. I also concluded that privacy would not be an issue as animals do not have rights under the General Data Protection Regulation. In less whimsical circumstances I would recommend completing a privacy assessment before publishing a dataset.

Octopub screen for adding a dataset

I used the ODI Labs’ Octopub tool to publish the CSV file. Octopub automatically creates an open data certificate and uses Github to store and publish the data with all of the functionality that provides.

After that step the data was accessible on the web, openly licensed to make it clear that people can use it and was open for collaboration so that people could help improve it. Do use the cat data, read how to submit some extra data or raise an issue if you want to.

This bit was easy. A dashboard was a little harder.

A minimum viable cat dashboard

To help with metrics and dashboards the Labs team have created Bothan: it brings you information in the form of a free platform for storing and publishing metrics as JSON or simple visualisations. This capability is built on top of another web tool, Heroku, that allows new applications to be quickly deployed to the web.

Bothan’s name is inspired by a pretty obscure line of dialogue about the many spies who died getting the plans for the death star in Return of the Jedi. I suspect the Labs team had many failures when building their tool…

The ODI’s lab teams have also built some sample code which can be copied and configured to present Bothan visualisations as a dashboard using Github Pages (another free tool).

Setting up a Bothan instance and reconfiguring an existing dashboard was relatively easy but automating the process of getting data, like the total number of cats, from the register into Bothan proved harder.

The team recommended Zapier, a web tool designed to help automate workflows. It’s less open than the other tools — I couldn’t easily share my config and the pricing plan seemed to scale fast — but it looked like it would do the job and help get even more cats on the web. The team have even integrated Bothan with Zapier to make it easy. Unfortunately I had to get to grips with the Python scripting language and my last foray into similar stuff was a while ago. Luckily there was help both on the web and in the office.

a bit of Zapier configuration which, to put it another way, says “if there’s a change to cat register, then run an algorithm and store the results in the Bothan metrics platform”

After getting the tech working I shared a couple of early drafts on twitter; got some feedback (at which point I learnt that Google had given me the wrong answer for the total number of cats in the UK (if only searching for data was as easy as searching for documents) and improved it to a point that I was happy to call it a minimum viable dashboard.

There is one bit of configuration and code looking for changes to the cat register and calculating new metrics for those values; whilst another bit is looking for changes to some official UK government data about cats. Everything runs automatically.

You will find a bit more detail and the code for the dashboard on Github. Feel free to suggest new features.

Peta is Schrödinger’s cat

Schrödinger’s cats

You might have noticed that the dashboard has an entry for “Schrödinger’s cats”. The reason for that is quite simple, just like the cat in Schrödinger’s famous experiment I could find no data that confirms whether some cats are alive or dead. I could make an educated assumption, after all one cat started duty in 1964…, but I thought it was worth leaving the status unclear. I simply left them marked“Inactive” and imagined the life of a retired UK government cat.

some cats from the swinging 60’s. Picture courtesy of National Archives via Wikipedia

Anyone who uses the data can make their own assumption about those cats whilst leaving it unclear might incentivise someone to help find the missing data and, perhaps, discover that an elderly cat from the swinging 60’s is still patrolling the corridors and clubs of Whitehall.

That incentivisation is interesting. A good register should, like any data infrastructure, be providing a foundation on which people can build services and find insights but a good dashboard should be incentivising behaviour in line with a particular goal or strategy. My goal was to get even more cats on the web. The register and dashboard was a way of getting other people to help me. Submit more cats.

Publish your own data or build your own dashboard

But enough of cats, for now. My whimsy also helped me explore a little bit of data publishing. Octopub, Bothan, Zapier and Python all turned out to be fairly easy to use so, if you fancy giving open data a go, why don’t you publish your own dataset or create your own dashboard?

You could start with a whimsical project (penguin register anyone?) or perhaps something more useful like this list of data science courses in Europe prepared as part of the ODI learning team’s work for the European Data Science Academy.

If the documentation for each of those tools doesn’t help you with a problem then there are plenty of people around to ask and, once you’ve learnt the answer, you can always suggest ways to improve the documentation and help the next person.

The hardest bit about publishing (cat) data is getting started. Tools like Octopub and Bothan are there to make it easy.

— — -

Update 21 April: since writing this blogpost I have done a bit more work on cat data, privacy and complexity.




Hacker Noon is how hackers start their afternoons. We’re a part of the @AMI family. We are now accepting submissions and happy to discuss advertising & sponsorship opportunities.

If you enjoyed this story, we recommend reading our latest tech stories and trending tech stories. Until next time, don’t take the realities of the world for granted!


Newcastle: data roundtable July 2016

2016-07-18 / / 0 Comments

Last Friday I was at Newcastle City libary at a roundtable event talking about “innovative uses of data in the North East”. It was fun. I thought it might be useful to share some high level notes.

Discussion points and attendees

The agenda had the following discussion points :

  • What can Newcastle do to become a genuinely ‘smart city’?
  • Could the North East region become a leader in open data?
  • How can the intelligent use of data improve the quality of life for people in the North East?
  • What lessons can be learnt from other cities around the world?

A range of attendees of differently sized organisations from both the public and private sector were in attendance:

Cobweb Information, Digital Catapult, Digital Leaders Northeast, Digital Union, Dynamo Northeast, Escher Group, , FSB Northeast, FutureGov, Geek Talent, Google, Google Deepmind, Middlesbrough Borough Council, Newcastle City Council, Newcastle City Futures, Newcastle University, North East Data Community, North East England Chamber of Commerce, Open Data Institute (me), TechUK, Teeside University, TechNorth. [Apologies if I missed any organisation/individual! Let me know and I’ll add/correct :)]

The session started with presentations from Newcastle City Council, Google and Newcastle City Futures

The event started off with a presentation from Joyce McCarty the deputy leader of Newcastle City Council. Joyce talked about the council’s responsibility to tackle inequality and need to share some data to deliver public services. She said that digital is how people run their lives nowadays and that some people struggled to get online to get benefits. Newcastle was an early adopter of superfast broadband and has funded free wi-fi across the city. Councils in the area are considering what role data should play in any deal for the devolution of power and control from Westminster to the North East. Joyce said that spaces like Google Garage — based within Newcastle city library — were important. They helped connect the council and innovaters so that they can learn from each other. The world will keep changing. Newcastle Council wants everyone to have the same opportunities and be able to benefit.

Andrew Eland from Google talked about the Google Mobility project to help tackle urban mobility using data. The project started off with a question “rather than telling people where there is congestion, can’t we tell people how to reduce it?”. It uses data published as open data by cities or provided by people using Google’s services. Google quickly learnt that they needed to provide cities with data which is accurate, private and useful. As Andrew said that they are now realising that “it’s not just about knowing the data about mobility, it’s about being able to change the infrastructure to act on what the data tells you”. The next question that Google might tackle could be “What does dynamic steel and concrete infrastructure that responds to data look like?”. I asked Andrew when the project would publish aggregated data as open data so that anyone could use it for any purpose. Perhaps combining the data with other sets about walking or cycling, like that published by Strava Metro, so that people could optimise city transport infrastructure for different modes of transport. I was told that the release of the Google Mobility product would be in about 6–12 months, Andrew said that he expected it to include walking and cycling data but was not able to confirm that Google would be able to release it as open data.

Mark from Newcastle City Futures talked about their vision of thinking about the whole place of Tyneside and bringing together communities to create a place that is owned and shared by people in the city and delivers innovation. He talked about the challenges Newcastle faces: an ageing population and a region where are 20% of the workforce are employed by the public sector at a time when significant cuts are being made to public expenditure. He said that Newcastle, and the North East, needed to think about governance, trust, how to reduce institutional overlap, how to think beyond five year electoral timescales and how to focus on delivering collaborative projects. Mark said that Newcastle City Futures had identified 20 potential projects in the last 8 weeks.

The roundtable talked about language, economics and building with communities

The roundtable started by talking about language and how ‘smart cities’ can seem exclusive and technocratic. Attendees discussed how we need to be as open as possible and build public services that meets people’s expectations.

We discussed the dividing line between the public and private sector. How cities could provide basic infrastructure to support innovation and make space for anyone to be creative. There was talk of fiscal devolution and the challenges it can cause when the beneficiaries of change sit in different departmental or organisational silos. Some people talked about creating commercial benefits through data sharing and by selling aggregated public sector data. I disagreed. The evidence says that public sector open data creates more value than paid data. We should try and build a better future, not replicate existing silos.

The room discussed different models for creating change and better use of data. Manchester’s approach is based on data sharing within the public sector preserving existing organisational bodies. Defra’s approach was to create culture change by setting a large target for open data release that made the organisations (or silos) within Defra work with each other and external stakeholders.

The consensus in the room seemed to be that the North East would benefit from something more ground up. As someone said “We do innovation and education really well. Let’s forget smart cities and focus on smart citizens.”

We reflected on the high ‘leave’ vote in the EU referendum in the North East and that many of the people that voted to leave the EU felt that the internet and technology was not a force for good. If Newcastle council believes, as I do, that these are a force for good then maybe it should look beyond physical infrastructure — like broadband and wifi — and help more people receive their benefits?

The roundtable ended by discussing Chicago’s approach to engaging citizens with data and technology. The city funds an organisation that sits outside the public sector, holds it to account, and builds capabilities with schools and existing community organisations. As best as I could tell no existing non-technical community organisations were in the roundtable but perhaps the approach used by Chicago may help create the kind of region that North East residents want?

(We didn’t have time to talk about the ODI Leeds approach)

What’s next?

As I said at the beginning: the roundtable was fun. There were differing opinions but a consensus that digital and data can be used to help build a future that meets the needs of the people that live and work in the North East.

From my own experience in the North East and from listening to the thoughts of Newcastle Council and Newcastle City Futures I suspect that the North East would welcome an open future.

There are many things that can help get to that future. There is no single right answer that will work in every city, region or nation.

Personally I’m looking forward to the next meeting to see what’s been tried, what’s been delivered and what lessons have been learnt!

« Older posts Newer posts »

Copyright notice

All content on this site is made available under a CC-BY-4.0 open licence unless stated otherwise.

© 2026

Theme by Anders Noren — Up ↑

This website stores cookies on your computer. These cookies are used to provide a more personalized experience and to track your whereabouts around our website in compliance with the European General Data Protection Regulation. If you decide to to opt-out of any future tracking, a cookie will be setup in your browser to remember this choice for one year.

Accept or Deny