Month: August 2016

Gov cats

In recent years the UK government has got into the habit of announcing that it has employed cats. Downing Street, the Foreign Office and the Treasury all have cats whilst the Cabinet Office are about to appoint one. An unusual habit for a government but, I suppose, life should be full of strangeness.

One afternoon I was feeling simultaneously bored and whimsical, a risky combination, so I spent 10 minutes building a UK gov cat register — a list of these cats — which I published on the web.

the cat register

The cat register is open data. Anyone can use it for any purpose. It is also open for contributions. Anyone can suggest changes and help improve it. Some people have done so already.

This week I created a dashboard for the cat register. That should have been relatively simple too but it took a little longer. Some of my skills are a bit rusty.

the cat dashboard

A list of cats that work for the UK government might seem like a silly joke – it was 🙂 – but it also gave me a chance to use, and give feedback on, some new tools developed by the Open Data Institute (ODI)’s Labs team.

Here’s what I did. It might help others publish some open data or build a dashboard. If you read it all you’ll also learn who Schrödinger’s gov cats are…

How I built cat register

I started off by pulling together some of the available data: names; the department the cats worked in; the dates when they started (or ended) their work; and social media accounts. Yes, UK government cats have social media accounts: both official and unofficial. The data was gathered into a spreadsheet application and saved as a CSV file.

I will shamefully admit that I did not think too much about the needs of potential users of the data. After all, this was a whimsical experiment which users would be able to help maintain if they wanted to be whimsical too. I also concluded that privacy would not be an issue as animals do not have rights under the General Data Protection Regulation. In less whimsical circumstances I would recommend completing a privacy assessment before publishing a dataset.

Octopub screen for adding a dataset

I used the ODI Labs’ Octopub tool to publish the CSV file. Octopub automatically creates an open data certificate and uses Github to store and publish the data with all of the functionality that provides.

After that step the data was accessible on the web, openly licensed to make it clear that people can use it and was open for collaboration so that people could help improve it. Do use the cat data, read how to submit some extra data or raise an issue if you want to.

This bit was easy. A dashboard was a little harder.

A minimum viable cat dashboard

To help with metrics and dashboards the Labs team have created Bothan: it brings you information in the form of a free platform for storing and publishing metrics as JSON or simple visualisations. This capability is built on top of another web tool, Heroku, that allows new applications to be quickly deployed to the web.

Bothan’s name is inspired by a pretty obscure line of dialogue about the many spies who died getting the plans for the death star in Return of the Jedi. I suspect the Labs team had many failures when building their tool…

The ODI’s lab teams have also built some sample code which can be copied and configured to present Bothan visualisations as a dashboard using Github Pages (another free tool).

Setting up a Bothan instance and reconfiguring an existing dashboard was relatively easy but automating the process of getting data, like the total number of cats, from the register into Bothan proved harder.

The team recommended Zapier, a web tool designed to help automate workflows. It’s less open than the other tools — I couldn’t easily share my config and the pricing plan seemed to scale fast — but it looked like it would do the job and help get even more cats on the web. The team have even integrated Bothan with Zapier to make it easy. Unfortunately I had to get to grips with the Python scripting language and my last foray into similar stuff was a while ago. Luckily there was help both on the web and in the office.

a bit of Zapier configuration which, to put it another way, says “if there’s a change to cat register, then run an algorithm and store the results in the Bothan metrics platform”

After getting the tech working I shared a couple of early drafts on twitter; got some feedback (at which point I learnt that Google had given me the wrong answer for the total number of cats in the UK (if only searching for data was as easy as searching for documents) and improved it to a point that I was happy to call it a minimum viable dashboard.

There is one bit of configuration and code looking for changes to the cat register and calculating new metrics for those values; whilst another bit is looking for changes to some official UK government data about cats. Everything runs automatically.

You will find a bit more detail and the code for the dashboard on Github. Feel free to suggest new features.

Peta is Schrödinger’s cat

Schrödinger’s cats

You might have noticed that the dashboard has an entry for “Schrödinger’s cats”. The reason for that is quite simple, just like the cat in Schrödinger’s famous experiment I could find no data that confirms whether some cats are alive or dead. I could make an educated assumption, after all one cat started duty in 1964…, but I thought it was worth leaving the status unclear. I simply left them marked“Inactive” and imagined the life of a retired UK government cat.

some cats from the swinging 60’s. Picture courtesy of National Archives via Wikipedia

Anyone who uses the data can make their own assumption about those cats whilst leaving it unclear might incentivise someone to help find the missing data and, perhaps, discover that an elderly cat from the swinging 60’s is still patrolling the corridors and clubs of Whitehall.

That incentivisation is interesting. A good register should, like any data infrastructure, be providing a foundation on which people can build services and find insights but a good dashboard should be incentivising behaviour in line with a particular goal or strategy. My goal was to get even more cats on the web. The register and dashboard was a way of getting other people to help me. Submit more cats.

Publish your own data or build your own dashboard

But enough of cats, for now. My whimsy also helped me explore a little bit of data publishing. Octopub, Bothan, Zapier and Python all turned out to be fairly easy to use so, if you fancy giving open data a go, why don’t you publish your own dataset or create your own dashboard?

You could start with a whimsical project (penguin register anyone?) or perhaps something more useful like this list of data science courses in Europe prepared as part of the ODI learning team’s work for the European Data Science Academy.

If the documentation for each of those tools doesn’t help you with a problem then there are plenty of people around to ask and, once you’ve learnt the answer, you can always suggest ways to improve the documentation and help the next person.

The hardest bit about publishing (cat) data is getting started. Tools like Octopub and Bothan are there to make it easy.

— — -

Update 21 April: since writing this blogpost I have done a bit more work on cat data, privacy and complexity.




Hacker Noon is how hackers start their afternoons. We’re a part of the @AMI family. We are now accepting submissions and happy to discuss advertising & sponsorship opportunities.

If you enjoyed this story, we recommend reading our latest tech stories and trending tech stories. Until next time, don’t take the realities of the world for granted!


We won’t give up, no matter how long it takes

This is a redrafted version of an article that was published by SixStars in The 1887 magazine sold at the Judgement Day 2 protest march in April 2016. More than 3000 fans marched. The march was organised by the Blackpool Supporters Trust and the Tangerine Knights.

It may seem like there’s a long way to go for Blackpool fans, but we should be proud of what we have achieved

Counting the fans in a recent picture by ex-Blackpool player (big) Ben Burgess showed 475 fans at a this week’s game vs Bolton. The club claimed there were 1,372 fans. Bolton had 2,261 away supporters. Nearly 5x the number of Blackpool fans.

The Oyston family have taken legal action against fans. Abused them. Taunted them. A fan was jailed. An unknown, but large, number of fans are banned from Bloomfield Road. Thousands of fans are boycotting and refusing to go back until the Oystons go. We’re not paying any more. I’m one of those thousands. More legal actions are on the way against more fans.

Protesting, like thousands of fans did at Judgement Day 2, takes time and effort. Boycotting your football club is hard: it hurts. Being on the end of legal action is horrendous. It can damage your job, your relationships, your life.

We all have doubts

Sometimes it’s hard to see if the boycotts and protests are achieving anything. Perhaps the Oystons will never leave?

After all the bid to buy the club by the Blackpool Supporters Trust was rejected, the Oystons refused to even negotiate a price. The club denied a recent bid even happened. The leader of the town council, Simon Blackburn, said the council “cannot take sides”. He is curiously silent on a matter that affects so many of the people he represents.

The Oystons have a long history in Blackpool and seem to want to cling on like a particularly unpleasant leech sucking blood out of its victim. Sometimes it can feel like failure is inevitable. That the club will sink and that no one will stand together with the fans and stand up for the club, the community and the town.

Perhaps we have to get behind the team despite the owners? Maybe a cheap season ticket offer will be available? Perhaps there’ll be some new players? Perhaps they’ll win a few games?

Many Blackpool fans will have had these nagging questions going around their head. We all have doubts.

We should ignore them.

We should ignore our doubts because the protests are working

I think the Blackpool fans have achieved something. The protests and boycotts are making a difference. You can tell by the reactions of the Oyston family. The protests and boycotts affect them. They reacted by taunting fans. They reacted by taking legal action against people trying to change the ownership of the club. They reacted by spending a tiny amount of the £90m Premiership windfall, just part of the millions that has gone into the club over the last few years, to buy some players. That must have hurt. Owen Oyston even attended an open meeting of fans for the first time in three decades.

But the protests also work by influencing other people. Crowd numbers have dropped to levels not seen since the late 1980s/early 1990s. Visiting fans spend less in the ground. Thousands of previous season ticket holders are boycotting the club. Ex-players and managers have spoken out. The opposition leader of the town council and the local MPs have called for action. The Oystons, and their disgraceful management of the club, regularly feature in the national press. There is the looming spectre of a court case from Valeri Belokon who owns 25% of the club.

Players don’t want to come to the club and supporters don’t want to go to matches. This makes it harder for the Oystons to keep hold of the club. Despite the tangled finances the club becomes less useful to the Oystons and possibly even damaging to their other businesses. More people now know how badly the Oystons run their businesses and treat people.

Perhaps as that court date approaches the Oystons might choose to approach the Blackpool Supporters Trust and start talking about a price to let them get out?

This doesn’t mean that we can expect a quick exit by the Oystons or that it will be an easy life afterwards. It took nine years after Wimbledon was stolen before its fans had a club back in the football league whilst eleven years after they were formed FC United Manchester have not made it into the league and there is unrest between fans in the stands and those in the boardroom. Some fan-owned clubs will face the same difficulties as ones owned by individuals like the Oystons. The grass is not always greener. We always need to scrutinise those in power whether the times are good or bad.

But we should take heart, the protests are working and getting the Oystons out will make the club and Blackpool a better place.

We won’t give up, no matter how long it takes

There’s another reasons we won’t give up.

Many of the people marching and boycotting have faced legal action from the Oystons or are banned from the club. Other people have been abused by the Oystons, whether directly or indirectly given the Oystons’ utterly unpleasant comments about people with special needs. Even if some of us give into our doubts then these people can’t go back unless the Oystons go. We should never forget, and I cannot forgive, the Oystons and the way they have treated our fellow fans.

Image from Blackpool Supporters Trust. If you’re not a member you should join them.

We are not just protesting for ourselves, we are protesting for other people. Blackpool fans are standing together, calling for the Oystons to go and for new owners who put football first.

We should be proud of our boycotts, our protests and of the work the Trust have done to show how a new democratic and fan-owned Blackpool FC could be run. We are fans. We stand together. We are a community. And we won’t give up, no matter how long it takes.

Gotta work together if you’re gonna catch ‘em all

A cuddly Psyduck from the 1998 launch of Pokemon Red/Blue. Picture by Mrs Gemstone, CC-BY-SA.

The launch of Pokémon Go has seen the rebirth of a 90s craze in the smartphone era. It’s a rebirth that leads to a tale about different worlds, data infrastructure, sensors and working together to improve our real world.

Pokémon Go is an AR (augmented reality) game. It is not the first AR game, it is not even the first where people roam their town catching monsters, but it is certainly the most popular so far. On one day last month 25 million people played it in the USA. The craze might fade but we will see a wave of AR games and products over the next few years.

Pokémon Go takes place across three worlds

We can think of Pokémon Go as taking place across three worlds: the real world that the player is walking around; the Pokémon world where the characters, gyms and pokestops are; and a world of data, or data infrastructure, that connects together the real and Pokémon worlds. Many other AR games and services will show a similar pattern.

A sample Pokemon map, with a pokestop in the pub just across the road from my flat.

Players wander the real world whilst their smartphone displays a map from the world of data with local parks and streets. The map also shows the location of characters in the Pokémon world that people can catch. When people try to catch a Pokémon the phone will overlay an image of the character on a live image of the real world captured through their phone camera. The map also shows places where people can collect items, called pokéstops, and gyms where people can battle each other using Pokémon characters.

The Pokemon world’s gyms and pokéstops are overlaid on real places: pubs, churches, restaurants that exist in the real world and whose name, type and location is also stored in the data infrastructure.

Pokémon Go shows us some of the gaps in our data infrastructure

Digital maps and lists of places pinpoint where we live, work and visit; connect us to local communities and services we rely on; and help us find our way around the world. They are important parts of the data infrastructure that underpins so many parts of a modern society and economy. I think about that data infrastructure a lot in my current job.

Unfortunately there are problems with our data infrastructure. If my address wasn’t in that data infrastructure then I might struggle to register to vote, get insurance or even order a pizza. Other parts of our data infrastructure are inaccessible because its locked away or priced too high for people to use. Other parts are simply missing or broken, no one’s got around to building or fixing them yet.

Despite this data infrastructure underpins Pokémon Go and will underpin other AR games and services in the future. We can see some of it’s weaknesses in this game.

A vicious looking Pidgey that got through security in a UK Parliament building. Characters in the Pokémon world don’t respect barriers in the real world because the barriers aren’t in the data infrastructure.

Shortly after the launch of the game someone who lived in a converted church reported that their house had been marked as a pokéstop and that players were gathering outside. The information about the place was out of date. People had to be asked to stop playing the game in Auschwitz museum. The information about this area of the real world was incomplete. Some areas and towns have few pokéstops or gyms because these areas are rural or contained few players of previous AR games so there is a lack of information. Pokémon Go is only playable in a few parts of South Korea due to national security concerns. Meanwhile any Pokémon player in an area with recent construction is likely to see bits of the map that are incorrect with missing roads or a park that has since been built over.

Collaboratively maintained data infrastructure will create better AR

Pokémon Go has ways for people to report problems with the data, but these are not easy to use and only pass data back to the people who control the Pokémon world rather than the people who maintain the underlying data infrastructure. As more AR services launch will the people who live in the converted church or people who live in poor and rural areas have to report the same problems to each of the new service providers? Will each service provider have to clean up the same data? This seems rather time-consuming and expensive when people can work together and solve the problem once.

Weak data infrastructure will be a a common problem for AR services, stronger, collaboratively maintained data infrastructure that anyone can use will provide a common solution.

A road found by a Pokémon Go player will be available to a car driver using Google Maps to find their way around a strange city. Players in one game can mark an area as out of bounds and players in other games will receive the same warning. Data about a new restaurant will be available to every service whether it be a game or an AR service that tries to entice you into that restaurant.

If the data is handled carefully, and privacy and openness are brought together to build trust, then perhaps some of the camera images could also be incorporated. Perhaps Pokémon Go could automatically spot and report a pothole caught on your camera whilst you were playing the game. People are more likely to trust uses of their data that respect their privacy and benefit society. This trust can lead to increased use of the AR service benefiting the service provider.

Collaboratively maintained and open data infrastructure will help build a better future

The benefits go beyond improving the AR services. Information about poorly mapped areas of cities can be gathered and used by AR services, public services or delivery firms. On election day a government could publish information about candidates and polling stations to the data infrastructure and every AR service would have access to the information and be able to incorporate it into their virtual worlds to encourage more people to vote.

The coming explosion in data. Image from the Open Data Institute, CC-BY-SA.

We are about to see an explosion in the number of sensors capturing data about our world. In Pokémon Go these sensors are smartphones and their cameras, but we are also seeing the rise of the Internet of Things; automated cars that will transport people whilst capturing data about the streets they travel; and delivery drones capturing images from the sky.

All of these sensors can capture data, respect privacy and publish useful information back to the underlying data infrastructure to help collaboratively maintain and improve it. By working together to maintain this data infrastructure and publishing data as openly as possible so that anyone can use it then it can help build a future which maximises the value we get from the data. To put it simply it can help make the real world a better place. One that can handle a growing population and our expectation for ever cheaper and better services. It will also improve AR games and might make Pokémon in their Pokémon world a bit easier to catch too.

It might seem strange to go from the 90s Pokémon craze, to an augmented reality game using three different worlds, to building a collaborative and open data infrastructure that can make the world a better place to live.

Perhaps it looks like a crazy dream that’s come from throwing too many balls at too many cartoon characters but it is an achievable future and one that comes from working together. Even if we start by throwing a small ball at some strange cartoon characters, we can still dream big.

© 2020

Theme by Anders NorenUp ↑