open service development and the Expenditure Database

On April 22, 2013 TBS released the Expenditure Database.  It allows one to browse and search expenditures organised in ways that are easier to understand than the "votes" system under which they are actually allocated.  It is great progress within the overall open government initiative.

However, from a technology, design and process perspective, I think there are some opportunities that are being missed.

Victorian Service Delivery

Minister Clement is fond of saying that the government's data is like grandmother's silver, hidden away.  In fact, our current model of delivering applications is more like hiding the entire kitchen from view.  In Ottawa's Victorian homes, the kitchen is usually at the back of the house, closed off from the dining room.  The servants were supposed to work there in obscurity, with only the final result appearing with a dramatic flourish in the formal dining room. 

In most of those homes that wall has now been smashed down, because we found that in the post-servant age, the kitchen is the hub of activity in the house, where we socialise and cook together openly, a very human and social activity.

But in government software development, that wall still stands.  Applications are developed behind closed doors by public servants, and then suddenly appear, fully formed, on release day. This model of closed software development has some very real consequences:

• There is no opportunity for issues to be caught and corrected during the development process; instead we have to wait post-launch to find design issues that are only visible after public use
• It makes application development seem mysterious and closed off from the world.  In a country where we want young people--in particular young women--to see software development as an interesting career, we've hidden all the real human aspects of development away, all the interactions, discussions, decisions, and compromises that have to be made as we assemble the software from the available ingredients in our coding kitchen.
• It closes the government developer community off from the much larger national and worldwide community of developers, both reducing the network that they can tap into as well as hiding our own government excellence and expertise.

We need not just open data, we need the entire philosophy of open source web development: we need to develop in the open.

Modern Government Service Design

The UK Government has released a fantastic Government Service Design Manual.  It has some explicit statements and some built-in assumptions.  The underlying concepts are to develop government services using modern software engineering processes.  This means being open about the code as it is in development (e.g. through blogs explaining the work and github repositories making the code available) and iterating through the design, from Discovery, through Alpha, Beta, Live and (all important in the government) Retirement.

The UK Government Digital Service does its work in the open, through blogs, twitter, and github.

Software development is a process: the application you release at a moment in time is not an end, what's important is that the application should tell a story about itself, so that it can be improved and so that it can be an inspiration for further work.

The Expenditure Database

The Expenditure Database is a great step in providing easy citizen access to the underlying data.  But it doesn't tell any of the story of how it works, how the data is processed, and who did the work.  The meal has been delivered and it looks very nice, but the kitchen and the process of making it are still closed off from view, hidden in mystery.

Code on the Web

Of course this is not quite as simple a story because it is a web application, so in theory you could look through all of the JavaScript.  But that means reverse-engineering the code, trying to figure out how it works without having access to documentation (the recipe) and the developers (the cooks).  When someone is trying to understand an application the first thing they will do is read its story about how it works and how it was put together, and then ask the developers when there are aspects not clear or not covered in the documentation.  Without open code development, this avenue of understanding is closed.

Instead one has to dig through the code and make guesses.  For example, it looks upon examination that it is drawing the data from a local file: http://www.tbs-sct.gc.ca/ems-sgd/edb-bdd/data.js

If instead the code were in a shared repository and documented, it would be much easier to understand how it works and the rights to reuse and modify would be a lot clearer.  As well one would expect there would be both examples of using the data file as well as an explanation of how it was generated.

Expenditure Database Development

What Expenditure Database development would look like if it followed this model:

• In advance of coding, in the Discovery phase there would be an internal call for requirements and expertise, e.g. on GCpedia (the Government of Canada internal wiki)
• An alpha using open source code would be quickly released
• The alpha would use public budget open data from data.gc.ca
• If the data wasn't available, it would have to be released (the rule is, internal developers shouldn't have any special or privileged access to information)
• A beta would be released to the public incorporating feedback and code changes
• At the beta phase at the latest, code would be shared on github (a public code repository and developer community) and there would be public explanations and documentation of the work going on.
• The developers would be using their own accounts; it would be clear whom to contact for questions about the software.
• Once the system goes live, there would be an ongoing process to incorporate improvements to the code, improve the documentation, and respond to internal and external questions from other developers and users.

What We're Missing

Just a few examples of what we're missing as a consequence of the current implementation:

• There are no permanent URLs to departments or searches.  That means if I want to point you at the expenditures for the National Research Council, I can't just give you a URL.  I have to say "go to http://www.tbs-sct.gc.ca/ems-sgd/edb-bdd/index-eng.html and search or scroll to National Research Council and then click on it and then scroll down and click on the details for various parts of it".  A good standards-based design and testing process should have revealed this issue.
• The Expenditure Database "stands alone" on a difficult-to-remember GC standard URL.  You have to know to navigate to it, or get the link from somewhere.  Why not tbs-sct.gc.ca/expenditures as a short link?  There is also no link from the database to data.gc.ca nor is there a link from data.gc.ca to the database.
• Without documentation about how the data were created, it's not clear if it's using budget data from data.gc.ca or whether it is using a custom database, and how the numbers are being generated; without access to the code there is a lost opportunity for other developers to learn how to parse, manage and display the budget data.
• Without the code, it's not possible for developers to either make their own websites using a similar approach, or to improve the Expenditure Database code itself.
• Without pulling the data directly from data.gc.ca there is no opportunity to explore all of the challenges that external developers need to solve, such as how to deal with versioning (what if a number has to be changed in the released data? how to tell if it has been changed?  how to see the previous version?) and performance (should there be an API instead of a download?  should the download be cached locally?  for how long?) and updates (how to tell when a new budget year has been released?)
• Without the code shared and the work publically described, people are left to speculate as to how the results are generated.
• Without the code shared and the work publically described, there is no entry point for external developers; there is no community-building for external developers.
• There's no contact point. The site gives no idea who developed the Expenditure Database, how it works, or who made the design decisions. The code and its development is made mysterious, the work of the kitchen hidden, rather than being demystified.
• There's no way to download the data displayed; you can only take a screenshot.
• There is no standard web navigation (e.g. the back button doesn't work)
• There's no way to compare departments

Additional information

TBS PR - Searchable Information on Government Spending is now Just a Click Away

Call for Feedback

I welcome your feedback on this post, with any clarifications, corrections, suggestions or pointers.

Thanks to Stephen Anthony, Adam Frankowski, Patrice Collin and Nick Charney for providing ideas and feedback that improved this post.