Science Library Pad

Problems with audiobook voice control interoperability on smart speakers

Telling your smart speaker to play one of your audiobooks seems like an obvious use case.

And indeed it is supported.  But with vendor-specific lockin only.  Each vendor will only let you voice control audiobooks from their own store.

I'd like to see the library tech community do an interoperability lab test for voice control of audiobooks on smart speakers.

Amazon smart speakers

The Amazon voice assistant Alexa can control audiobooks on Amazon smart speakers (e.g. Amazon Echo), but for Amazon Audible only (Amazon owns Audible).

See Audible (an Amazon Company) - Voice Commands for Listening on Alexa Devices.

Google smart speakers

Google's voice control is called Google Assistant, and it can control audiobooks on your Google smart speakers (e.g. Google Nest or Google Home), but for Google Play audiobooks only.

See e.g. Listen to audiobooks on your Google Nest speaker or display and Google Assistant Help - Listen to audiobooks.

Apple HomePod smart speakers

Information below for iOS 17 / iPadOS 17 / HomePod 17.

The situation with the Apple HomePod is even stranger, as it does not currently have any built-in Siri support for playing audiobooks from Apple Books.

It currently only supports voice control of Apple Music and Apple Podcasts.

Oddly, you can use Siri to CarPlay audiobooks in your car no problem though.  See Play audiobooks with CarPlay.

Sonos smart speakers

Sonos supports three voice assistants: Sonos Voice Control, Amazon Alexa or Google Assistant.  This would seem like an awesome way to have multiple services available together, just distinguished by saying Hey Sonos, Hey Alexa, or Hey Google.  But that's not how it works, you have to pick one assistant and that's the only one that will be active (I can't see any technical reason why this should be the case, but it is).

However, on Sonos the voice assistants can't control audiobooks.  So you can connect your Sonos to your Amazon Audible account, but you can't use Amazon Alexa to control the audiobooks (again, there is no technical reason why this should be the case).

See How can I listen to Audible using the Sonos app?

Audible with Alexa integration is not currently compatible with Sonos devices.

Similarly, you can't use Google Assistant to control audiobooks on Sonos either.

A nonsensical mess

This is a nonsensical mess.  These services should all just interoperate.

Libby, Spotify and Audiobooks.com

There are other major sources of audiobooks, including OverDrive Libby used by most libraries, Spotify audiobooks, and Audiobooks.com (I think Audiobooks.com is owned by Storytel, it has had a chain of corporate owners).

SIDEBAR: If this looks like a diverse market, the underlying source for most audiobooks is Amazon Audible.  END SIDEBAR

As far as I can tell, there is no support for voice control of Libby or Spotify audioboks.

Audiobooks.com does have an Alexa skill with a long but reasonable command string e.g. "Alexa, ask Audiobooks dot com to play my book" but I assume this only works on Amazon smart speakers.

September 2023 - Apple HomePod 17 "Third-Party Music Services"

Apple has just, as in just this month in HomePod software version 17, released support for Siri integration with "third-party music services", by which it means anything other than Apple.  This is basically a standard way for Siri to start and control an AirPlay session from an iPhone or an iPad (I don't know if it would work from an Apple TV or a Mac).  It requires iOS 17 / iPadOS 17.  In essence Siri gains the capability to run what are currently manual steps of launching an app on a device, then selecting the content you want to play, and then selecting AirPlay to your HomePod.

Technical Details - SiriKit Media Intents

Apple puts the onus on the apps for "third-party services" such as Audible, Google Play, Libby, Spotify and Audiobooks.com to code support for Siri audiobook control.  This is done using Siri Media Intents.  I found it was clearly explained in the Apple WWDC2023 session Integrate your media app with HomePod (video).  There is specific support for podcasts and audiobooks.

This is actually a good standardized API to make a universal experience across different services.  One can hope that the other services will add this support. 

Technical Details - Personal Voice Recognition

Unfortunately, depending on AirPlaying from a device means that Apple has to be able to tell which device, and to know that you're authorized to play content from that device.  This means that you have to enable what Apple calls Personal Requests for HomePod, a service that requires Recognize My Voice to be enabled.

This means e.g. you won't be able to ask HomePod to play content for someone else, I don't think there is a concept of other authorized users.  If I understand how this works, if your grandfather wants to play audiobooks from his iPad, only he can request it from HomePod (which is also going to make setup and testing more challenging).

HomePod 17 - Whither Apple Books?

It seems like Siri should be able to play Apple Books directly on-device without the elaborate hoop-jumping of AirPlay to an iPhone or an iPad, but at a minimum one would expect Apple Books to be a demonstration of support for the SiriKit Media Intents needed to play audiobooks.  It's not clear if this is the case yet in iOS 17 / iPadOS 17, which is kind of puzzling.

---

Crossposted to my tech blog.

my review of The World is Flat

NOTE: This 2006 draft blog post is being published in 2022; it may be incomplete and the information is certainly out of date.

Despite having a huge stack of books to read, I found myself on vacation without anything to read, so I ended up getting The World is Flat (Updated Edition) mostly by accident.

I had rushed into the mall to get the Globe, which no one was selling, so on a quick scan of the bookstore, I thought, hmm, if I'm going to sling around the term I should probably actually read the book.

At 566 pages, it's no light read, but Thomas Friedman presents a comprehensive framework for understanding the impact of technology on our societies.

I had assumed the book was mostly about outsourcing, but it actually covers a wide range of current developments in work and technology.  Friedman describes our society as experiencing "Globalization 3.0".

I was once again surprised, particularly given that he was even using "3.0" terminology, that there was no mention of Toffler's The Third Wave but other than that, the book is very strong.

It covers technology change with explanations that are both clear and comprehensive.  The last thing I would have expected in a book about globalization would be to get a good discussion of underlying technology change, but lo and behold, I read

"But then," added Microsoft's Mundie, "we said to ourselves, 'Geez, if we really want to automate everything, then we need to make it much easier not just for people to talk to people, but for machines to talk to machines -- for machines to interact with other machines about any subject without any humans involved at all or any a priori relationship between the different companies whose machines are communicating."  This was the next work flow breakthrough.

Technically, what made it possible was the development of a new data description language, called XML (extensible markup language) and its related transport protocol called SOAP (simple object access protocol).  Together, they allow any two computer programs to exchange formatted data or documents that contain any form of information.

(from "The Ten Forces that Flattened the World - #3 - Work Flow Software" - page 82)

Issues

"globalization" is missing from the index.

There is no comprehensive reference list or bibliography.  While he does a great job of citing things within the text, it would be nice to have them all gathered together at the end as well.

Microsoft Dev Box cloud-based development environments

Microsoft Dev Box, a new cloud service that provides developers with secure, ready-to-code developer workstations for hybrid teams of any size. Microsoft Dev Box empowers developers to focus on the code only they can write, making it easy for them to access the tools and resources they need without worrying about workstation configuration and maintenance. Dev teams preconfigure Dev Boxes for specific projects and tasks, enabling devs to get started quickly with an environment that’s ready to build and run their app in minutes. At the same time, Microsoft Dev Box ensures unified management, security, and compliance stay in the hands of IT by leveraging Windows 365 to integrate Dev Boxes with Intune and Microsoft Endpoint Manager.

from Microsoft Tech Community - Azure Developer Community Blog - Introducing Microsoft Dev Box - May 24, 2022

Blog post covered e.g. in Ars Technica - Microsoft Dev Box will virtualize your Windows development PC in a browser window - May 25, 2022

I've written before that just getting a development environment set up with all the necessary supporting libraries and dependencies sorted out is a major barrier to casual development these days, and that language-specific cloud services such as Google Colaboratory for Jupyter Notebooks help address this.

Additionally in a corporate environment there are often rights restrictions that limit the end-user's ability to install and manage software, which can make it difficult to maintain a complex software development environment. 

Pre-packaged virtual machines in the cloud offer a potential quick path to productive development environments.

This may also help scientific researchers who may want to do software development but have neither the time nor expertise to install and configure a complete environment themselves.
Additionally as MS indicates, this means access to a full Microsoft development environment even if your underlying platform is macOS or Linux.

because Microsoft Dev Boxes are hosted in the Microsoft cloud, you can access them from anywhere: Windows, MacOS, Android, iOS, or your web browser

Microsoft Dev Box has Microsoft 365 integration (unsurprisingly) which may also make it an easier fit for corporate environments.

Currently in private preview, public preview coming "in the next few months".

Recognizing an open data success

Note: This is a technical examination of available COVID-19 open data.  For general information please visit your relevant local website, including:

• Canada.ca/coronavirus
• Ontario.ca/coronavirus
• OttawaPublicHealth.ca/coronavirus

Summary

Sometimes when advocating for a particular change, it can be hard to step back and recognize the progress that has been made.

COVID-19 open data in Ontario is available and is provided in a way that meets many of the goals that open data advocates had set out.  The City of Ottawa and the Province of Ontario both prominently link directly to COVID-19 open data on their dashboard pages.  The Government of Canada also links to open data, but not as prominently or consistently.  There is also an academic site providing data and code for Ottawa.  All sites provided clear licensing.

These data visualisations and open data can be considered a kind of infrastructure.  In the context of academic use of open data, particularly with the addition of open source computer code, these sites can even be considered part of a digital research infrastructure.

Goals for Open Data

There were several elements that were envisioned for open data in general:

• available
• open license
• usable - clear descriptions
• usable - machine readable
• visible, in the sense that clear links to underlying data would be provided e.g. on websites
• discoverable through search

Basically, that anyone could find it and use it.  That it would provide a usable machine-readable alternative to scraping HTML web tables or hand-extracting data from PDF or Word documents.  And also that it would be provided from every organisation at every level of government.

For sustainability, it was also hoped that open data would be part of each organisation's own internal processes:

• used by the service providers themselves

There is an additional key element I have to say wasn't strongly considered by many of the initial advocates, which is that there should be tools provided to view and work with the data without having to go in and do your own spreadsheets or write your own code:

• tools are provided for anyone to work with the data

And I would add lastly, as the community thought through the provision of tools (e.g. website visualisations):

• are the tools themselves made available for open reuse, e.g. can you access the underlying code and pipelines that are used to make the website views

These goals sometimes seemed very far off.  But with COVID-19 data in Ontario, I would consider it a success.

COVID-19 open data is available at the municipal level (e.g. City of Ottawa), at the provincial level, and federally.  (And beyond, through international aggregation sites.)

 

City of Ottawa

Ottawa COVID-19 Open Data

Ottawa's open data site is https://open.ottawa.ca/

COVID-19 open data can be found just using "COVID-19" as a search.  In this case I've also filtered for just CSV format files.

https://open.ottawa.ca/search?q=covid-19&type=csv

There are 8 datasets, including:

• COVID-19 Cases and Deaths in Ottawa
• COVID-19 Weekly Cases and Rates by Age in Ottawa (Last 6 Weeks)
• COVID-19 Reproduction Number (R(t))
• Data tables for Public COVID-19 Maps

The only one that doesn't seem to be updated frequently is "Data tables for ONS Neighbourhood COVID-19 Maps".

So the good news is the data exist, the data are being kept up-to-date, and the data are in a usable, machine-readable format.

So let's look at three more aspects:

• Is the data surfaced with tools for people to work with the data?
• Is the data clearly linked when it is used?
• Do the city's own websites/tools use the open data (and is it clear that they do so)?

Ottawa COVID-19 website

The main general use site is Ottawa Public Health - Statistics on Coronavirus 19 (COVID-19) in Ottawa, it has links to:

• Daily COVID-19 Dashboard
• COVID-19 Vaccination Dashboard
• Mapping Products
• Wastewater COVID-19 Surveillance
• Projections

The dashboard is in Microsoft Power BI, which I have seen used on a number of COVID-19 information sites.

The data pipeline is clear, but I don't think it is one that is available to the public: "Data are extracted from [the provincial Case and Contact Management (CCM) reporting system] at 3 pm daily and loaded to the dashboard the following day."  In other words, it doesn't look like the Power BI visuals are generated directly from the Ottawa open data, but instead from the provincial closed system.

Open Data Clearly and Prominently Available

However, the really good news is right below the explanation of the data source used (CCM), and before showing the dashboard, the page links directly to the open data: "The following data tables are available for download on Open Ottawa". 

So this meets a key goal, it makes the data a kind of equal, so that you easily click the links and do your own analysis whether in a spreadsheet, by writing code, or using other data analysis tools.

It's hard to overstate what a big change this is in how data is provided to the general public.  Linking to the open data at all is a big deal, but linking to it right at the top before the visualisation is a really strong and clear statement of openness.

There is a gap in that as far as I know there's no way to access the underlying Power BI setup (which incidentally generates five different pages, use the < > way down at the bottom of the Power BI frame on the page to see other Power BI pages), which means that you can see the really complicated visuals but you don't get much insight into how they're made or how you could use Power BI to make your own.

There is also a gap in that, I guess due to different cut-off time for updates, the municipal dashboard numbers are different from the provincial ones.

In terms of discoverability there are basically two pathways to the open data, one directly from Open Ottawa and one from Ottawa Public Health, but they are not quite equal.

If you land directly on Open Ottawa, you would find out it has COVID-19 open data even without a search as they are surfacing several datasets (I would guess this is automated based on traffic to the datasets, but it could be manually curated).

But you wouldn't find wastewater data, because it's on a different site https://613covid.ca/wastewater/

The wastewater site uses more of an academic format, and presents all of the data visualisations up front, before you find out below that you can "See the Methods page for more information on how the samples were collected, access to the data, and how the plots were created." 

There are some issues at this point that are more web design and user-centric design than data.  For example the Methods page is actually called model, and you can't get directly to the top of the page by clicking the Methods menu item, you have to click "Projection models" within that menu.  If you're looking for data you're more likely to pull down the Methods menu and go to "Plot data", but that won't take you directly to the open data - basically you will have to navigate around the page quite a bit before you find out that the data are actually linked under "Waterwater surveillance".  The data are made available on GitHub.

https://github.com/Big-Life-Lab/covid-19-wastewater

So not only could the wastewater site benefit from more directly and clearly linking to the GitHub data near to where it is shown in visualisations and tables, but you can't find the wastewater data at all if you start on the Open Ottawa site.  So there is a gap in integrating these different open data sources.  However, the use of GitHub does provide another path for discoverability, and adds the ability to easily share not just data but also code.

The situation for data is similar for COVID-19 projections, which can be found at

https://613covid.ca/

The source data for the projection models is the Ottawa open data, which is a good clear reuse of the data directly.  The projection models themselves are drawn from a variety of clearly linked sources.  You can see that the wastewater data is kind of added on to the models, which may be one reason for some content visibility issues in the website design.  By navigating on GitHub I can also find that the 613covid.ca projections website is generated from

https://github.com/Big-Life-Lab/Ottawa-COVID-Projection

but I don't think you can find this on the projection website itself.  It's good that the website content is on GitHub as it means pull requests can be submitted with suggested changes; however creating and submitting pull requests for a site built from GitHub is a high barrier for most users.

 

Province of Ontario

Ontario COVID-19 Open Data

The Province of Ontario's open data site is https://data.ontario.ca/

COVID-19 open data can be found just using "COVID-19" as a search.  In this case I've also filtered for just CSV format files.

https://data.ontario.ca/dataset?keywords_en=COVID-19&res_format=CSV

There are 19 dataset pages, which may link to more than one dataset per page.  For example, to find daily new cases you first go to the page

Status of COVID-19 cases in Ontario

and then select the dataset

Daily change in cases by PHU

(PHU means Public Health Unit.)

Having a full page means that a lot of explanatory and related information can be provided.  The data website itself has good functionality including the ability to preview a CSV dataset on the website itself as a grid (spreadsheet-like view), as a graph, or as a map.

https://data.ontario.ca/dataset/status-of-covid-19-cases-in-ontario/resource/8a88fe6d-d8fb-41a3-9d04-f0550a44999f

This is a great way to allow people to explore data without needing their own tools.

There is also an API available for the datasets.

https://data.ontario.ca/api/1/util/snippet/api_info.html?resource_id=8a88fe6d-d8fb-41a3-9d04-f0550a44999f

Ontario COVID-19 website

The main general use site is COVID-19 (coronavirus) in Ontario.

Right below a website display of COVID-19 cases, it has a prominent button:

Get COVID-19 data and details

and next to it,

Download the raw data from the Ontario Data Catalogue

So you have two highly visible pathways to the data.  The data and details page provides tables, maps and charts, so you can work directly with the data on the web without needing specialised tools.  At the bottom of the data and details page, it again links to the underlying data, in a "Where the numbers come from" section.  You are left a bit to navigate the data catalogue to figure out which datasets generate which charts.  The page would benefit from adding a CSV download button with each visualisation.

For example, I had to try several different data catalogue pages and examine a number of datasets before finding the new cases dataset.

Nevertheless, this is overall a really strong open data success, with multiple really clear linkages to the open data.

If you land directly on the Ontario Data Catalogue, you would find out it has COVID-19 open data even without a search as they are surfacing several datasets both under What's New (which I'm guessing is manually curated) and in the lists of Most popular and Recently updated datasets.

I will note however that their custom 2019 Novel Coronavirus group includes many more datasets than you would find just by searching COVID-19.  There is a pathway from a dataset to the group, but you would have to notice that there is a Groups option at the top of every dataset.

 

Government of Canada

Government of Canada COVID-19 Open Data

The Government of Canada open data site is https://open.canada.ca/en/open-data

COVID-19 open data can be found just using "COVID-19" as a search.  In this case, I've filtered to include CSV, JSON, GeoJSON, XLS, XLSX, XML only:

https://search.open.canada.ca/en/od/?sort=score%20desc&page=1&search_text=covid-19%20OR%20coronavirus&od-search-col=Non-Spatial|Open%20Maps&od-search-format=CSV|JSON|GEOJSON|XLS|XLSX|XML

With these filters, there are 118 datasets.

If you look at e.g. Cumulative number of COVID-19 vaccine doses administered in Canada by jurisdiction

https://open.canada.ca/data/en/dataset/3b75a8d6-c5a9-48f9-834b-626eec16363f

you can see a clear description of the dataset and a link to download it in CSV format, but no tools to work with the dataset on the page itself.

Government of Canada COVID-19 website

The main general website is Government of Canada - Coronavirus disease (COVID-19).

A page about data can be found linked right in the top section

Current situation - COVID-19 data trends

Once we get there though, the situation is somewhat mixed.

The first link, Daily epidemiological report, takes you to many visualisations.  But it is really PDF bound, including a link to a PDF version of the page at the top, as well as (somewhat indirectly) linking to a weekly PDF summary (week of April 9, 2021 in this example).

There are some links to underlying open data, e.g. to CSV open data underlying "Current situation", but you have to be pretty sharp-eyed to spot them.

That being said, on the data trends page, there are two links in trend data that will take you to data:

• Downloadable data on COVID-19 across Canada takes you to the Government of Canada open data website, but it takes you to a single dataset, Public Health Infobase - Data on COVID-19 in Canada rather than a group or search or summary page listing all of the 100+ COVID-19 datasets.
• You can also find CSVs linked from some of the visualisations in Visual data gallery: COVID-19 graphs, charts, maps and more.

Overall I would say that while kudos is due to the Government of Canada for linking to open data, it could take lessons from the City of Ottawa and the Province of Ontario in more prominently and consistently surfacing the open data as part of its websites.

Similarly, if you land directly on the Government of Canada open data website

https://open.canada.ca/en/open-data

you get no indication that it has COVID-19 open data, unlike the prominent highlighting of COVID-19 open data on both the Ottawa and Ontario websites.

International

I will mention very briefly Our World in Data, which aggregates and then republishes international open data.  For COVID-19 specifically see

https://ourworldindata.org/coronavirus

There is a clear open license at the top of the web page, and there are visualisation tools (charts) provided.

Notably both the data and the code are made available on GitHub.

https://github.com/owid/covid-19-data/tree/master/public/data

Sidebar: Coding and Open Data

It is now much easier to work directly with open data in modern programming languages, particularly with CSV format data.  As an example, you can see my previous blog post about Juypyter Notebooks (Python coding) using Google Colaboratory, and a Jupyter Notebook that I created to chart new cases of COVID-19 in Ottawa is available on GitHub: https://github.com/scilib/ontario_cv19_opendata_plot

License for this blog post

This blog post is Copyright © 2021 Richard Akerman, and the text and associated images are licensed in the Creative Commons BY-NC 4.0 International.

This blog post is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

This license applies only to this blog post and not to the entire blog as a whole.

Jupyter Notebooks in Google Colaboratory

One of the issues with modern coding is the barriers to getting started.  As I wrote in Can't Code There From Here, programming languages used to just be available, already configured at your command prompt, but now to start coding you have to download, and install, and then update, and then add code library upon mysterious code library.  To quote my previous posting:

We've frontloaded one of the most tedious parts of computing, setting up a working coding environment, into the beginning of every code adventure.

And in an environment where there are constant cybersecurity issues, including attacks on coding supply chains, it's not exactly comfortable either to watch line after line of mysterious installs scrolling across the screen, or to try to wade through dependency alerts every time you try to write some simple code.

For these reasons, I've found Jupyter Notebooks in Google Colaboratory to be quite refreshing, as it's all already set up for you to access from your Google Account.  No downloads, no install, just Internet ("cloud") coding.  It is a research offering, so be mindful that it may go away at any time, but for now it gets you back to being able to code without any setup.

The good news is it is very straightforward on-screen.  You can have text blocks, with, well, text (in Markdown) and separate executable Python code blocks.  You can mix text and code blocks for a verbose, explanatory coding style.

File Access

Getting files (e.g. CSV data files) into Colaboratory is fairly straightforward.  To get files from your Google Drive you have to mount the drive

from google.colab import drive
drive.mount('/content/drive')

and then get an authentication code.  There's information on the Colaboratory site - Mounting Google Drive locally.

Once mounted your actual Google drive will typically be one layer down, /content/drive/My Drive/

You can also access files from your local file system, and apparently there is some kind of direct access to Google spreadsheets as well.

.ipynb files

The sort-of bad news is that the nicely formatted text and code within a Jupyter environment turns into a jumble of JSON when saved in the .ipynb plain text format. You can't really work with Notebook .ipynb files outside of Jupyter. It's all chunks of

 "cells": [
    {
      "cell_type": "markdown",

etc.

UPDATE 2021-04-04: I didn't realise that a Jupyter Notebook really is a notebook, in the sense that it preserve the last state of its output.  I was wondering why a small .ipynb notebook file that I emailed to myself was 1MB, but it's because the .ipynb stores the last output state (including charts).  I only found this out when viewing a notebook I had downloaded from Colaboratory and then uploaded to GitHub.  Clicking on the notebook in GitHub shows not just the code and text, but all the outputs from the last code run at each stage.

If you don't intend to also share the outputs from your last code run, clear the contents before sharing a notebook. 

Thanks to Juan Miguel Lavista on Twitter for additional information making this clear.

END UPDATE

GitHub Integration

In theory you can integrate Colaboratory directly with GitHub, but that means full read/write access.

If you're not keen on tying your Google Account and GitHub account together in this way, you can just do a more indirect download and upload pathway. 

Colaboratory can read files directly from GitHub with no special permissions, just go to File -> Open notebook and select the GitHub tab.  It didn't behave the way I expected when I put in a GitHub URL, but it worked fine when I put in my username.

Within GitHub itself there's not a lot you can do with a notebook file, as far as GitHub is concerned it's a "non-code file".

UPDATE 2021-04-04: As indicated in the previous section, you can view a Jupyter Notebook as basically a web page, including any outputs that were saved with it, by clicking on the .ipynb link in GitHub.  END UPDATE

End Program

And that's about it.  It is not of course exactly a free-for-all.  There are some kind of RAM and disk limits that you will hit at a certain point.

If you do want to go the locally-installed route, there are several options for Jupyter Notebooks including Anaconda and Microsoft Visual Studio Code (for Windows, Mac, and Linux), but they require not only the aforementioned multiple dependency levels of downloads, but in a work environment may also require passing a proxy parameter if your web connection goes through a proxy.

Canada's Digital Research Future - Call for White Papers

Canada's New Digital Research Infrastructure Organisation (New DRI Organisation, or NDRIO) has put out a call for white papers.

https://engagedri.ca/ndrio-call-for-white-papers-on-canadas-future-dri-ecosystem/

Submissions should briefly describe individual perspectives on current challenges accessing Digital Research Infrastructure (DRI) tools, services and support; the ideal future state of DRI in Canada; and how NDRIO could achieve such a state.

Deadline for submissions is 17:00 Canadian Pacific Standard Time (PST), Friday December 4, 2020.

NDRIO's scope includes Research Data Management, Research Software, and Advanced Research Computing / High-Performance Computing, for Canadian researchers.

Contact [email protected] for general details, or Felipe Pérez Jvostov, Senior Analyst for Outreach and Engagement at [email protected] about this specific call.

Disclaimer: I work on the Digital Research Infrastructure Strategy at ISED.

Can't Code There From Here

December 5-11, 2016 is Computer Science Education Week, and thus also the main Hour of Code week.

Hour of Code used to be pretty straightforward, back in the day.  You just got access to the computer and then

That was it.  Everything else was up to the student.

Ok, it was a little more complicated than that.  In actual fact, there was an entire ecosystem in the 1980s that supported entry into computer programming (unfortunately mostly for boys, since one of the main routes was through games that were primarily played by boys - for more see When Women Stopped Coding).

There were basically three computer worlds:

• computers you had access to outside the home, typically through school
• your home computer (Commodore or Atari or TI or Tandy)
• video games in the arcade

And there were a few paths into programming:

• exiting out of running computer games or other code, and looking at the code and modifying it to see what would happen
• trying out commands just to see what would happen, just to get your computer to do something
• typing in code from magazines and books
• being inspired by arcade games to try to code some kind of game

Alas, just like we let the blogging ecosystem fade away, it seems we have let the coding entry-level ecosystem fade away.  The extraordinary thing is even companies where most of the coders learned in the above way, by e.g. writing BASIC on their home Commodore 64, there is no support for this kind of entry path into coding.

David Brin wrote about this issue a decade ago, in 2006, in Why Johnny can't code.

Since then, quite frankly, things have gotten worse.  In 2006 at least people were still sitting in front of desktop computers.  Now we have smartphones and tablets that put us even farther away from direct interaction with code.

A typical elementary school student request in 2016 is to how to make mods with Minecraft.  This is hugely exciting, in that kids are now re-engaged with the concept of coding.  But the ecosystem to support them quite frankly sucks.

In 2016, the main options are:

• using one of the grab-an-instruction-block toy coding abstraction environments, e.g. Scratch or Apple's Swift Playgrounds
• getting a task-specific learning to code book, e.g. Minecraft Modding with Forge or Adventures in Minecraft
• trying to find a simple (shall we say, basic?) coding language that actually runs on a modern computer
• Raspberry Pi type things

So here are some issues.

Blockland

I just don't buy the idea that dragging around code blocks is a good way to enter into programming or to understand deeply how computers work.  No one actually programs by dragging around code blocks, we all type in commands.  Plus which the nature of the code block oriented training tends to be highly structured, step-by-step rather than discovery oriented.  Quite frankly, it doesn't look like much fun to me.

Plus which, you're working too many layers away from the actual code.  When I played games and then exited out, I could see the actual BASIC code.  If you're doing Hour of Code Minecraft, you see this.

This is a lie on multiple levels.  First of all, this isn't actually Minecraft, it's a kind of Minescratch, some layers of JavaScript with maybe some kind of server backend.  The blocks you are moving around on the screen are not the code.  They're not even close to the code.  You can go through the entire coding exercise without ever seeing the actual code that is running the coding environment.  For anything other than some maybe-benefits of getting some CS concepts, this is useless at teaching about coding.

Bookland

Moving to "intro to coding" books on the other hand takes you from one extreme (blockland) to another (bookland).  There are at least two problems with bookland.

First, since we no longer have a simple baseline computing environment and language, every book starts with "downloading and installing".  Page after page of downloading and installing.  You must get this specific version of this thing, and that specific version of that, and then do this step and this step and on and on.  And oh by the way if you're reading the book more than a few months after it was published, all of those versions are obsolete and none of the code works on the new versions.  It takes the entire first chapter of Minecraft Modding to get to the part where you can in theory start coding.  It takes 30 pages of Adventures in Minecraft before you can write code.

So we've frontloaded one of the most tedious parts of computing, setting up a working coding environment, into the beginning of every code adventure.  What kid is going to wait the hours it takes to set up all of that?

Secondly, because there's no baseline environment, there isn't even consensus on where you start, there's no shared common experience.  Minecraft Modding wants you to write Java in Eclipse for Forge.  Adventures in Minecraft wants you to write Python in IDLE for Bukkit + RaspberryJuice.  And these are the easy, entry-level books for kids.

Codeland

I just want to have a command-line prompt where the kid can type in some text and make something happen.  But since the big tech providers killed all their entry-level coding options, it's a muddle of half-abandoned side projects that may or may not install on your current computer and may or may not have any kind of support.

Yes, Apple has Xcode, but learning to code using Xcode is like learning about space physics by playing Kerbal Space Program.  Yes, you can do it, but you have to learn layer upon layer upon layer of things just to even get the rocket on the launch pad.

BASIC

In response to David Brin's article, in 2008 Small Basic was released.  For Windows only of course.  But nevertheless with a fairly extensive support presence.  And it was last updated in 2015 and there are still blog posts, so it's not dead yet.  And it has a Turtle command so you can do Logo-like things with it.  One can of course debate whether TextWindow.WriteLine(“Hello”) is actually BASIC at all, but nevertheless.

On the Mac side, it looks like maybe Chipmunk Basic.

PureBasic (which I have never used) also looks like it might be an option.  Lots of platforms supported.

Back in the day I used to use True BASIC a lot (on the Amiga).  It is now Windows only.

Logo

I like the idea of the immediacy of Logo. 

For Logo, the situation is pretty muddy, with lots of different implementations and not many updates.  UCB, MSW, FMS, aUCB, Terrapin, ACS...

On the Mac side, ACSlogo hasn't been updated since 2011.  Other than that it looks like basically Terrapin Logo.

There are more options on the Windows side.

Pi Land

Raspberry Pi, or custom environments like Kano, or wall-climbing robots... maybe?

But purchasing them and setting them up is a barrier.  It's a lot easier when the computer you program on is your actual main computer.

What Could Be Done - Emulation and Built-in Languages

1. Provide legal, out-of-the-box emulated environments of classic computers on Windows and Mac

There is no reason that every Windows and Mac computer, and for that matter every smartphone and tablet, couldn't come with full emulations of the major 80s computers.

There is VICE but I don't know its legal status or ease-of-use.

Maybe this is super-easy to do through the Internet Archive's Software Collection.  It's not clear to me.

2. Provide entry-level computer languages on all Windows and Mac computers

BASIC and Logo would be a start.  Python maybe.

What Could Be Done - Minecraft

1. Make the Minecraft command line more powerful

An easy entry point for learning to program in Minecraft would be through the Minecraft command line.  If it had simple programming capabilities, it could become the new BASIC prompt.

2. Fix the Minecraft mod ecosystem mess

Right now trying to do anything with Minecraft mods is a mess. The mods are Minecraft-version dependent.  This means you have to have the right combination of the Minecraft version, the Forge version and the mod version.  There is no easy way to switch between different profiles when it comes to mods; it won't remember which mods go with which version.  Plus which, instead of being in a central place with a central community, the mods are all scattered all over the place, with downloads from dodgy popup-ad websites.  And the mods are not maintained, so the compatible version is never the one you would want it to be.

3. Have many easy pre-packaged Minecraft mod coding entry points

Mods in BASIC.  Mods in Python.  Mods in Java.  Just make it easy to set the environment up in one package, without having to firewall off every special combination of versions and support tools.  Standards build ecosystems.  Right now there is only complicated fragmentation.

Solving this could be a huge, easy win for Microsoft, generating both goodwill and a pipeline of future programmers; I don't understand why they don't fix this.

Readings

• Why Johnny can’t code by David Brin in Salon - September 14, 2006
• Forward 40: What Became of the LOGO Programming Language? by Aaron Rowe in Wired - October 15, 2007
• How are students learning programming in a post-Basic world? by Lamont Wood in ComputerWorld - June 23, 2011
• Midnight programming, 1979 vs. 2011 by Lamont Wood in ComputerWorld - June 23, 2011
• Fifty Years of BASIC, the Programming Language That Made Computers Personal in Time - April 29, 2014
• Can a Wall-Climbing Robot Teach Your Kid to Code? by Chris Berdik in Slate - June 22, 2016

Previously:
December 11, 2014  Hour of Code 2014

NRC jobs - dotNET - Business & Web Applications Developer

National Research Council Canada

Ottawa – Ontario
CS-2, English

This is a 1-year term position from the date of reporting.

Apply at http://www.nrc-cnrc.gc.ca/eng/careers/competitions/16_14_0796.html

Technical competencies

• Advanced knowledge of .NET technologies such as ASP.NET, C# and .NET API;
• Knowledge of Windows PowerShell scripting;
• Solid knowledge of JavaScript;
• Solid knowledge of HTML coding standards and CSS;
• Knowledge of software engineering concepts, methodology and practice, including Object Oriented analysis and design;
• Knowledge of Microsoft SQL server relational database management system and the SQL query language;
• Ability to transform user requirements into technical concepts;
• Ability to write technical documentation in English in a fluent and consistent manner;
• Working knowledge of SharePoint 2010/2013 server side API is a definite asset.

Closing date

August 19, 2015 at 23:59 Eastern Standard Time.

hacking and learning about hacking - in Ottawa - May and June 2015

Open events

• #FlashHacks at Maker Space North - May 24, 2015
  "Come and join us for a day of mapping the corporate network of the largest public companies in Canada and releasing this as open data."
• Bring your own data and learn about hackfests - register - June 1, 2015
  part of #CongreSSH
• Open Data Book Club (dataset club) - register (all welcome) - June 3, 2015
  by @opendataottawa
• Random Hacks of Kindness Ottawa - register ($5) - June 5-7, 2015
  by @rhok_ottawa

Closed / Teams Required

• #HackingConflict #Diplohack - see http://new.secdev-foundation.org/hackingconflict - http://hackingconflict.org/ - @diplohack - May 27-28, 2015
• #AquaHacking - http://aquahacking.com/en/ - @aquahacking - presentation to jury May 30, 2015

NRC jobs - Database, Repository and Application Specialist

Knowledge Management
Ottawa - Ontario
CS-2, English

This is a 3-year term position from the date of reporting.

Closing Date: January 5, 2015

http://www.nrc-cnrc.gc.ca/eng/careers/competitions/16_14_0340.html

He/she is involved in the installation, design, development, testing, support, and maintenance of digital repositories and business applications required by Knowledge Management to deliver its services. He/she also provides basic database administration services for related database management systems.

The incumbent of this position will be part of a small team where communication and teamwork are a must and interactions with clients and partners very frequent. We expect members of the team to be very resourceful, dynamic and self-sufficient, and to become experts in a wide array of technologies.

Why Hackathon?

CPS Renewal asserts: "Hackathons produce apps, not solutions to wicked problems"

I think this is a mis-framing. There are two separate issues:

1. what are hackathons for and how to make them better
2. how do we create events (and in general, processes) that help solve wicked problems

A hackathon is supposed to be a simple thing

an event in which computer programmers and others involved in software development, ... collaborate intensively on software projects

Wikipedia - Hackathon

The basic idea is you create time for people who are experts in coding to work on coding problems.  What has happened is this idea has collided with the open data and open government movement, and been loaded with expectations about public outreach and systematic transformation.  Now (to exaggerate somewhat) hackathons are expected to:

• produce commercial-quality smartphone apps in 48 hours
• stimulate billions of dollars in economic activity
• convince millions of schoolchildren to choose computer programming as a career

This is a lot to ask for out of an event structure that was originally a dozen people with laptops trying to fix a few bugs or to make a prototype of a web service that brokenly half-does something maybe interesting.

What are Hackathons For?

For a general public hackathon, they are mostly for fun, networking and awareness.  They're not actually about coding.  They're definitely not about making smartphone apps.

To pick Ottawa Aquahacking as an example, some reasonable goals are:

• to raise awareness of the Ottawa River ecosystem, water quality and related issues
• to connect e.g. the river science experts, river data experts, and coders, to make a few small things that may illuminate a few aspects of the river
• to expose people to a different way of working together
• to have fun

This is not the place to:

• get free resources for the complicated app or website you've always wanted to build
• solve river ecosystem issues

Basically, we have some new tools: data, code, the web.  This is a way to bring those new tools to an existing problem and see if anything interesting emerges.  For people who aren't familiar with the possibilities of these tools, the experience may change their perceptions of how we can use technology.  For people who aren't used to open collaborative work, the experience may help them to reimagine some of their problem-solving.

Apps Contests

Apps contests are a good way to reward people for coding.

They're particularly good at rewarding coding if they connect to mentoring, internships, and investment opportunities.

Apps contests are not a good way to get apps.  If you want apps, you should procure apps.  If you use an apps contest you will get a one-off app that won't be updated and will probably eventually disappear.

Apps contest are an ok way to show the possibilities of bringing new technology to bear on a particular problem space.  But they should be considered illustrative examples, not solutions.

Hour of Code

D5 Charter (PDF)

The Participants have decided to commit to working towards the following principles of digital
development, acknowledging that they will not be able to meet all of the criteria on joining:

3.7. Teach children to code - commitment to offer children the opportunity to learn to code and build the next generation of skills

Everyone wants children to code.  People want some hackathon / apps contest / Hour of Code / Elsa-skating website to make this happen.

Motivating people to code is kind of an odd thing.  In a list of just the 15 top tech billionaires, the poorest has eight billion dollars.  This would seem fairly motivational. 

All of which to say, there are many pieces that determine people's choices of career.  Events that give a tiny sense of what coding is can maybe help a tiny bit, if they're well-structured enough that they aren't intimidating, exclusionary, and confusing.

But if you actually want people to code you need to build it into the curriculum, as both a separate track and an integrated tool.  That is to say, you need to have both computer science courses and actually use the computer science skills in the other classes.  And reward and recognize all along the way, not just wait for the students to emerge out of highschool with a few coding classes and expect them to want to continue clawing their way up to work at Google or become startup billionaires.

And it's way more than "coding" (which means everything and nothing, from writing a line of HTML to making some JavaScript to building an entire iPhone app).  It's computational thinking, logic, problem-solving using technology.  While it is WAY better to have kids come out of school with JavaScript skills than "teaching them computers" by training them in Microsoft Office, it's guaranteed that whatever language and approach the schools will codify will be obsolete by the time the students emerge.  So it has to be both fundamentals and practical application.

It has to be education that prepares people to solve wicked problems.

Wicked Problems

Hackathons are not supposed to solve wicked problems.  They're supposed to do very small things, with technology.

The good news is we have all kinds of other emerging approaches, some of them perhaps inspired by the hackathon ethos.

So in this space, we have things like:

• @PolicyLabUK
• @18F
• Presidential Innovation Fellows
• US Digital Service (see e.g. White House announcement, Playbook)
• UK Government Digital Service

These are organisations made up of people who use innovative approaches to solve problems.  Sometimes these approaches will include bringing new technologies to bear, sometimes not.

People solve problems, not events or technology.

Eventually, these organisations should demonstrate enough successful approaches that the overall system can be improved, with a combination of better education in school and better training on the job. 

We don't want a world where all forms are fixed by a special Form Innovation Service.  We want a world where everytime a form is created, the employee knows how to design it properly.

Wicked Problem Events

That being said, the way you construct an event is to first choose the right people, and then choose a variety of approaches.  You might unconference, you might do ideas generation, you might do sticky notes and then all leave for several days (as suggested in Hearing Every Voice in the Room).  We have lots of approaches that we can use now.

One of the underlying challenges is that hard problems are hard.

I see lots of events about e.g. IT infrastructure for science where you hear the same big problems over, and over, and over again.

Wicked Problem Spacetime

Ultimately, what you need is not one-off events, but trust, networks of collaborators and time.

The fact is that even to build a simple thing, you need lots of people working together.  You can see this from the building of ORCid.  Assigning a unique number to every researcher is not a conceptually or technically complicated thing.  But making it happen took deep thinking about trust, governance and sustainability. 

As another example, getting people in general and scientists specifically to use version control isn't a conceptually or technically difficult problem.  But almost no one uses version control, even though it is foundational for many other layers of advanced things we can imagine.  (And there is a related issue of whether people should need to "use" version control at all, or it should just happen automatically for them in their tools.)

We need to get better on the strategic side, to make sure we're solving the right problems, but the notion that we're particularly focused or "good" at doing tactical technical pieces I think is off-track.  We're actually terrible at tactical technology. 

So in response to

I think the focus on apps (to the detriment of research) is a by-product of our shortened shadow of the future, our culture of immediacy and the omnipresence of attention economy...

from An Alternative to the Hackathon

I would say yes, a focus on apps is misguided.  But don't confuse delivering apps with delivering tactical technology.  We could benefit from being a lot better at quickly delivering usable technology, in support of an overall strategy that addresses wicked problems.  We need both a better short game and a better game plan.

Hour of Code 2014

The Hour of Code challenge was launched by Code.org during Computer Science Education Week 2013.

Computer Science Education Week 2014 runs from December 8 to 14.

President Barack Obama took part at an event in the White House.

Prime Minister David Cameron took part in an event at 10 Downing Street, with an accompanying announcement about STEM education.

Twitter:

• @codeorg
• #hourofcode
• @csedweek
• #csedweek

2nd Canadian National Hackathon - CODE 2015

CODE 2015

February 20-22, 2015

registration: https://canadianopendataexperience.ca/register/?lang=en

main site: https://www.canadianopendataexperience.ca/?lang=en (scroll down on the site to see all info)

gov site: http://open.canada.ca/en/canadian-open-data-experience-code

PR: http://news.gc.ca/web/article-en.do?nid=907729

A 48-hour hackathon using Canadian federal open data from http://open.canada.ca/en/open-data

Hashtag #CODE2015

Previously:
October 13, 2013  Canadian national app development competition - CODE 2014

csvconf 2014 - Data Hacking with Wikimedia Projects - notes

csv,conf 2014 - July 15, 2014

12:20 Data-Hacking with Wikimedia Projects: Learn by Example, Including Wikipedia, WikiData and Beyond by Matt Senate @wrought and Max Klein @notconfusing

Matt and Max have clearly done a lot of thinking about how to make a project work within the WikiMedia context.  The project they described is about automatically extracting information about open access content and then enriching multiple WikiMedia sites with the appropriate information.

Video available at https://www.youtube.com/watch?v=w5T6RoEckAQ

• Wikipedia is by far the largest of the WikiMedia projects
• In order to contribute successfully, you need to understand the community for each project, the cultural protocols.
• Rambot added US Census data to WikiMedia
• When contributing from an open access article to WikiMedia
• text to WikiSource
• media (images etc.) to WikiMedia Commons
• metadata to Wikidata
• signals (information about the article such as license) used to enrich e.g. citations in Wikipedia
• the bot is called RecitationBot
• goals:
• improve citation management
• integrate metadata into wikidata
• forge deep knowledge resources
• automation combined with human judgement
• Paths for Data Hackers (things to work on in order to succeed, particularly in WikiMedia context)
• reputation
• understanding of cultural protocols
• knowing the history and context of previous efforts, and fitting your work into that narrative
• connecting to community e.g. IRC, mailing lists
• RecitationBot is essentially a process of enriching newly posted citations
• can also initiate RecitationBot with a DOI as you create a reference

csvconf 2014 - CSV Validation at UK Archives - notes

csv,conf 2014 - July 15, 2014

11:55 session CSV Validation by Adam Retter @adamretter

Video available at https://www.youtube.com/watch?v=9pIBG5iv5XI

• has been working at UK National Archives
• there is a new digital archive for the UK - the Digital Records Infrastructure (DRI)
• need to ingest metadata from many sources
• CSV was decided upon as file format for metadata
• many possible errors in the CSV provided (including some people just rename Excel files, rather than exporting as CSV)
• approach: fail fast as repeat
• was decided to create a CSV Validator
• providing code to suppliers - now the suppliers can do up-front data validation
• want to be able to write rules, beyond just basic checks, and have it make easy enough to do so that domain experts can write rules
• CSV Schema language as a way to write rules
• https://github.com/digital-preservation/
• looking for developers to help advance this work

See also UK National Archives blog: CSV validator – a new digital preservation tool.

csvconf 2014 - Data usability: the tabular data package - notes

csv,conf 2014 - July 15, 2014

10:35 session Data usability: the tabular data package and friends by Nick Stenning @nickstenning

Data publishers at #csvconf: my message to you today is “put it in a box.” Publish your data as data packages: http://dataprotocols.org/

— Nick Stenning (@nickstenning) July 15, 2014

Slides at https://github.com/nickstenning/put-it-in-a-box

• analogy with international shipping: things in containers vs. people carrying them individually
• things not in containers are very expensive to move
• which means you do a lot of work carrying the things, rather than using the things
• containers are: 1. simple, 2. opaque, 3. diverse within constraints ("moderated diversity")

Introducing The Data Package, datapackage.json

See info at http://dataprotocols.org/data-packages/#specification

My question: What about BagIt?

Answer from Rufus Pollock (@rufuspollock): Other formats were examined, there is a white paper.

UPDATE 2014-08-01: Review of Existing Packaging Work via @rufuspollock

csvconf 2014 - introduction - notes

csv,conf 2014 introduction by Max Ogden and Rufus Pollock - @maxogden and @rufuspollock
July 15, 2014

(csv is Comma-Separated Values, a way of representing data.)

• wanted to reach out to data hacker community
• wanted to bring together open data / open tools / open standards communities
• csv has strength of simplicity
• csv inherently "hackable"
• csv can be used for big or small data, it "scales", it "streams"

Hashtag #csvconf

Government Open Source: Web Experience Toolkit

The Web Experience Toolkit has been in the news lately, with mentions in Mashable and Fast Company due to appearing on GitHub and Government.

It also got positive mentions from David Eaves and Ben Balter at the Code for America Summit.

Below is a Storify of recent mentions:

Canadian national app development competition - CODE 2014

The Canadian Open Data Experience – CODE – is a 48 hour nationwide app development competition

The event is sort of softlaunched, in that the date isn't posted yet. Minister Clement announced it, and later said

The other thing that I announced today, by the way, is our first appathon, where we're going to invite entrepreneurs of all ages next February [2014] to go onto the data.gc.ca website/portal and in a 48-hour period come up with the next app that uses Canada's open government data and develop something that will be of use to citizens

Possibly this will be timed to coincide with International Open Data Day 2014 on February 22, I don't know.

UPDATE 2014-01-09: CODE event will be February 28th to March 2nd  ENDUPDATE

There is a basic signup site at http://www.canadianopendataexperience.com/ and social media

• Twitter @CODE_hackathon and currently the rather overloaded hashtag #CODE
• Facebook https://www.facebook.com/CanadianOpenDataExperience

The government is supporting it, but not running it.  It will be run by XMG, who do the similarly-named but separate Great Canadian Appathon (it's not clear if the events will be one and the same in 2014).

Web Experience Toolkit - CodeFest 2013

A modern technology for web components, the Web Experience Toolkit, had its second CodeFest.

The open source code and community are hosted on GitHub.

https://github.com/wet-boew/

Government of Canada employees have played a substantial leadership role in the creation of the toolkit, in its release on GitHub, and in growing the community.  It's a great example of government working in a modern way.

I've embedded links to the keynotes below, as well as a Storify of the event (I didn't attend, but I watched the keynotes on YouTube).

Event info at http://wet-boew.github.io/codefest2.0/

Keynote presentations

Ben Balter @BenBalter - http://ben.balter.com/open-sourcing-government/
https://github.com/benbalter/open-sourcing-government (repository for slides) - video from 14m43s http://youtu.be/E-AxzfaZQVc?t=14m43s to 1h13m

Raymond Camden @cfjedimaster -
https://github.com/cfjedimaster/CuttingEdgeHTML (repository for the slides) - video from 1h35m32s http://youtu.be/E-AxzfaZQVc?t=1h35m32s

Hillary Little @hilittle - http://www.hilarylittle.com/blog/2013/08/09/codefest-2013-keynote-2/ - video http://youtu.be/51hN1trnpXA

Previously:
January 11, 2013  open source in government - The Web Experience Toolkit

Related articles

The Software That Builds Software

two hackfests in Ottawa: Citizen Attache and CanLII

Tapping into Canada’s Foreign Aid Data

August 24th-25th, 2013

HUB Ottawa, 71 Bank Street

"The hackathon will bring together technologists, data analysts, and international development experts from across the sector to create useful products, insights and analysis of Canada’s international aid data."

You have to apply to attend, deadline for applications is August 8, 2013.

They are also soliciting project ideas.

Site: http://citizen-attache.github.io/

Twitter: @citizenattache

CanLII law, government and open data conference and hackathon

September 13th-14th, 2013

University of Ottawa

"The first day of the event will be a series of speaker panels about different data sources and applications and the second day will be a hackathon with various datasets and the CanLII API."

Site: CanLII hackathon (hackerleague.org)

Twitter: @CanLII

---

You can see previous postings in category open data

Related articles

G8 Open Data Charter, updated Canadian national open data portal

CanLII

open service development and the Expenditure Database

On April 22, 2013 TBS released the Expenditure Database.  It allows one to browse and search expenditures organised in ways that are easier to understand than the "votes" system under which they are actually allocated.  It is great progress within the overall open government initiative.

However, from a technology, design and process perspective, I think there are some opportunities that are being missed.

Victorian Service Delivery

Minister Clement is fond of saying that the government's data is like grandmother's silver, hidden away.  In fact, our current model of delivering applications is more like hiding the entire kitchen from view.  In Ottawa's Victorian homes, the kitchen is usually at the back of the house, closed off from the dining room.  The servants were supposed to work there in obscurity, with only the final result appearing with a dramatic flourish in the formal dining room. 

In most of those homes that wall has now been smashed down, because we found that in the post-servant age, the kitchen is the hub of activity in the house, where we socialise and cook together openly, a very human and social activity.

But in government software development, that wall still stands.  Applications are developed behind closed doors by public servants, and then suddenly appear, fully formed, on release day. This model of closed software development has some very real consequences:

• There is no opportunity for issues to be caught and corrected during the development process; instead we have to wait post-launch to find design issues that are only visible after public use
• It makes application development seem mysterious and closed off from the world.  In a country where we want young people--in particular young women--to see software development as an interesting career, we've hidden all the real human aspects of development away, all the interactions, discussions, decisions, and compromises that have to be made as we assemble the software from the available ingredients in our coding kitchen.
• It closes the government developer community off from the much larger national and worldwide community of developers, both reducing the network that they can tap into as well as hiding our own government excellence and expertise.

We need not just open data, we need the entire philosophy of open source web development: we need to develop in the open.

Modern Government Service Design

The UK Government has released a fantastic Government Service Design Manual.  It has some explicit statements and some built-in assumptions.  The underlying concepts are to develop government services using modern software engineering processes.  This means being open about the code as it is in development (e.g. through blogs explaining the work and github repositories making the code available) and iterating through the design, from Discovery, through Alpha, Beta, Live and (all important in the government) Retirement.

The UK Government Digital Service does its work in the open, through blogs, twitter, and github.

Software development is a process: the application you release at a moment in time is not an end, what's important is that the application should tell a story about itself, so that it can be improved and so that it can be an inspiration for further work.

The Expenditure Database

The Expenditure Database is a great step in providing easy citizen access to the underlying data.  But it doesn't tell any of the story of how it works, how the data is processed, and who did the work.  The meal has been delivered and it looks very nice, but the kitchen and the process of making it are still closed off from view, hidden in mystery.

Code on the Web

Of course this is not quite as simple a story because it is a web application, so in theory you could look through all of the JavaScript.  But that means reverse-engineering the code, trying to figure out how it works without having access to documentation (the recipe) and the developers (the cooks).  When someone is trying to understand an application the first thing they will do is read its story about how it works and how it was put together, and then ask the developers when there are aspects not clear or not covered in the documentation.  Without open code development, this avenue of understanding is closed.

Instead one has to dig through the code and make guesses.  For example, it looks upon examination that it is drawing the data from a local file: http://www.tbs-sct.gc.ca/ems-sgd/edb-bdd/data.js

If instead the code were in a shared repository and documented, it would be much easier to understand how it works and the rights to reuse and modify would be a lot clearer.  As well one would expect there would be both examples of using the data file as well as an explanation of how it was generated.

Expenditure Database Development

What Expenditure Database development would look like if it followed this model:

• In advance of coding, in the Discovery phase there would be an internal call for requirements and expertise, e.g. on GCpedia (the Government of Canada internal wiki)
• An alpha using open source code would be quickly released
• The alpha would use public budget open data from data.gc.ca
• If the data wasn't available, it would have to be released (the rule is, internal developers shouldn't have any special or privileged access to information)
• A beta would be released to the public incorporating feedback and code changes
• At the beta phase at the latest, code would be shared on github (a public code repository and developer community) and there would be public explanations and documentation of the work going on.
• The developers would be using their own accounts; it would be clear whom to contact for questions about the software.
• Once the system goes live, there would be an ongoing process to incorporate improvements to the code, improve the documentation, and respond to internal and external questions from other developers and users.

What We're Missing

Just a few examples of what we're missing as a consequence of the current implementation:

• There are no permanent URLs to departments or searches.  That means if I want to point you at the expenditures for the National Research Council, I can't just give you a URL.  I have to say "go to http://www.tbs-sct.gc.ca/ems-sgd/edb-bdd/index-eng.html and search or scroll to National Research Council and then click on it and then scroll down and click on the details for various parts of it".  A good standards-based design and testing process should have revealed this issue.
• The Expenditure Database "stands alone" on a difficult-to-remember GC standard URL.  You have to know to navigate to it, or get the link from somewhere.  Why not tbs-sct.gc.ca/expenditures as a short link?  There is also no link from the database to data.gc.ca nor is there a link from data.gc.ca to the database.
• Without documentation about how the data were created, it's not clear if it's using budget data from data.gc.ca or whether it is using a custom database, and how the numbers are being generated; without access to the code there is a lost opportunity for other developers to learn how to parse, manage and display the budget data.
• Without the code, it's not possible for developers to either make their own websites using a similar approach, or to improve the Expenditure Database code itself.
• Without pulling the data directly from data.gc.ca there is no opportunity to explore all of the challenges that external developers need to solve, such as how to deal with versioning (what if a number has to be changed in the released data? how to tell if it has been changed?  how to see the previous version?) and performance (should there be an API instead of a download?  should the download be cached locally?  for how long?) and updates (how to tell when a new budget year has been released?)
• Without the code shared and the work publically described, people are left to speculate as to how the results are generated.
• Without the code shared and the work publically described, there is no entry point for external developers; there is no community-building for external developers.
• There's no contact point. The site gives no idea who developed the Expenditure Database, how it works, or who made the design decisions. The code and its development is made mysterious, the work of the kitchen hidden, rather than being demystified.
• There's no way to download the data displayed; you can only take a screenshot.
• There is no standard web navigation (e.g. the back button doesn't work)
• There's no way to compare departments

Additional information

TBS PR - Searchable Information on Government Spending is now Just a Click Away

Call for Feedback

I welcome your feedback on this post, with any clarifications, corrections, suggestions or pointers.

Thanks to Stephen Anthony, Adam Frankowski, Patrice Collin and Nick Charney for providing ideas and feedback that improved this post.

open source in government - The Web Experience Toolkit

The Government of Canada has its Web Experience Toolkit on GitHub

https://github.com/wet-boew

By making the WxT open source, it has been possible for organisations outside of the government to use it for their websites and to improve it.  Both the University of Ottawa and the City of Ottawa are now using WxT.

There was a writeup in Wired about the WxT: Canadian [Coders] Solve Mystery of Open Source Government.

the Treasury Board of Canada hosted a CodeFest to invite hackers — mostly government staffers — to hack its Web Experience Toolkit, or WET — a set of open-source tools that the Treasury Board uses for building websites. One hundred and fifty people came. Many of them were young developers, excitedly swapping code and sharing ideas across tables.

There are some minor issues: It's Treasury Board Secretariat, not Treasury Board, and TBS is the government's central policy agency not an "obscure Canadian tax-collecting agency".  (Admittedly it is difficult to decode that from the TBS website, but if you read far down enough you get to "policies, directives, regulations, and program expenditure proposals with respect to the management of the government's resources".)

You can follow Web Experience Toolkit info on Twitter using the hashtag #wxt and the account @WebExpToolkit

Author of the article is Robert McMillan @bobmcmillan

Featured or indirectly mentioned are:

• Paul Jackson @pjackson28
• Mary Beth Baker @bethmaru
• Lucia Harper @Lucialand
• Laura Booker (of OpenPlus) @bookerlaura
• @OpenPlusSoft
• Mike Gifford (of OpenConcept) @mgifford
• Luke Charde @lukecharde

UPDATE: There is a Drupal working group, the next meeting is January 25, 2013 at Ottawa City Hall.

Hackfests and Apps Contests

Here's a thing I wrote on the government internal wiki (which is only available to government employees).  Some of the content was based on an email I received.

If you have ideas about how to make it better, please feel free to leave a comment (or edit it directly on the wiki, if you're in the Government of Canada).

 

Hackfests and Apps Contests

There are a number of ways to encourage the use of open data and the building of a community. This article uses the broad umbrella of "hackfest" (hackathon, codefest, etc.) to cover some approaches.

Common approaches include:

• an online community discussion channel (e.g. mailing list, discussion board)
• ideas-generating events or contests
• code development events or contests

Community-Building

Sites that offer data, APIs, or other technical capabilities often need support from a broader community in order to sustain and grow their capabilities. This is a key element of community engagement.

Communities may be sector-specific, or they may be based on particular skills or expertise.

Key communities may include:

• software developers
• user interface design experts
• technology standards experts
• domain experts (e.g. health experts, energy experts)
• intended users of a particular technology (e.g. parents for a consumer recall application)

Developer Outreach

In cases where a desired outcome is the development of new applications or hardware in order to promote economic growth, software development experts are a community that needs specific targeted outreach.

In the United Kingdom, the national Open Data White Paper identifies the need for a Developer Outreach Strategy

We need to work collaboratively to ensure that developers are aware of what datasets are being released, in what timeframes, and to maintain relationships with those at the cutting edge of technology who can help government do things differently and in more agile ways. This kind of conversation between government and users facilitates capacity building both ways to great benefit for the public good.

One approach to this for government websites is to ensure that every site has a /developer subpage that provides contact and community information. Such a page needs to be backed up with active community engagement.

In the US, the national Digital Government Strategy states

To establish a “new default,” the policy will require that newly developed IT systems are architected for openness and expose high-value data and content as web APIs at a discrete and digestible level of granularity with metadata tags. Under a presumption of openness, agencies must evaluate the information contained within these systems for release to other agencies and the public, publish it in a timely manner, make it easily accessible for external use as applicable, and post it at agency.gov/developer in a machine-readable format.

Ideas Generation

Many events have idea generation as a first stage of the event (typically online-only to get people started thinking). Others are ideas-only (no code created), or run idea generation in parallel with the coding.

Examples:

• ideas as first stage: Nokia Creating Innovation for People and Planet challenge closes July 31, 2012 followed by an apps contest (the "Do Good Hackathon") http://dogoodhackathon.com/ - if you don't "claim" your idea by registering for the hackathon, anyone can use it
• ideas-only: the GC IT Innovation Challenge ran in 2009
• ideas in parallel: the City of Ottawa offered separate prizes for ideas and for code in its 2010-2011 Apps4Ottawa contest

Apps Contests

There has been an upsurge in organisations offering prizes for creative use of their technology assets, most typically an "apps contest" using open data that they have released. This has ranged from the municipal to the international level.

It is an easy entry point into leveraging the value of released data, but it should be considered only one piece of an overall engagement strategy.

A caution on expectations from cash prizes - they will incentivize creation of individual apps, but they are not sufficient to help build community. Four Ways Summer of Smart has Reinvented Civic Hackathons states

Don’t (just) offer cash prizes. (Hackathons should build a community.) We have seen dozens of apps contents and hackathons where it’s assumed that a few thousand dollars of incentive as a prize will help seed the projects and produce innovation. But as many have found, this model simply isn’t effective – and there is a clear reason why: civic hackers are driven not by money, but by the potential of their work to create civic and social change.

Canadian Apps Contests

Completed or Currently Running events

• Apps4Ottawa - the City of Ottawa in 2010-2011 - with organising support and hackfests from citizen's group Open Data Ottawa
• Apps4Edmonton - August 2010
• Apps4ClimateAction BC - August 2010
• Apps4Good - May 2010
• London Open Data Budget App
• Canpages API
• e-Health 2012 Apps Challenge
• Royal Canadian Mint (RCM) MintChip Challenge - Deadline for submissions is August 1, 2012
• «Let’s Develop Together!» [Société de transport de Laval (STL)
• AppStar (UTSC Mobile App Development Competition)

Upcoming

The City of Ottawa will have a second contest in fall 2012. The lead at the City of Ottawa is Rob Giggey (Twitter: @rob_giggey).

US Government

The US government offers a general platform for citizen competitions of many different types called Challenge.gov The tagline is: "On Challenge.gov, the public and government can solve problems together."

Hackfests (General Information)

Developers will sometime create simply based around a data release or a hackfest event, without needing to have a specific contest. For example a wide range of applications for bus arrival times were created following the City of Ottawa's release of real-time transit open data - there was no contest, but the data release was widely promoted and there was a related 2012 Ottawa Transit Data Day event.

Canadian Hackfests

• Hack Regina Fall 2012
• Startup Edmonton Summer Hackathon 2012 - #hackathonyeg
• Hackons la corruption 2012 organised by Quebec Ouverte
• Gatineau Hackathon 2.0 - May 5, 2012
• 3rd Open Data Ottawa Hackfest - December 3, 2011 - @opendataottawa - #hackott
• Random Hacks of Kindness Montreal - December 2011 - @rhokmtl
• Random Hacks of Kindness Toronto - December 2011 - @rhokto

GC Hackfests and related events

• WET Codefest - July 2012 - event to work on government web standards
• Open Policy@DFAIT - February 2012 - although not a hackfest per se, it included an unconference where various aspects of application development and coding were discussed

National-level Hackfests

• Dutch Parliament Hosts Apps4Democracy Event - Sept. 2012 - "The Dutch Parliament is planning a major release of parliamentary data as open data, and the hack day is aimed at letting coders explore and work with the data."
• UK - 4th National Hack the Government Day - April 2012 - #nhtg12 - also see blog post National Hack The Government Day 2012 - hosted at http://digital.cabinetoffice.gov.uk/

Ottawa & Gatineau Groups

There are many Ottawa-area groups with expertise related to community-building and software development.

These include:

• Open Data Ottawa
• Gatineau Ouverte
• Girl Develop It Ottawa
• Geek Girl Dinners Ottawa
• Hacks & Hackers Ottawa

Ottawa Transit Data Day - June 2, 2012

Having liberated Ottawa's real-time bus location data, the city is now moving to the next step of bringing developers and OC Transpo together to make (even more) amazing things.

Ottawa Transit Data Day is June 2, 2012 at Ottawa City Hall

Register at http://transit-data-yow.eventbrite.com/

Find more information about the available data in the Open Data Ottawa blog

Follow @opendataottawa

Suggest datasets at DataOtt

Ask questions in the GPS API mailing list

Provide feedback directly about OC Transpo open data: [email protected]

Event hashtag will probably be #yowdata

So many communications channels.  Whew.