Note: This is a technical examination of available COVID-19 open data. For general information please visit your relevant local website, including:
Sometimes when advocating for a particular change, it can be hard to step back and recognize the progress that has been made.
COVID-19 open data in Ontario is available and is provided in a way that meets many of the goals that open data advocates had set out. The City of Ottawa and the Province of Ontario both prominently link directly to COVID-19 open data on their dashboard pages. The Government of Canada also links to open data, but not as prominently or consistently. There is also an academic site providing data and code for Ottawa. All sites provided clear licensing.
These data visualisations and open data can be considered a kind of infrastructure. In the context of academic use of open data, particularly with the addition of open source computer code, these sites can even be considered part of a digital research infrastructure.
There were several elements that were envisioned for open data in general:
Basically, that anyone could find it and use it. That it would provide a usable machine-readable alternative to scraping HTML web tables or hand-extracting data from PDF or Word documents. And also that it would be provided from every organisation at every level of government.
For sustainability, it was also hoped that open data would be part of each organisation's own internal processes:
There is an additional key element I have to say wasn't strongly considered by many of the initial advocates, which is that there should be tools provided to view and work with the data without having to go in and do your own spreadsheets or write your own code:
And I would add lastly, as the community thought through the provision of tools (e.g. website visualisations):
These goals sometimes seemed very far off. But with COVID-19 data in Ontario, I would consider it a success.
COVID-19 open data is available at the municipal level (e.g. City of Ottawa), at the provincial level, and federally. (And beyond, through international aggregation sites.)
Ottawa's open data site is https://open.ottawa.ca/
COVID-19 open data can be found just using "COVID-19" as a search. In this case I've also filtered for just CSV format files.
https://open.ottawa.ca/search?q=covid-19&type=csv
There are 8 datasets, including:
The only one that doesn't seem to be updated frequently is "Data tables for ONS Neighbourhood COVID-19 Maps".
So the good news is the data exist, the data are being kept up-to-date, and the data are in a usable, machine-readable format.
So let's look at three more aspects:
The main general use site is Ottawa Public Health - Statistics on Coronavirus 19 (COVID-19) in Ottawa, it has links to:
The dashboard is in Microsoft Power BI, which I have seen used on a number of COVID-19 information sites.
The data pipeline is clear, but I don't think it is one that is available to the public: "Data are extracted from [the provincial Case and Contact Management (CCM) reporting system] at 3 pm daily and loaded to the dashboard the following day." In other words, it doesn't look like the Power BI visuals are generated directly from the Ottawa open data, but instead from the provincial closed system.
However, the really good news is right below the explanation of the data source used (CCM), and before showing the dashboard, the page links directly to the open data: "The following data tables are available for download on Open Ottawa".
So this meets a key goal, it makes the data a kind of equal, so that you easily click the links and do your own analysis whether in a spreadsheet, by writing code, or using other data analysis tools.
It's hard to overstate what a big change this is in how data is provided to the general public. Linking to the open data at all is a big deal, but linking to it right at the top before the visualisation is a really strong and clear statement of openness.
There is a gap in that as far as I know there's no way to access the underlying Power BI setup (which incidentally generates five different pages, use the < > way down at the bottom of the Power BI frame on the page to see other Power BI pages), which means that you can see the really complicated visuals but you don't get much insight into how they're made or how you could use Power BI to make your own.
There is also a gap in that, I guess due to different cut-off time for updates, the municipal dashboard numbers are different from the provincial ones.
In terms of discoverability there are basically two pathways to the open data, one directly from Open Ottawa and one from Ottawa Public Health, but they are not quite equal.
If you land directly on Open Ottawa, you would find out it has COVID-19 open data even without a search as they are surfacing several datasets (I would guess this is automated based on traffic to the datasets, but it could be manually curated).
But you wouldn't find wastewater data, because it's on a different site https://613covid.ca/wastewater/
The wastewater site uses more of an academic format, and presents all of the data visualisations up front, before you find out below that you can "See the Methods page for more information on how the samples were collected, access to the data, and how the plots were created."
There are some issues at this point that are more web design and user-centric design than data. For example the Methods page is actually called model, and you can't get directly to the top of the page by clicking the Methods menu item, you have to click "Projection models" within that menu. If you're looking for data you're more likely to pull down the Methods menu and go to "Plot data", but that won't take you directly to the open data - basically you will have to navigate around the page quite a bit before you find out that the data are actually linked under "Waterwater surveillance". The data are made available on GitHub.
https://github.com/Big-Life-Lab/covid-19-wastewater
So not only could the wastewater site benefit from more directly and clearly linking to the GitHub data near to where it is shown in visualisations and tables, but you can't find the wastewater data at all if you start on the Open Ottawa site. So there is a gap in integrating these different open data sources. However, the use of GitHub does provide another path for discoverability, and adds the ability to easily share not just data but also code.
The situation for data is similar for COVID-19 projections, which can be found at
The source data for the projection models is the Ottawa open data, which is a good clear reuse of the data directly. The projection models themselves are drawn from a variety of clearly linked sources. You can see that the wastewater data is kind of added on to the models, which may be one reason for some content visibility issues in the website design. By navigating on GitHub I can also find that the 613covid.ca projections website is generated from
https://github.com/Big-Life-Lab/Ottawa-COVID-Projection
but I don't think you can find this on the projection website itself. It's good that the website content is on GitHub as it means pull requests can be submitted with suggested changes; however creating and submitting pull requests for a site built from GitHub is a high barrier for most users.
The Province of Ontario's open data site is https://data.ontario.ca/
COVID-19 open data can be found just using "COVID-19" as a search. In this case I've also filtered for just CSV format files.
https://data.ontario.ca/dataset?keywords_en=COVID-19&res_format=CSV
There are 19 dataset pages, which may link to more than one dataset per page. For example, to find daily new cases you first go to the page
Status of COVID-19 cases in Ontario
and then select the dataset
(PHU means Public Health Unit.)
Having a full page means that a lot of explanatory and related information can be provided. The data website itself has good functionality including the ability to preview a CSV dataset on the website itself as a grid (spreadsheet-like view), as a graph, or as a map.
This is a great way to allow people to explore data without needing their own tools.
There is also an API available for the datasets.
The main general use site is COVID-19 (coronavirus) in Ontario.
Right below a website display of COVID-19 cases, it has a prominent button:
and next to it,
Download the raw data from the Ontario Data Catalogue
So you have two highly visible pathways to the data. The data and details page provides tables, maps and charts, so you can work directly with the data on the web without needing specialised tools. At the bottom of the data and details page, it again links to the underlying data, in a "Where the numbers come from" section. You are left a bit to navigate the data catalogue to figure out which datasets generate which charts. The page would benefit from adding a CSV download button with each visualisation.
For example, I had to try several different data catalogue pages and examine a number of datasets before finding the new cases dataset.
Nevertheless, this is overall a really strong open data success, with multiple really clear linkages to the open data.
If you land directly on the Ontario Data Catalogue, you would find out it has COVID-19 open data even without a search as they are surfacing several datasets both under What's New (which I'm guessing is manually curated) and in the lists of Most popular and Recently updated datasets.
I will note however that their custom 2019 Novel Coronavirus group includes many more datasets than you would find just by searching COVID-19. There is a pathway from a dataset to the group, but you would have to notice that there is a Groups option at the top of every dataset.
The Government of Canada open data site is https://open.canada.ca/en/open-data
COVID-19 open data can be found just using "COVID-19" as a search. In this case, I've filtered to include CSV, JSON, GeoJSON, XLS, XLSX, XML only:
With these filters, there are 118 datasets.
If you look at e.g. Cumulative number of COVID-19 vaccine doses administered in Canada by jurisdiction
https://open.canada.ca/data/en/dataset/3b75a8d6-c5a9-48f9-834b-626eec16363f
you can see a clear description of the dataset and a link to download it in CSV format, but no tools to work with the dataset on the page itself.
The main general website is Government of Canada - Coronavirus disease (COVID-19).
A page about data can be found linked right in the top section
Current situation - COVID-19 data trends
Once we get there though, the situation is somewhat mixed.
The first link, Daily epidemiological report, takes you to many visualisations. But it is really PDF bound, including a link to a PDF version of the page at the top, as well as (somewhat indirectly) linking to a weekly PDF summary (week of April 9, 2021 in this example).
There are some links to underlying open data, e.g. to CSV open data underlying "Current situation", but you have to be pretty sharp-eyed to spot them.
That being said, on the data trends page, there are two links in trend data that will take you to data:
Overall I would say that while kudos is due to the Government of Canada for linking to open data, it could take lessons from the City of Ottawa and the Province of Ontario in more prominently and consistently surfacing the open data as part of its websites.
Similarly, if you land directly on the Government of Canada open data website
https://open.canada.ca/en/open-data
you get no indication that it has COVID-19 open data, unlike the prominent highlighting of COVID-19 open data on both the Ottawa and Ontario websites.
I will mention very briefly Our World in Data, which aggregates and then republishes international open data. For COVID-19 specifically see
https://ourworldindata.org/coronavirus
There is a clear open license at the top of the web page, and there are visualisation tools (charts) provided.
Notably both the data and the code are made available on GitHub.
https://github.com/owid/covid-19-data/tree/master/public/data
It is now much easier to work directly with open data in modern programming languages, particularly with CSV format data. As an example, you can see my previous blog post about Juypyter Notebooks (Python coding) using Google Colaboratory, and a Jupyter Notebook that I created to chart new cases of COVID-19 in Ottawa is available on GitHub: https://github.com/scilib/ontario_cv19_opendata_plot
This blog post is Copyright © 2021 Richard Akerman, and the text and associated images are licensed in the Creative Commons BY-NC 4.0 International.
This blog post is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
This license applies only to this blog post and not to the entire blog as a whole.
One of the issues with modern coding is the barriers to getting started. As I wrote in Can't Code There From Here, programming languages used to just be available, already configured at your command prompt, but now to start coding you have to download, and install, and then update, and then add code library upon mysterious code library. To quote my previous posting:
We've frontloaded one of the most tedious parts of computing, setting up a working coding environment, into the beginning of every code adventure.
And in an environment where there are constant cybersecurity issues, including attacks on coding supply chains, it's not exactly comfortable either to watch line after line of mysterious installs scrolling across the screen, or to try to wade through dependency alerts every time you try to write some simple code.
For these reasons, I've found Jupyter Notebooks in Google Colaboratory to be quite refreshing, as it's all already set up for you to access from your Google Account. No downloads, no install, just Internet ("cloud") coding. It is a research offering, so be mindful that it may go away at any time, but for now it gets you back to being able to code without any setup.
The good news is it is very straightforward on-screen. You can have text blocks, with, well, text (in Markdown) and separate executable Python code blocks. You can mix text and code blocks for a verbose, explanatory coding style.
Getting files (e.g. CSV data files) into Colaboratory is fairly straightforward. To get files from your Google Drive you have to mount the drive
from google.colab import drive
drive.mount('/content/drive')
and then get an authentication code. There's information on the Colaboratory site - Mounting Google Drive locally.
Once mounted your actual Google drive will typically be one layer down, /content/drive/My Drive/
You can also access files from your local file system, and apparently there is some kind of direct access to Google spreadsheets as well.
The sort-of bad news is that the nicely formatted text and code within a Jupyter environment turns into a jumble of JSON when saved in the .ipynb plain text format. You can't really work with Notebook .ipynb files outside of Jupyter. It's all chunks of
"cells": [
{
"cell_type": "markdown",
etc.
UPDATE 2021-04-04: I didn't realise that a Jupyter Notebook really is a notebook, in the sense that it preserve the last state of its output. I was wondering why a small .ipynb notebook file that I emailed to myself was 1MB, but it's because the .ipynb stores the last output state (including charts). I only found this out when viewing a notebook I had downloaded from Colaboratory and then uploaded to GitHub. Clicking on the notebook in GitHub shows not just the code and text, but all the outputs from the last code run at each stage.
If you don't intend to also share the outputs from your last code run, clear the contents before sharing a notebook.
Thanks to Juan Miguel Lavista on Twitter for additional information making this clear.
END UPDATE
In theory you can integrate Colaboratory directly with GitHub, but that means full read/write access.
If you're not keen on tying your Google Account and GitHub account together in this way, you can just do a more indirect download and upload pathway.
Colaboratory can read files directly from GitHub with no special permissions, just go to File -> Open notebook and select the GitHub tab. It didn't behave the way I expected when I put in a GitHub URL, but it worked fine when I put in my username.
Within GitHub itself there's not a lot you can do with a notebook file, as far as GitHub is concerned it's a "non-code file".
UPDATE 2021-04-04: As indicated in the previous section, you can view a Jupyter Notebook as basically a web page, including any outputs that were saved with it, by clicking on the .ipynb link in GitHub. END UPDATE
And that's about it. It is not of course exactly a free-for-all. There are some kind of RAM and disk limits that you will hit at a certain point.
If you do want to go the locally-installed route, there are several options for Jupyter Notebooks including Anaconda and Microsoft Visual Studio Code (for Windows, Mac, and Linux), but they require not only the aforementioned multiple dependency levels of downloads, but in a work environment may also require passing a proxy parameter if your web connection goes through a proxy.
The G7 Open Science Working Group was created in the 2016 Science Ministers' Communiqué, the Tsukuba Communiqué (PDF)
Establish a working group on open science with the aims of sharing open science policies, exploring supportive incentive structures, and identifying good practices for promoting increasing access to the results of publicly funded research, including scientific data and publications, coordinating as appropriate with the Organisation for Economic Co‐operation and Development (OECD) and Research Data Alliance (RDA), and other relevant groups
The 2017 Science Ministers' Communiqué (PDF) is now available, with a statement about Open Science as well as a summary of the G7 Open Science Working Group (G7 OS WG) report. The Communiqué directs the Open Science Working Group to continue its work.
We recognize that ICT developments, the digitisation and the vast availability of data, efforts to push the science frontiers, and the need to address complex economic and societal challenges, are transforming the way in which science is performed towards Open Science paradigms. We agree that an international approach can help the speed and coherence of this transition, and that it should target in particular two aspects. First, the incentives for the openness of the research ecosystem: the evaluation of research careers should better recognize and reward Open Science activities. Secondly, the infrastructures for an optimal use of research data: all researchers should be able to deposit, access and analyse scientific data across disciplines and at the global scale, and research data should adhere to the FAIR principles of being findable, accessible, interoperable, and reusable.
We support the work and results achieved so far by the G7 Open Science Working group. The OS Working Group has identified priorities that deserve and require common aligned actions, both in encouraging openness and data skills in scientific research practice, through workforce development and training. We encourage the OS WG to follow-up actions taken by G7 members according to the WG’s recommendations and to collect good practices, in order to report to the next G7 Science Minister’s Meeting. In particular, we support the OS WG deepening its efforts on the two topics identified above (paragraph 19), namely the incentives for openness of the research ecosystem, including the role of research indicators and metrics relevant to open science, and the infrastructures and standards for optimal use of research. The summary report of the OS working group is attached to this Communiqué.
To extract specific priorities and direction:
The statement "The summary report of the OS working group is attached to this Communiqué." is a bit confusing - the summary report is actually a separate annex document, it's not part of the main Communiqué PDF.
The summary report (PDF) also contains a key opening recommendation:
Overall, the G7 Open Science Working Group recommends that each G7 nation convene and engage with relevant stakeholders who can support moves towards incentivizing the increased adoption of Open Science in their national context.
as well as potential future discussion for the G7 in the areas of incentives and infrastructure for Open Science:
Incentives -
Potential future discussion at the G7 level: Consider approaches towards developing principles, for measuring the quality and impact of research enabled by Open Science practices. Discussions could also include means to identify, promote and implement best practices for Open Science, for example, through codes of conduct that build on existing procedures. This might lead to international consensus on the roles and responsibilities of researchers, institutions, and funders under Open Science.
Infrastructure -
Potential future discussion at the G7 level: Establish a forum for continued multilateral discussion to share information about major research data infrastructure initiatives in G7 countries and beyond. Discussion objectives could include:
Development of a common understanding of Open Science and the related infrastructure requirements (e.g. for whom? Under which conditions?), including consideration of best practices in different disciplines of science and technology as well as in social sciences, humanities and the arts.
Agreement to promote the FAIR principles for data resulting from government- funded research, increase discussion on sustainability and working towards shared guidelines for incorporating FAIR principles into the operations of infrastructure that support Open Science.
Raising awareness of the necessity of well-qualified human resources and professional expertise for curation, annotation and other management of research data and incorporate them in the research processes.
Disclaimer: I was one of the Canadian representatives on the G7 Open Science Working Group in 2016-2017.
SIDEBAR: The FAIR principles mentioned are that data should be Findable, Accessible, Interoperable and Re-usable. See e.g. https://www.force11.org/fairprinciples END SIDEBAR
The G7 will be held in Canada in 2018. The dates and location for the 2018 Science Ministerial have not been announced.
The 2016 Communiqué website is http://www8.cao.go.jp/cstp/english/others/communique_en.html and there were two annexes to the Communiqué:
- G7 follow-up workshop on mapping, coordination and joint activities in the field of Neglected Tropical Diseases (NTDs) and Poverty-Related Infectious Diseases (PRDs) - Recommendations – G7 expert workshop on future of the oceans and seasThe 2017 Communiqué website is http://www.g7italy.it/en/documenti-ministeriali and there were four annexes to the Communiqué:
G7 Science Communiqué Annex 1 - Future of the Seas and Oceans Annex 2 – Report of the group of senior officials on global research infrastructures Annex 3 - WG Neglected Tropical Diseases and Poverty Related Diseases Annex 4 - WG Open ScienceUPDATE 2017-10-03: You can also find a standalone page about the Science Ministerial at http://www.g7italy.it/en/news/the-ministerial-meeting-on-science-has-ended END UPDATE
UPDATE 2018-02-08: There is also an Italy 2017 Science Ministerial page at http://www.g7italy.it/en/science-ministerial-meeting but the link on that page to the Communiqué is currently broken. END UPDATE
The 2018 G7 website is http://www.international.gc.ca/g7/ - updated to https://g7.gc.ca/
Twitter:
![[add RSS feed]](http://www.feedburner.com/fb/images/pub/fb_pwrd8831.gif)
![[add RSS feed]](http://www.feedburner.com/fb/images/pub/feed-icon32x32.png)
Recent Comments