Science Library Pad

csvconf 2014 - The Content Mine - notes

csv,conf 2014 - July 15, 2014

14:00 The Content Mine by Peter Murray-Rust @petermurrayrust

Presentation: http://www.slideshare.net/petermurrayrust/csvconf

• "liberating scientific facts for humanity"
• crawl, then data mine the literature, publish resulting facts CC0
• uses technology to extract information from PDFs, "can now turn PDFs into science"
• information will go to Wikipedia and as linked open data to DBpedia
• site is contentmine.org

Peter is using the UK Exceptions to Copyright for Research (PDF), specifically the text and data mining exception.

csvconf 2014 - Data Hacking with Wikimedia Projects - notes

csv,conf 2014 - July 15, 2014

12:20 Data-Hacking with Wikimedia Projects: Learn by Example, Including Wikipedia, WikiData and Beyond by Matt Senate @wrought and Max Klein @notconfusing

Matt and Max have clearly done a lot of thinking about how to make a project work within the WikiMedia context.  The project they described is about automatically extracting information about open access content and then enriching multiple WikiMedia sites with the appropriate information.

Video available at https://www.youtube.com/watch?v=w5T6RoEckAQ

• Wikipedia is by far the largest of the WikiMedia projects
• In order to contribute successfully, you need to understand the community for each project, the cultural protocols.
• Rambot added US Census data to WikiMedia
• When contributing from an open access article to WikiMedia
• text to WikiSource
• media (images etc.) to WikiMedia Commons
• metadata to Wikidata
• signals (information about the article such as license) used to enrich e.g. citations in Wikipedia
• the bot is called RecitationBot
• goals:
• improve citation management
• integrate metadata into wikidata
• forge deep knowledge resources
• automation combined with human judgement
• Paths for Data Hackers (things to work on in order to succeed, particularly in WikiMedia context)
• reputation
• understanding of cultural protocols
• knowing the history and context of previous efforts, and fitting your work into that narrative
• connecting to community e.g. IRC, mailing lists
• RecitationBot is essentially a process of enriching newly posted citations
• can also initiate RecitationBot with a DOI as you create a reference

csvconf 2014 - CSV Validation at UK Archives - notes

csv,conf 2014 - July 15, 2014

11:55 session CSV Validation by Adam Retter @adamretter

Video available at https://www.youtube.com/watch?v=9pIBG5iv5XI

• has been working at UK National Archives
• there is a new digital archive for the UK - the Digital Records Infrastructure (DRI)
• need to ingest metadata from many sources
• CSV was decided upon as file format for metadata
• many possible errors in the CSV provided (including some people just rename Excel files, rather than exporting as CSV)
• approach: fail fast as repeat
• was decided to create a CSV Validator
• providing code to suppliers - now the suppliers can do up-front data validation
• want to be able to write rules, beyond just basic checks, and have it make easy enough to do so that domain experts can write rules
• CSV Schema language as a way to write rules
• https://github.com/digital-preservation/
• looking for developers to help advance this work

See also UK National Archives blog: CSV validator – a new digital preservation tool.

csvconf 2014 - Data usability: the tabular data package - notes

csv,conf 2014 - July 15, 2014

10:35 session Data usability: the tabular data package and friends by Nick Stenning @nickstenning

Data publishers at #csvconf: my message to you today is “put it in a box.” Publish your data as data packages: http://dataprotocols.org/

— Nick Stenning (@nickstenning) July 15, 2014

Slides at https://github.com/nickstenning/put-it-in-a-box

• analogy with international shipping: things in containers vs. people carrying them individually
• things not in containers are very expensive to move
• which means you do a lot of work carrying the things, rather than using the things
• containers are: 1. simple, 2. opaque, 3. diverse within constraints ("moderated diversity")

Introducing The Data Package, datapackage.json

See info at http://dataprotocols.org/data-packages/#specification

My question: What about BagIt?

Answer from Rufus Pollock (@rufuspollock): Other formats were examined, there is a white paper.

UPDATE 2014-08-01: Review of Existing Packaging Work via @rufuspollock

csvconf 2014 - introduction - notes

csv,conf 2014 introduction by Max Ogden and Rufus Pollock - @maxogden and @rufuspollock
July 15, 2014

(csv is Comma-Separated Values, a way of representing data.)

• wanted to reach out to data hacker community
• wanted to bring together open data / open tools / open standards communities
• csv has strength of simplicity
• csv inherently "hackable"
• csv can be used for big or small data, it "scales", it "streams"

Hashtag #csvconf