Science Library Pad

Ottawa Transit Data Day - June 2, 2012

Having liberated Ottawa's real-time bus location data, the city is now moving to the next step of bringing developers and OC Transpo together to make (even more) amazing things.

Ottawa Transit Data Day is June 2, 2012 at Ottawa City Hall

Register at http://transit-data-yow.eventbrite.com/

Find more information about the available data in the Open Data Ottawa blog

Follow @opendataottawa

Suggest datasets at DataOtt

Ask questions in the GPS API mailing list

Provide feedback directly about OC Transpo open data: OCOpenData@ottawa.ca

Event hashtag will probably be #yowdata

So many communications channels.  Whew.

let's all sing a song of code

To develop is to be in the physical world, with all its rich social connections (and all its frustrating constraints) and yet to be able to tap into the invisible virtual world, to craft new things out of thin air.

This touches on the whole "should X be coders" debate. I'm inclined to agree with Rushcoff's Program or Be Programmed - we have to not just use the tools, we have to understand at least some of the underpinnings and be empowered to hack away to change things.

The trend today seems to be to teach JavaScript and I think that's fine.  HTML and JavaScript will get you a long way.  (If you protest that JavaScript is too far away from proper programming to give people a good start, remember that many of the people who are now professional programmers got their start writing in BASIC in the 1980s on various small computers.)  I do think for people who want to do more, there's a whole set of sysadmin skills that are very useful as well.  The important thing is there is a whole range of skills - just in the same way that everyone is expected to read and write but only a few people will have a PhD in English Literature, we should expect (and provide the opportunity for) some basic computer science skills, without everyone needing to do a full degree in CS.

In the UK there's quite a bit of discussion about moving the computer curriculum from teaching how to use Microsoft Office (I shudder to think this is actually considered "computer training") to covering more foundational concepts and teaching how to code.

A story arrived in my Twitter stream that illustrates just how fundamental this set of skills can be - with painful honesty Shawn Graham tells a tale of failure and lessons learned - a community that succeeded on the front end, but fell apart (no documentation, taken over by spammers, no backups) on the back end: How I Lost the Crowd: A Tale of Sorrow and Hope.

We need to get to a place where people can have great ideas and know how to implement them, either themselves or by knowing the right resources to draw upon.  To some extent the new outsiders are the people who don't know how to code.  In professions that once seemed perhaps distant from technology such as journalism and the humanities, coding is becoming a core competency.  Fortunately there are great initiatives like Girl Develop It, Young Rewired State and Mozilla Webmaker (the source of the video I embedded above).  No matter where you are it's almost certain there's a meetup or three for beginning coders.  Let's get coding.

Previously:
July 24, 2007  software development, staffing and new library technology
December 7, 2005   librarians 2.0 don't need to be coders 2.0

international open data hack days

Random Hacks of Kindness (RHoK) will be December 3 & 4, 2011.

It's a global collaboration with the possibility of local events.

It happens that International Open Data Hackathon also falls on December 3, 2011.  It is focused on getting cities to host hackathon events.

Ottawa is hosting one at City Hall, you can register for free

http://www.eventbrite.com/event/2488780008/

It's not just a technical event, it's about bringing everyone with an interest in open data, open government, and citizen engagement together so we can exchange ideas and make stuff.

The hashtags are #rhok and #odhd

20th century organisations with 21st century challenges

In general, people want their organisations to do wonderful things. Yet even when organisations are staffed with individually excellent employees, often the result is less than the sum of its parts. Or in other words, "Why can't my organisation do x?"

You can think of this as the lifecycle of an idea.

idea -> implementation -> operations

The above of course only works if you have infinite resources, to take the vast number of ideas, implement them all, and maintain them forever.  Since this is impossible, instead we try to talk about this using an analogy to living things, e.g. from seed to growing tree to mature tree to chopped down tree.

There are many different angles on this, in technology commonly we talk about the Software Development Life Cycle.  Unfortunately in most organisations, this is actually the Software Development Lie Cycle, for many reasons.

First is that the above line from idea to operations is woefully oversimplified.  It's actually a loop with current state turning into future state (which becomes the new current state)

(entire current organisation and current technology and idea environment) -> new ideas and improvements on current systems -> gateway -> prototypes and upgrade projects -> gateway -> new and existing operations (future state) -> death gateway -> decommisioned systems

There are at least three problems that organisations encounter as they go around this loop

1. The processes are bad at each step

2. The staff and management are unable to properly execute the processes, even if the processes themselves are good

3. The entire world changes while you're trying to move from your fixed current state to a wishfully-fixed future state

Specifically when it comes to information technology, big organisations are bad at IT.  Also small organisations.  Also software engineers.  Also, well, basically everyone.

The good news is we have a handle on some of the main problems.  In fact we've been trying to find solutions to these problems for decades.  The bad news is that solving these problems is very hard, and so quick fixes rather than real fixes are very tempting.

One of the underlying issues is the overall factory model we have both for producing and managing.  This is very deeply embedded both in every organisation as well as in the assumptions of every employee and manager.  It is hard to break out of a century-long mindset; in fact it's hard to realise you even have that mindset.

Software engineers have tried to attack this problem from all fronts: better languages (e.g. Ada from the US Defence Department), better methodologies (e.g. agile instead of waterfall), better overall context (e.g. Service-Oriented Architecture), better component sharing (e.g. github).

All of these are challenging because of the underlying factory organisation model.  For large organisations in particular, certain approaches pull the entire organisation as if by magnetism.  In particular, organisations will tend to do

* big projects (in people resources, dollars, and time elapsed)

* complex projects (e.g. many different technologies to be integrated)

* ambitious projects (e.g. trying to fix many problems at once)

* business-incremental projects (i.e. to add new features to a current process or product offering)

* perceived low-risk projects (e.g. with elaborate cost analysis and justifications)

* poorly-communicated projects (management doesn't communicate their intentions "down", IT doesn't communicate complexities "up" and comms doesn't talk to EA which doesn't talk to the coders who don't talk to... etc.)

Also business tends to measure certain costs while ignoring others (typically capital expenditures and coding time are carefully tracked, but having endless meetings is often considered to have zero cost).

This force pulls on all initiatives, pulling them towards having more stakeholders, more committees, more senior-level buy-in, more analysis, more consultations.  And once substantial effort is invested, the "sunk cost" psychology means that it is almost impossible to stop a running initiative, or to turn off a running system.

This force pulls towards bigness and complexity in general - towards big bang Enterprise Architecture, elaborate Service-Oriented Architecture, big-iron computers and networking hardware, "enterprise-class" heavyweight software backed by giant consulting firms... organisations somehow believe if they can just do big enough things with big enough technology, their problems will be solved.

There are many, many reports over decades that have documented this.  A recent one is System Error: fixing the flaws in government IT from the Institute for Government in the UK.

Pretty much everyone wants to fix this.   (Of course consultants may in some cases have perverse incentives to perpetuate dysfunctional processes that generate a lot of money for them, and managers may want to retain processes that give them power, but in general people want to do good work.)

One key problem is that usually people don't take a systems view.  They identify one problem area in the cycle and want to attack just that, thinking that will solve the entire problem.  For example:

* if we could just generate more / better ideas, everything else would work

* if we had a better planning process, everything else would work

* if we did more prototypes, everything else would work

* if we had more hackers, everything else would work

* if we changed development methodologies (e.g. to agile development), everything else would work

The reality of course is that unless you attack ALL steps in this cycle, the end results will always be, as they say, hashtag #fail

And even if you do each individual step well, you may still be choosing to do the wrong things.

Let's take a typical example.  Your organisation has an existing telephone system (PBX with voicemail) and knows to the penny most of the costs (in reality it probably knows most of the clear expenditures but has a more hazy idea of systems maintenance costs and opportunity costs).  Someone, probably at a senior level, has read about Voice over IP (VoIP).  Investigation initiated.  RFP written.  Consultants consulted.  ROI laid out.  Network upgrades needed.  Hardware to purchase.  Desktop software to integrate.  Options to evaluation.  Choices are made, purchasing, coding, implementing... a nice project that takes a year+ to replace a telephone on an employee's desk with... a telephone on an employee's desk.  Everyone happy.

Except in all this process no one stepped back to say: do we need the telephone at all?  Does anyone like using the telephone?  Are there other cheaper solutions?  What if we got rid of telephones altogether?  What do telephones do to knowledge work?

In fact, you can find whole threads of productivity literature that will tell you that telephone interruptions disrupt knowledge work, and that other methods of communication may be more effective.  In parallel the trends are clearly towards a global decline in voice traffic and a huge increase in data traffic.  People are not talking on their cellphones, they're texting and Facebooking.  Cartoons are appearing about the "phone gap", where texting users expect to receive a text asking whether a call can be made, rather than the phone just ringing out of the blue.

Over a decade ago in The Time Trap, Telephone Interruptions get an entire chapter

There is something wholly irresistible about a ringing telephone. ... Yet tune it out you must, for telephone interruptions can shatter your concentration and fracture your productivity as nothing else can.

(In fact the solution to this is quite simple, it just requires throwing out 20th Century convention and turning the ringer on your phone off, which is what I do.  My work phone and Blackberry never ring.  Although mostly these days they never ring because no one uses the phone any more.)

All this to say, even if somehow every stage of the giant VoIP project was perfectly executed, they still would have done a good job building the wrong thing.  But in general of course, every single stage is badly executed.

So you need a plan of attack for every stage.  At every stage you need BOTH better process and better-trained capable people.  Like a bike lane system which is barely used at 98% complete but starts to hum when every pathway is connected, you only get the big improvements when every stage of your ideas cycle works well.  In addition, we need the ability to continuously step back to the big picture and decide whether an entire initative still makes sense in light of changing conditions.

So you must

* know your current state well (most organisations have no idea what their actual processes are and are often shocked to see ridiculous legacy processes once they're diagrammed on a whiteboard)

* have a good process for gathering and generating ideas (often perceived as a major barrier, this is actually quite straightforward - the issue is rarely a shortage of ideas)

* have a very good process for gatewaying ideas to the investigation stage (making sure to let through both some high-risk prototype ideas and some improve-the-business ideas)

* be very very fast and agile at cranking out high-risk prototypes

* have a very good and very strict process for gatewaying prototypes into full implementation (sunk cost and idea attachment tend to make it hard to kill a running system no matter how many times it was emphasized that it was temporary)

* have a short process to take very focused features to production (e.g. a few months for a few very specific features)

* be adaptable as everything changes along the way

* have a process to kill things from production once they're no longer relevant (this is like the prototype killing problem, except multiplied by 100 as there may be decades of sunk cost)

* have a process to continuously review running initiatives and kill them (or modify them dramatically) if the world has changed while they were running

* Start tracking real employee costs not just specific activities and expenditures.  If you have to have a committee and burn $5000 of people time to decide to spend $500 on an Internet tool license, you have a costing misallocation problem.

At every stage just the process alone won't do it, you also need people who are creative, well-trained, adaptable, communicative, sharing their knowledge and documenting what they're doing as they go along.

So for example you might:

* immediately document your current systems (enterprise architecture)

* introduce an environment scanning process (and/or partner with existing horizon-scanning initiatives)

* devise a ruthless gating process that senior management can implement with input from all staff

* reward success and failure - reward successful products but also celebrate the lessons from prototypes that don't pass the gates

* create a prototyping group (innovation team, tiger team, skunkworks etc.) but ensure it has a close working relationship (or is simply a role within) the main implementation/operations team

* Ensure everyone is getting continuous training.  Including senior management.  Create time for reverse mentoring and innovation.

* Commit to FULLY FUNDING THE OPERATIONS of prototypes which pass the gate into full implementation

* Commit to annually review the relevance of running systems and to plan to kill the ones that need to retire entirely or be replaced with new approches

* If you are doing technology projects, ensure you aren't a technology island.  That means many different things including:

** the teams need full, unblocked Internet access, because that's where the information and the tools are

** the teams need to connect out to modern approaches and other developers - this means participating in local hackfests, sharing code on github, etc.

** the teams need to be able to share, publically, their challenges, the work they're doing - through blogs, presentations etc.

This is all a long, long way from our current factory model.  In the factory model, interchangeable employees toil on the factory floor, silently, while high above the managers gaze down benevolently from their glass-encased offices, and every project is initiated from senior management, for senior management.  It is a long way from a hierarchical 20th century factory organisation to a 21st century agile adhocracy that is both well-connected internally and well-connected out to the entire world of agile innovation.

Plus which, even if you have all of this top of mind, you can still fail.  Microsoft and Google hire the top computer scientists (and PhDs in many other areas) every year, AND have deliberate initiatives to attack every stage of this process, to try to ensure they don't fall into the big organisation trap... and yet both Microsoft and Google continue to fall into this trap in various ways.  The point being, even organisations that set out to build agility into their DNA like Google can fail, which is why you get endless "why Google can't build (small startup x)" or "why Yahoo Flickr didn't invent (new photo sharing approach y)".  In fact to prove my point here's both Why Google Can't Build Instagram AND Flickr Should Have Built Instagram. But They Didn’t. Here’s Why.

The fact of the matter is, whether your organisation is 300, 3000, or 300,000 it is never going to be the same as 8 people in a room with a coffeepot.  The best you can do is be mindful of that, and have a mix of internal improvements and standalone innovation, spinning in and out of your organisation continuously.

And that is my answer to @dbast's question

@sdkibb @jesgood @s_dav @mrpwalt @jeromefb how can I convince you to organize a full-day #goc "hackothon", besides that it would be awesome?

Our organisations are not incapable of coming together to create awesome prototypes.  The data.gc.ca prototype was done with a mix of dedicated resources and corner-of-desk, in 7 weeks.  It would be another year before it was made publically available.  GCPEDIA was done in months and has run for over two years, but it has yet to be operationally integrated or sustainably funded.  Hacking for hacking's sake will produce more orphaned prototypes that no one wants to sustain but no one is willing to shut down.

Hacking for hacking's sake also won't help if at least at a minimum the organisation itself has not recognized the problems of big projects and backed up that recognition with both public statements and effective internal initiatives.

I'm not saying hacking to demonstrate the art of the possible isn't great - it is.  Hackathons and apps contests have in many cases opened the eyes of CIOs to what can be done with openness combined with outreach and enthusiasm.  But unless you're willing to turn everything off the day after the demo, you still need to have an organisation-wide transformation in order to have a repeatable process to get from innovation through a gateway to full implementation and sustainable operations.

Basically the situation sucks.  This is not a factory, Taylorism doesn't apply to solve the entire problem.  We can't chop this up into smaller and smaller problems and make each piece more and more efficient.  If you have The Innovation Group and The Gateway Group and The Operations Group you just end up with efficient silos and an inefficient organisation.  This problem needs both bottom-up attacks to demonstrate local efficiencies and possibilities as well as systems thinking, a systematic organisation-wide attack on the problem as a whole.  Taylorism is easy and high-achievers get to be individual Linchpin superheros.  Systems change is hard.  But systems change is what we need.

Facebook and privacy in Canada

The Office of the Privacy Commissioner of Canada has released their final agreement with Facebook, in which Facebook notably agrees to change the API to third-party applications, enabling much more granular control of the personal information you share with them.

News Release - Facebook agrees to address Privacy Commissioner’s concerns - August 27, 2009

(announced via the the @PrivacyPrivee Twitter feed, incidentally)

This is an issue that has concerned me for some time, so it's great to see it being resolved in a positive manner.

Here's a question I asked about Facebook in 2007, when I was blogging the OECD Participative Web forum

QUESTION: Hi, Richard Akerman from the National Science Library of Canada.

One of the things that I've seen in the discussion is we are talking mostly about silos, but Web 2.0 is about mashing sites up, about linking sites together, about crossing between sites and combining them together.

Not to pick on Facebook, but Facebook has a fabulous feature, which is Facebook Applications. However, in order for me to give my informed consent, I have only one choice. To use this application, I share my information with a third party.

I think that is a valid option, but the question, the broader question, the policy question is: How do we deal with privacy when we expect that sites will want to interlink like this, that people will want to connect their information like this? How do we control the spread of the information?

Are there technological ways to do that? Are there policy ways to manage it? If I share with a third party, how do I stop the third party from sharing on?

So I'm interested obviously particularly in the Facebook experience but the broader panel as well.

Thank you.

and here's the panel discussion that ensued - the cast of characters is Mozelle Thompson from Facebook, John Lawford from the Public Interest Advocacy Centre, and Gary Davis from the Irish Internet Data Authority, with Hugh Stephenson from the U.S. Federal Trade Commission chairing.

MR. THOMPSON: I think that question is there for a reason. I mean, when I say that, when it warns you that in order to use this application, you have to share some information with that application, it's because if you don't want to share your information with that application, you should not download that application.

One of the things, you are absolutely correct we have over 5,000 applications. And aside from the applications that are created by Facebook itself, it is very difficult to police every single other one for what everybody else does.

For example, if Amazon has an application that you can download on Facebook, then you are going to have to be guided by Amazon's policy.

That being said, do we have certain standards about data mining and other things? Absolutely.

We tell sites that if they want to create an application and they want to ask you for information, that's great. We are not going to give you information about our users. We leave it then up to the user to determine whether they want to use this application or not. And that has to do with a trusted site relationship.

MR. STEVENSON: Thank you.

John, I think you wanted to get on this, and then Gary, and then one more question.

MR. LAWFORD: The way you dealt with that in legislation, you just ask for someone's consent, right, and that should be the end of it. If you don't want to use that program, you don't consent, except that what you are getting for that application is they are asking for more personal information probably in your sign‑up than they need to to provide that application to you.

They've already got the fact that you have been referred from Facebook and now they are asking for additional personal information.

That's where we are saying that for a Web 2.0 type statute, whether internationally or nationally, you should be able to ask for the plain vanilla transaction. So you have name, address, if you need it, and I get my application, not all this other stuff.

MR. THOMPSON: That's a little bit misleading in the following sense: that is you are Amazon and you have an application on Facebook or some other company has an application on Facebook, if it's Expedia or Travelocity, they are going to need some information from you in order for them to do a transaction with you. That's your relationship with them.

We are not collecting that information. That third party is collecting that information. That's the purpose of the warning. Not because we need that information. We already know what we need to know because you are our user. You are absolutely right.

But we put the warning there so that if you are using a third party application, you know that they are collecting information about you. It's a benefit to consumers.

MR. STEVENSON: Thank you.

Let's give Gary a chance to intervene on this and then I think we have one more question.

MR. DAVIS: Just from a data protection perspective, I don't know the actual characteristics of Facebook applications and there could be anything else.

One of the principles is the purpose limitations. So if I give my information for one purpose, which is to sign up to that, the third party, then if they anything else with it other than the reason for which you gave it, then you would have a valid complaint to us as the Data Protection Commissioner's Office and we would investigate it.

Also, and again understanding the nature of the relationship that exists, if Facebook applications could be deemed to be handling the information on behalf of Facebook, well then there's a contractual obligation there. And one might say that a privacy standard would be that the contract that is entered into would specify between Facebook and whoever manages Facebook applications, that they may not use the information for any other purpose.

I would expect to see that. If you weren't seeing that going forward, well then that's a privacy point that one would expect to be articulated.

MR. STEVENSON: Thank you.

from OECD transcript

Web APIs explained on CBC Spark - and Open Data Under Construction

(I almost wrote Spark CBC, since that's their Twitter name.)

Spark Episode 76 (audio link available directly in the post, as well as various podcast options)

At 22:02 or so in, they take on the challenge of explaining web APIs, or more specificially, they ask Jer Thorp to help walk them through the concept.  It's always interesting to hear the descriptions people use.  For example, I would generally say "machine-to-machine", which is probably way too abstract.  I also tend(ed) to describe APIs in the context of Service-Oriented Architecture, which probably confused the issue (and the audience).  I don't generally talk about computer programs communicating with other computer programs.

I think in general what's presented on the show is a pretty good explanation: websites are opening up their information using APIs, so they can leverage open innovation - outside developers.  We are a long long way from a completely interoperable web of standard APIs though.

Here's the Twitter-sized explanation I had proposed (taking quite a lot of my space to talk about how there wasn't enough space):

scilib: @sparkcbc I don't think that will fit in a tweet. Basically standard interfaces (APIs) allow data to flow between sites, which = mashups.

I would argue as well that web development has gotten sophisticated enough that, while APIs are ideal (at least if well constructed), you can actually get a lot by opening your data, which is the key first step.  Open data enables mashups, APIs just make mashups easier.  Open data means sharing the information your organisation has, out on the web - ideally your default becomes to share.

We're still in early days of open data.  The Guardian calls their approach "Data Store - Facts You Can Use".  I've written previously about the US Data.gov initiative, which currently has the world's simplest website (a giant box reading "coming soon"), but I think is supposed to launch this month.  It's similarly challenging to point to open data cities, because while the Twitter-enabled Toronto @MayorMiller announced toronto.ca/open at Mesh, it also reads simply "Under Construction".

What will be possible is mashups, visualizations, APIs, analysis and much more.

I believe the long term success of projects like StimulusWatch Canada and ChangeCamp Ottawa will depend on open data, and (eventually) on all levels of government having open APIs as well.

Which circles me around to the opening topic of the podcast, about whether online activism ("slacktivism") can actually translate into meaningful real-world activity.  The answer, I think, is tied in with the segment about lurking... the web is mostly lurk, only maybe 10% participate.  Some tiny fraction of those online participants might translate into offline actions.  Maybe one in a thousand?  But nevertheless, it does happen.

While I generally refuse to join these "click your support" Facebook groups (in part because I don't like FB much anyway), they can be low barrier entry points, in particular since so many Canadians (who may otherwise not be very social-web enabled) are in FB.

The kind of canonical Canadian example is the Fair Copyright for Canada group, with its (at time of posting) 90,071 members.  It was brought up in the House of Commons.  It did translate into some offline activism.  And the sheer numbers did, I think, get both attention and generate concern for the party proposing the bill.  There are still lots of issues with that number.  Lots of people around the world care about copyright.  For all I know, that's 81,000 copyright-concerned Americans, and 9000 Canadians.  Such is the global web.

I do think "feel-good clicks" are a bit dangerous, they give you the perception of action without actually doing anything.  I've long been concerned by this kinda of almost mystical power ascribed to online organising.  In my review of Al Gore's The Assault on Reason, I said

Don't get me wrong, I think the Internet has a role to play in reasoned discourse. A small role. A useful tool for pointing attention to falsehoods and referencing inconvenient truths. But electronic communications have a fatal allure of virtual action.

Concerned about the environment? No need to go outside and walk in the woods, or clean up a polluted lot in your neighbourhood, or knock on your representative's door and explain the urgency of your position.

No, instead you can just fire off an email, write a blog posting, and then turn up the air conditioning and the lights and stretch out on the couch and read a good book.

That being said, I have myself translating the online into real world action on a number of occasions.  As I wrote in the StimulusWatch blog, it was an online posting that led me to an event that started a chain leading to the creation of the project.

That same event, and online chatter about a local conference, also led me (as partially outlined in my posting Making government data visible - and is Change coming to Ottawa?) to ChangeCamp Ottawa, a very real event happening at City Hall on May 16, which I have been helping to organise, an event which of course has a substantial online presence including a social network for the specific event, as well as being part of the larger ChangeCamp group on Facebook.

Similarly, a local news article in a free neighbourhood paper (yes, in print, with ink and everything) about a small garden/park space led me to a Facebook group which led me to an offline meeting which led me to create http://www.savethegarden.ca/

And of course, on a much more spectacular scale, the Obama campaign used (and continues to use) online organising as a tool, but they were very clear that the purpose of online was to drive a very extensive (and successful) ground game, people talking and knocking on doors, calling on phones, out in the real world.

So I think when it works best, the online world leads you offline, and offline leads you back online.  It's an ongoing discussion that flows across place and time.

Discussions enable meetings, data enables websites, websites enable more meetings, meetings come to consensus on APIs, APIs enable mashups... round and round it goes.

library and publishing contests to develop software and innovate

Current Contests:

* OCLC launched their third Research Software Contest on April 13, 2009.
* The second repository Developer Challenge will be at Open Repositories 2009 (May 18-21, 2009)

Past Contests:

* The Elsevier Grand Challenge just wrapped, reflect.ws was the winning entry
* CISTI's internal Innovation Challenge wrapped up on March 19, 2009.  The winner was "Research in the News" (you can read more on the site).

This is only a brief survey and I'm sure I have missed some, but I'm blogging on the tiny keyboard of my netbook, so I am only doing a short posting.  Please let me know of any others.

NYPL is calling all widget builders

Our first widget is going to be a List Building Widget that will include platform integration for iGoogle, Facebook, MySpace, blogs and web pages, and/or desktops.    The widget will allow users to:

• Build lists of favorite book, movie, music, game, etc. lists
• Build lists of materials needed for homework projects
• Share lists with friends

We are looking for open source designs that can be made available to and repurposed by other organizations seeking to engage young people. The widgets will be developed as part of a 2008 National Leadership Grant from the Institute of Museum and Library Services titled Homework NYC Widgets: A Decentralized Approach to Homework Help By Public Libraries.

Read the RFP.

The project team will accept and answer questions about the proposal via comments on the NYPL Labs blog. 

See the blog post for more information, including contact information.  Deadline to submit proposals is May 1, 2009.

It's great to see libraries reaching out to their community, to build technology that will benefit lots of organisations.

Access 2008 preliminary programme up

http://access2008.blog.lib.mcmaster.ca/program/

open source, mashups, browser extensions, next gen catalogues/opacs and such

via Library Boy

Mashed Libraries UK 2008

Owen Stephens, inspired by Mashed Museum, is proposing Mashed Libraries UK 2008.

the idea is to have a reasonably informal event at which we try to do interesting stuff with library technology and/or data

He's put together a starting list of possible APIs... there must be many more that people could add or offer...

Services/APIs/Systems/Technology/Data that we could use

via Twitter

Open Repositories 2008

Through an unexpected series of events I find myself going to Open Repositories 2008

http://or08.ecs.soton.ac.uk/

The lineup looks great including a keynote from Peter Murray-Rust, and two (!) sessions on Scientific Repositories.

There is also a Repository Challenge for developers with a £2,500 prize, which is like a million US dollars now (finally, Canadians get to make US dollar jokes).  Kudos to David Flanders for leading this "let's just build stuff and see what works" approach.

I will be blogging under tag/category or08, and twittering under hashtag #or08

I made an Upcoming event, mainly because then if you add the machine tag

upcoming:event=455039

to your Flickr photos, it will automatically put in a nice "Taken at Open Repositories 2008" logo.

NLA announces Library Labs

The [National Library of Australia] has recently opened this "Library Labs" wiki space:

https://wiki.nla.gov.au/display/LABS/Home

The aim of this space is to let our colleagues know what we are doing, to invite comments, questions and feedback and to provide a space for discussion and collaboration.

We have started to redevelop our digital library services using a service-oriented architecture and open source software solutions where these are functional and robust.  We are also aiming to take a common ("single business") approach to collection management, discovery and delivery.

We are interested in forming a community of Australian business analysts and developers who are working on similar problems and who are interested in  interoperable, standards-based solutions. We are also interested in working with colleagues at an international level to provide prototypes and testbeds for new and emerging standards.

via Warwick Cathro
Assistant Director-General, Innovation
National Library of Australia

Google Book Search API

The Google Book Search Book Viewability API enables developers to:

                   

• Link to Books in Google Book Search using ISBNs, LCCNs, and OCLC numbers
• Know whether Google Book Search has a specific title and what the viewability of that title is
• Generate links to a thumbnail of the cover of a book
• Generate links to an informational page about a book
• Generate links to a preview of a book

http://code.google.com/apis/books/

via LibraryThing blog - Google Books in LibraryThing - March 13, 2008

We need APIs everywhere for everything.

Adobe adds AIR to cloud

AIR is intended to help software developers create applications that exist in part on a user’s PC or smartphone and in part on servers reachable through the Internet.

To computer users, the applications will look like any others on their device, represented by an icon. The AIR applications can mimic the functions of a Web browser but do not require a Web browser to run.

...

“There is a big cloud movement that is building an infrastructure that speaks directly to this kind of software and experience,” said Sean M. Maloney, Intel’s executive vice president.

New York Times - Adobe Blurs Line Between PC and Web - February 25, 2008

AIR has graduated from Adobe Labs, and is now available for free download at

http://www.adobe.com/products/air/

I have to say, I'm not really clear how this is any different from, or better than, Java.
I guess the argument is that it uses web standardsy stuff, so it's easier to program than Java.

Plus it's all well and good to say cloud this and cloud that, for example, but I don't see any indication that AIR provides you with some storage cloud from Adobe that you can use.

I found a posting by a GWT developer that briefly introduces the major competing technologies in this space

Adobe AIR/Apollo vs Ajax vs Gears vs Flash vs Silverlight vs JavaFX vs GWT - June 11, 2007

As well I wonder about the implications for searching.  Our search engines mostly eat text or things that can be converted easily to plain text (Word documents, PDFs, PowerPoints).  You can do a whole fancy site in Flash and to a search engine it will barely exist.  If we move from building text-based web sites to interaction based web apps, how will we ever be able to find anything again?

Access 2008 CFP

Location: Hamilton, Ontario, Canada
Date: October 1-4, 2008 (Hackfest: Oct 1; Conference: Oct 2-4)
CFP Deadline: Friday, February 22, 2008
URL: http://access2008.mcmaster.ca

Access is Canada's premier library technology conference, featuring a single stream of sessions that deal with technology planning, development, challenges and solutions. ... now accepting proposals for prepared talks on the following topics (other ideas are more than welcome):      

• customized web applications and search interfaces
  
• open source software
  
• national and provincial/state-wide consortia technology initiatives
  
• information policy
  
• digital and social media
  
• library catalogue innovations
  
• digitization projects
  
• institutional repositories
  
• end-user searching behaviours
  
• protocols and metadata

...or anything else suitably geeky, innovative and/or awe-inspiring

Previously:
Access2005 Access2006 Access2007

future directions of library systems

It was great to talk with various enterprise/IT/technology architects at DLF Fall Forum 2007 about our various ideas an approaches to modeling and implementing the new library technology infrastructure.  But it's clear there are still more library technology architects out there (even if they may not have that in their job title), I ran across an interesting blog posting by Jonathan Rochkind:

Notes on future directions of Library Systems from Bibliographic Wilderness blog, September 28, 2007

Here are some notes on near/medium future directions of the library systems environment/architecture, and a sketch of requirements on where we want to go. These notes may or may not end up as part of an internal white paper here, as we analyze where we want to be headed (something good to do anyway, but we got a kick in the pants when the vendor ended development on our current ILS).

...

The library systems environment has grown from being composed of a single “Integrated Library System” at the origin of library systems in the 1980s, to being composed of an ‘ecology’ of systems in most libraries today. These different systems could be from various sources (proprietary and open source), fulfill various–both overlapping and distinct–functions, are used in various ways by different subsets of users (both library staff and our end-users), and interact with each other to varying degrees (some well, some poorly). 

I wonder if I should consider adding a "library technology architecture" category to SLP?

PS Jonathan was at DLF but we didn't formally meet, it's odd how sometimes virtual connection and discovery is easier than in the physical world.

redesign the library web site as a set of network services

The last three weeks I’ve been thinking a great deal about the role of my department in fulfilling the Libraries mission and where the department needs to go in the next 3 years. Part of getting where we want to go has been this whole site redesign process. But not in the way most of the library thinks.

...

Long term I’d like a site which has a series of web services that can be exploited by my developers but also my the university web developers and who knows who else.

...

For faculty and grad students who want to do known item searching in our catalog, maybe something like LibX is the way to go. Or maybe allowing users to create their own search interface to a set of particular resources that they can embed in their browsers search bar or on their desktop as a search widget.

Ultimately, I feel like it is these kinds of services that will make of break a library’s virtual presence not the library website.

Library Web Chic - The future of Web Services isn’t the Library website - September 16, 2007

Err, ditto.

Previously:
July 16, 2007  Library Journal netConnect - Web Services and the Social Catalogue

Charles Simonyi in Technology Review

I must admit that even though there were a remarkable number of famous and extremely accomplished people at SciFoo, I had the benefit of not knowing who most of them were.  You might think this is a rather odd perspective, not least of which because I'm usually the last person to laud the benefits of being ill-informed, but it made it much easier to sit across from Eva Vertes and have a conversation without knowing in advance that she's a super-genius cancer researcher.

Anyhoo, I found this profile of Charles Simonyi in Technology Review interesting, I didn't know about his foundational involvement with Microsoft Office, and his new project to create a new type of programming environment, intentional software.  It might have been interesting to hear a question like "how was it like going back to Russia to train for your spaceflight after having left Hungary all those many years ago" rather than some the questions that were asked (e.g. "how does it smell in space").

software development, staffing and new library technology

Librarian in Black responded to my article in Library Journal netConnect, and Richard Wallis' commentary thereon.  She says

What makes me sad is that both Ackerman [sic] and Wallis have missed a key point: if the future is in web services, how can libraries take advantage of that with their current staff configurations?  How many libraries in the U.S. have a honest-to-goodness computer programmer on staff?  How many have staff with Computer Science degrees?  How many staff do they have devoted to the library's hardware, software, and network?  How many staff do they have devoted to web services?

In the smallest libraries, perhaps all of these are the same one person.

My article is about modern software engineering for libraries, not on how to staff them.  However, in the very same issue of netConnect there's an article by Karen Coombs "Digital Promise and Peril" calling for library staffing to reflect the digital content environment

Many of these digital materials are in jeopardy of being lost because librarians have not yet found adequate ways to collect and manage them. In part, this is because roles and skill sets have been siloed in libraries. Materials preservation issues have typically been the purview of special collections and archives units within the library. In contrast, cataloging expertise has resided in technical services, and technology expertise has typically resided in systems. To collect and manage born digital objects adequately requires these roles and skill sets to come together.

So let me summarize some of the goals and targets of the article, as well as talk about the relationship to promising developments:

• The main focus of the article is to convey to library administrators, managers and planners that the world of networked digital content requires new ways of thinking about developing library systems, and that there are modern software engineering methods and technologies that can support new systems development.
• It may be the case that, as in Sarah's words, the library blogosphere knows that "Of course the future is in web services" but it took years of hard lobbying and education within my organisation to convince my own library leadership - that's why I was happy to have the forum of Library Journal, to reach a wider audience.  I have to believe there are many library managers who have no idea where the technology future lies, and have only a vague notion of web services.
• If I can get one library manager to be able to ask good questions about web services and service-oriented architecture, and even (dare I dream) read one of the books I recommended with more detailed information on the topic, my goal is achieved.
• I recognize that many public libraries don't have the staff for development, and never will.  That's why I have said in the past that librarians should be scripters, not coders.  Most libraries should be using technology developed externally, not trying to do their own custom internal development, or as I said in my article, they should be technology service consumers, not necessarily producers. There are only a few libraries in the world (including mine) which have the size to have a substantial development staff.
• I work at a research library, not a public library nor an academic library.  Research libraries like mine now interact with patrons almost entirely online - our walkin traffic is basically zero.  That means we have to move into the networked information space very aggressively, or basically disappear as a presence for our users, replaced by publisher web sites.
• By moving to standard APIs, using standard modern development methods, and standardizing web services, libraries can take advantage of a much broader development community, and potential staffing pool.  How many software developers do you think know Z39.50, MARC, and SRU/SRW?  That's why we have a tiny community of library hackers trying to make things work.  Now imagine that instead your library software job poster just says "developer needs to work with standard Java tools to develop software using modern methods for standardized APIs".  Getting access to a better, broader staff base is intimately connected to moving library technology into the mainstream of software development.

Where can we look?

Maybe the DLF project on ILS APIs will help.

Maybe the OASIS effort on standardising search services will be useful.

Maybe it happens by using OpenSearch and simple REST interfaces rather than custom library protocols.

UPDATE: There is another important piece, which is about libraries reaching out and speaking the right language.  That's why you need to understand how to express things in terms of SOA, Web Services, and APIs.  There is way more innovation capacity outside your walls than you can ever get inside, even if you have the perfect IT staffing policy and budget.  From your local superpatrons, highschool CS students, and local college and university computer science departments, to, basically, every CS student in the entire world.  You can reach them with contests, with collaboration requests, with invitations to improve your systems... but here's the important part... if you speak their language.  CS people love challenges and programming, but they're not going to learn obscure library jargon and usage like OPAC, Z39.50 and database (which means something completely different in CS).  You can't say "hey, can you help us improve our OPAC because the Z39.50 doesn't federate across our databases".  They're not going to know what the f*** you're asking.  Learn the CS language, and a whole world of programmers will open up to you.

To me one of the single biggest missed opportunities is in the digital library community.  Ever year, lots of computer science groups, flush with energetic grad students, toil away and produce results that are presented at JCDL and ECDL.  And, based on my experience at ECDL 2006, they then present those results entirely to a community of other computer scientists.  Where are the librarians?  Why are you all at library conferences talking to other librarians?  Come to *CDL and ask the computer scientists to build stuff you need.  Yes, it's a difficult transition from research to production, but at least join the conversation.  ENDUPDATE

I don't have the answers and I have certainly asked again and again where people see all these frameworks and groups fitting together, with no response.

The good news is that there are lots of projects out there already - I don't think it's a case that there is no activity.  The fundamental point of my article is that for these projects, we have to use enterprise architecture, service-oriented architecture, web services / standard APIs and the whole toolkit of modern network-based standard-data software development.  Because if we don't, WE WILL BUILD SILO SYSTEMS AGAIN.

That's what I said in my IATUL presentation, and in the short accompanying paper (Scribd), and what I've been saying over and over again in this blog.

[Sidebar on Scribd: be careful browsing around this document hosting site.  Many of the profiles, profile images, and documents are unfortunately very not safe for work.  Scribd really needs to put in a moderation / adult content filtering system.]

Why do I think it's important to talk about these topics?  Because there really are lots of new developments in the library catalogue and OPAC world, including:

• Scriblio - "free, open source CMS and OPAC with faceted searching and browsing features based on WordPress"
• VuFind - "The goal of VuFind is to enable your users to search and browse through all of your library's resources by replacing the traditional OPAC"
• Evergreen Open ILS - including British Columbia Pines project - "The phased implementation of the Evergreen Open ILS for all of BC ("BC PINES") be implemented over the next 5 years. We hope that eventually all public libraries in BC will join"
• eXtensible Catalog (XC) - "an open-source online system that will unify access to traditional and digital library resources"
• CERN systems redevelopment - build a complete high-energy physics (HEP) information system with full-text, data-mining and demonstrate and deploy Web 2.0 applications in the domain of sciences

There are also some great modular browser tools out there, including

• LibX - a Firefox extension that provides direct access to your library's resources [and] an open source framework from which editions for specific libraries can be built.
• Zotero - a free, easy-to-use Firefox extension to help you collect, manage, and cite your research sources

I'm very much hoping that these developments will open libraries up to be better network participants, with a broader community of developers able to build pieces, and with standards enabling libraries with limited development capability to simply plug-and-play.

Some links via The Ten Thousand Year Blog: eXtensible Catalog open source project, VuFind released as open source software.

Previously:
June 29, 2007  Casey Bisson on Scriblio and OpenLibrary

Library Journal netConnect - Web Services and the Social Catalogue

Library Journal netConnect for July is out (July 15, 2007).  The cover theme is the "Social Catalogue".  All of the content is free online as usual.

On this theme are an article I wrote about Library Web Services, advocacy by John Blyberg about the need for open APIs ("Always Pushing Information"), and an explanation of one way to Visualize Your Catalog using Grokker, by Kate Bouman et al.

I want to thank Jay Datema for this opportunity to share these concepts with a wider audience, and in particular for his patience during the extremely long genesis of my article.

I do want to add some supplementary information. 

"web services also refers to one specific technology implementation" - by this I mean the industry Web Services stack, SOAP-based services.  This shouldn't be confused with SOA, Service-Oriented Architecture, which is a conceptual framework.  There is a short version of my sidebar on SOA in the article, but with Jay's permission, the full original version is attached below.

I did a set of supplementary bookmarks for the article, but somewhere along the line I forgot to include them in the article itself, anyway, they're available at

http://www.connotea.org/user/scilib/tag/ljwebservices

UPDATE: As it happens, Jon Udell blogged a plea today for the most basic of catalogue interfaces, a standard ISBN syntax in the URL.  Via panlibus.  ENDUPDATE

I will also try to finish the blog posting of a long-overdue review of the 4 SOA / Web Services books I recommended at the end of the article.

You can of course read more as well in my (sometimes overlapping) blog categories on Service-Oriented Architecture and Web Services.

As well, Peter Murray has been advocating for library SOA, and has set up an aggregator to gather various blog postings on the topic:

http://librarysoa.dltj.org/

Finally, here's the full sidebar that I wrote:

Library SOA

In the library world, we are often bombarded with new technology terms and ideas.

In this article, I will discuss Service-Oriented Architecture (SOA) and Web Services, providing a framework for understanding their differences and complementary nature.

In this way, it should become evident which aspects will be of enduring value, and which will change as technology evolves.

Often SOA and Web Services are used interchangeably but they are actually quite different. SOA is a methodology for software architecture and for guiding software development. The library technology landscape is littered with towering monoliths of software and teetering siloed systems. We shouldn't feel too bad about this: when catalogues and digital libraries were being developed, these were reasonable implementations. It is only as the software engineering field has evolved that we have realized the problems caused by developing closed, inflexible systems.

SOA is a way of thinking about software, of breaking it down into appropriately-sized component parts, and minimizing the dependencies between those parts. It also helps to guide your thinking by emphasizing reusability. Having reusability as a core consideration reminds the developers that their code is not ephemeral - it will almost certainly be around for a long time, which means that it is critical to make it maintainable and extendable.

Web Services are a technology that can be used to implement a Service-Oriented Architecture. They have some characteristics that mesh well with the requirements of an SOA. But Web Services are just the latest technology to come, and are sure to be replaced or added to by future developments. As well, there is a potential danger of complexity creep, as more and more WS-* standards are developed.

So the important thing is to take the concept of Web Services, that of providing a well-defined interface, an API that exposes useful capabilities or data from within your organization, and select a technology as appropriate. This may be full Web Services with SOAP, it may be a simple HTTP REST interface, or both, or neither. It’s the idea that’s important, not the particular technology. In fact, what is a service, within an SOA? It’s simply a bundle of functionality with a well-defined interface.

When developing software, regardless of methodology or technology, it is important is to have the concept of sustainability in mind. There is a balance of course between long-term benefits and short-term gain. Many will argue that too much design or planning interferes with organizational agility, that it is better to rapidly deploy something, than to tilt against the windmill of perfectly engineered software.

But the fact that we now have OPAC 2.0 and “new ILS” initiatives tells us that we need to learn the lesson of sustainable software. To put it bluntly: with the library catalogue, we slowly built a software system that no longer meets all of our needs. With mashups and quickly hacked APIs, we run the risk of rapidly building software systems that will not be able to meet our needs.

Service-Oriented Architecture is the latest, and by no means perfect, method of addressing enduring challenges of software design. It is a new way of describing and communicating the concepts behind the software you build; a way of maintaining the essential conceptual integrity needed to successfully build sustainable computer systems.

SOA is a framework that sits above the development process itself. SOA does not require a waterfall model of development. We know a failed approach is trying to do all the design up front and then, perhaps after substantial time has passed in the design phase, writing the code. In fact, I believe the best results will be obtained by iterating the architecture, that is, continuously refining the framework by going back and forth between implementing defined services, and updating service definitions based on the results. I think it is important to remember that modern software engineering is about the development of software as a conversation between design and implementation, not a one-way street.

Like any methodology, there is a risk that SOA can lead to an excess of complexity, and to trying to over determine requirements that may change quickly. Be wary of elaborate SOA processes that are better matched to giant Fortune 500 companies than to the needs of basic library systems. But with all those caveats in mind, the core of SOA is quite simple: define what functions the business wants to perform, and then break those functions down into the smallest useful pieces.

This is the concept of service granularity: identify services that are large enough to provide substantial useful functionality, without being so large that they are in danger of becoming new silos. As luck would have it, we already have a fairly good idea of basic library functions, as defined by our current catalogues. We first need to break those services out of the ILS silos, and then look at adding new capabilities beyond what the ILS could offer.

There is a possibility for us to all escape the isolation of our OPAC islands, by working together to create common library service definitions, so that we move beyond the catalogue and are interoperable right from the start. I believe this is a tremendous opportunity for the library community, because standard services are the very foundations upon which innovation is built. We look to Amazon and want to capture some of the software development energy that builds upon Amazon’s services, but we have to realize that behind the scenes, Amazon has an internal SOA that helps to guide and sustain its service framework.

SOA is, in a way, like infrastructure. We benefit enormously from the fact that electrical, water, and other services in buildings are standardized, with uniform interfaces and attributes. I don’t think many people would want to work in an environment of “agile electricity”, where every wall socket had a different interface and electrical characteristics. That basic foundation of standards is what SOA can provide to the library world.

We all know that there are successful standards efforts, and there are those that languish. This is the moment, as libraries are starting to explore the world of Web Services, for us to work together as organizations to put in place the infrastructure, the simple set of basic standards, the library Service-Oriented Architecture that will form a platform for sustainable innovation for years to come.

Yahoo Home Values - an example of combining APIs

Why do you want to have an ecosystem of services?
So that developers can do stuff like this.

Yahoo Real Estate's new Home Values page. Launched this morning, the new page combines three APIs available right here on the Developer Network with two more from Zillow, to provide a 360-degree view of what homes are worth in the neighborhood of your choice.

Yahoo! APIs In Use:

    * AJAX Maps - finds the home you're searching for, recently sold comparables, and nearby similar homes for sale.
    * Local Search - finds and displays local appraisers and customer ratings.
    * Answers Question Search - finds and displays questions and answers for the query "home value."

Zillow APIs In Use:

    * GetDeepSearchResults - finds valuations, home data, sales and tax records.
    * GetDeepComps - gets data on comparable homes in the area.

Covers homes in USA only.

library service discovery and delivery

I get the sense that Daniel Chudnov and I are talking about similar problems from different angles.
I'm talking about how to get from ideas to more standard (network) services for libraries to expose and consume, he's talking about how to ensure that all available services get jammed everywhere in a user's environment.

One Big Library - NASIG 2007 talk: A New Approach to Service Discovery and Resource Delivery

the Google Mashup Editor

Everybody's gone mashin', mashin's A-OK.

On the heels of Yahoo Pipes and Microsoft Popfly, comes Google's Mashup Editor, complete with a blog and a site to host your creations

The Google Mashup Editor's simple online interface enables you to build, test, deploy and distribute your mashup.Using familiar technologies like HTML, JavaScript and CSS, you can create mashups in minutes and test them in our sandbox, a testing ground for your mashup.

You can also take advantage of a few of our extended XML tags so that you can reuse some standard AJAX UI components -- things like editable lists, Google Maps, Date Pickers, Calendars, and ratings controls -- with one line of code. These UI elements operate on feeds and can be reused in any application

Once you've created your mashup, you can publish it using Google's infrastructure and we'll serve it for you on the subdomain googlemashups.com. You don't need to set up a server and hosting, or a database or authentication for your users -- we do it all for you. You concentrate on the user experience of your app, and we'll make sure people can access it. You can even test a new version of your application while the current one is running. Once you're ready, you can publish it with one click of a button to get your site up and live.

via Mashups made easy
via ResourceShelf

Note: it's currently in limited release, you'll have to sign up on a waiting list.

Previously:
March 2, 2007  mashup platforms and SOA

2007 IATUL and ELPUB programmes - open access and beyond

The ELPUB 2007 and IATUL 2007 programmes are up.  Both will feature topics in the areas of open access and scholarly communication.  It's interesting to me how much the technology and interests of the scholarly publishing community and the academic library community seem to be converging.

At ELPUB 2007, CISTI's Judy Best will be presenting

Challenges in the Selection, Design and Implementation of an Online Submission and Peer Review System for STM Journals

two tech positions available at OpenWetware

Jason Kelly contacted me and informed me that there are two technical positions open for your wetware at OpenWetware, one's called senior tech developer, the other senior knowledge developer

http://openwetware.org/wiki/OpenWetWare:Hiring

Perhaps of interest to those of you who bridge the boundary between librarianship and developer.

Previously:
December 20, 2005  OpenWetWare, biological research sharing wiki