Library Journal netConnect - Web Services and the Social Catalogue

Library Journal netConnect for July is out (July 15, 2007).  The cover theme is the "Social Catalogue".  All of the content is free online as usual.

On this theme are an article I wrote about Library Web Services, advocacy by John Blyberg about the need for open APIs ("Always Pushing Information"), and an explanation of one way to Visualize Your Catalog using Grokker, by Kate Bouman et al.

I want to thank Jay Datema for this opportunity to share these concepts with a wider audience, and in particular for his patience during the extremely long genesis of my article.

I do want to add some supplementary information. 

"web services also refers to one specific technology implementation" - by this I mean the industry Web Services stack, SOAP-based services.  This shouldn't be confused with SOA, Service-Oriented Architecture, which is a conceptual framework.  There is a short version of my sidebar on SOA in the article, but with Jay's permission, the full original version is attached below.

I did a set of supplementary bookmarks for the article, but somewhere along the line I forgot to include them in the article itself, anyway, they're available at

http://www.connotea.org/user/scilib/tag/ljwebservices

UPDATE: As it happens, Jon Udell blogged a plea today for the most basic of catalogue interfaces, a standard ISBN syntax in the URL.  Via panlibus.  ENDUPDATE

I will also try to finish the blog posting of a long-overdue review of the 4 SOA / Web Services books I recommended at the end of the article.

You can of course read more as well in my (sometimes overlapping) blog categories on Service-Oriented Architecture and Web Services.

As well, Peter Murray has been advocating for library SOA, and has set up an aggregator to gather various blog postings on the topic:

http://librarysoa.dltj.org/

Finally, here's the full sidebar that I wrote:

Library SOA

In the library world, we are often bombarded with new technology terms and ideas.

In this article, I will discuss Service-Oriented Architecture (SOA) and Web Services, providing a framework for understanding their differences and complementary nature.

In this way, it should become evident which aspects will be of enduring value, and which will change as technology evolves.

Often SOA and Web Services are used interchangeably but they are actually quite different. SOA is a methodology for software architecture and for guiding software development. The library technology landscape is littered with towering monoliths of software and teetering siloed systems. We shouldn't feel too bad about this: when catalogues and digital libraries were being developed, these were reasonable implementations. It is only as the software engineering field has evolved that we have realized the problems caused by developing closed, inflexible systems.

SOA is a way of thinking about software, of breaking it down into appropriately-sized component parts, and minimizing the dependencies between those parts. It also helps to guide your thinking by emphasizing reusability. Having reusability as a core consideration reminds the developers that their code is not ephemeral - it will almost certainly be around for a long time, which means that it is critical to make it maintainable and extendable.

Web Services are a technology that can be used to implement a Service-Oriented Architecture. They have some characteristics that mesh well with the requirements of an SOA. But Web Services are just the latest technology to come, and are sure to be replaced or added to by future developments. As well, there is a potential danger of complexity creep, as more and more WS-* standards are developed.

So the important thing is to take the concept of Web Services, that of providing a well-defined interface, an API that exposes useful capabilities or data from within your organization, and select a technology as appropriate. This may be full Web Services with SOAP, it may be a simple HTTP REST interface, or both, or neither. It’s the idea that’s important, not the particular technology. In fact, what is a service, within an SOA? It’s simply a bundle of functionality with a well-defined interface.

When developing software, regardless of methodology or technology, it is important is to have the concept of sustainability in mind. There is a balance of course between long-term benefits and short-term gain. Many will argue that too much design or planning interferes with organizational agility, that it is better to rapidly deploy something, than to tilt against the windmill of perfectly engineered software.

But the fact that we now have OPAC 2.0 and “new ILS” initiatives tells us that we need to learn the lesson of sustainable software. To put it bluntly: with the library catalogue, we slowly built a software system that no longer meets all of our needs. With mashups and quickly hacked APIs, we run the risk of rapidly building software systems that will not be able to meet our needs.

Service-Oriented Architecture is the latest, and by no means perfect, method of addressing enduring challenges of software design. It is a new way of describing and communicating the concepts behind the software you build; a way of maintaining the essential conceptual integrity needed to successfully build sustainable computer systems.

SOA is a framework that sits above the development process itself. SOA does not require a waterfall model of development. We know a failed approach is trying to do all the design up front and then, perhaps after substantial time has passed in the design phase, writing the code. In fact, I believe the best results will be obtained by iterating the architecture, that is, continuously refining the framework by going back and forth between implementing defined services, and updating service definitions based on the results. I think it is important to remember that modern software engineering is about the development of software as a conversation between design and implementation, not a one-way street.

Like any methodology, there is a risk that SOA can lead to an excess of complexity, and to trying to over determine requirements that may change quickly. Be wary of elaborate SOA processes that are better matched to giant Fortune 500 companies than to the needs of basic library systems. But with all those caveats in mind, the core of SOA is quite simple: define what functions the business wants to perform, and then break those functions down into the smallest useful pieces.

This is the concept of service granularity: identify services that are large enough to provide substantial useful functionality, without being so large that they are in danger of becoming new silos. As luck would have it, we already have a fairly good idea of basic library functions, as defined by our current catalogues. We first need to break those services out of the ILS silos, and then look at adding new capabilities beyond what the ILS could offer.

There is a possibility for us to all escape the isolation of our OPAC islands, by working together to create common library service definitions, so that we move beyond the catalogue and are interoperable right from the start. I believe this is a tremendous opportunity for the library community, because standard services are the very foundations upon which innovation is built. We look to Amazon and want to capture some of the software development energy that builds upon Amazon’s services, but we have to realize that behind the scenes, Amazon has an internal SOA that helps to guide and sustain its service framework.

SOA is, in a way, like infrastructure. We benefit enormously from the fact that electrical, water, and other services in buildings are standardized, with uniform interfaces and attributes. I don’t think many people would want to work in an environment of “agile electricity”, where every wall socket had a different interface and electrical characteristics. That basic foundation of standards is what SOA can provide to the library world.

We all know that there are successful standards efforts, and there are those that languish. This is the moment, as libraries are starting to explore the world of Web Services, for us to work together as organizations to put in place the infrastructure, the simple set of basic standards, the library Service-Oriented Architecture that will form a platform for sustainable innovation for years to come.

Comments

[David F. Flanders]

I think the next post we need to see (following on from this post and Eric Shnell's posts) is some pragmatic action on how tech-savey OPAC administrators can have a play with some SOA components!? Maybe a simple RoR example of what might be accomplished? Or a how-to on making xml-http calls to a lib service? Build it and they will come.