Science Library Pad

review - Out of the Box web services

NOTE: This 2007 draft blog post is being published in 2022; it may be incomplete and the information is certainly out of date.

Out of the Box by John Hagel III

# Hardcover: 240 pages
# Publisher: Harvard Business School Press (October 28, 2002)
# Language: English
# ISBN-10: 1578516803
# ISBN-13: 978-1578516803

The full title is Out of the Box: Strategies for Achieving Profits Today & Growth Tomorrow Through Web Services

The bottom line is that managers needs to develop more flexibility and collaboration capability to deliver continuing rounds of operating cost and asset savings as well as accelerated growth.  They need to escape from the high cost and inflexible infrastructures that make both cost reduction and growth such daunting challenges.

This book discusses how a new generation of IT and related architectures--known generally as Web services technology and distributed services architectures--will provide a significant catalyst in helping management break out of the boxes that confine it today.

The book is mainly targetted at managers, and talks mostly about business transformation challenges.  I liked the emphasis on bringing more flexibility to all aspects of business.  It doesn't talk about Service-Oriented Architecture directly at all, although it does talk sometimes about "Web service architectures".

If you're a manager and looking for ideas about how to transform your business, taking advantage of new technology and new IT processes, this would be a good book to read.

next generation open library system

I'm so used to bookmarking stuff into my FriendFeed these days I have to remind myself my blog is also a communications vehicle... I'd be remiss in not posting about this major SOA & libraries initiative.

The Open Library Environment (OLE) Project

With support from The Andrew W. Mellon Foundation, the Open Library Environment (OLE) Project will convene the academic library community in the design of an Open Library Management System built on Service Oriented Architecture.

http://oleproject.org/

Duke University is the lead on this project.

UPDATE: Just for clarity I should probably mention that my organisation (CISTI) is neither a core nor an advisory partner.  ENDUPDATE

If you want to get an idea of the framework that this fits into, you can see my posting about the presentation Enterprise Services Architectures - Chris Mackie - DLF Fall Forum 2007.

It is great to see library projects starting to use a service-oriented approach, although given that this is the dominant industry trend these days, I'm not convinced that my four years of online and offline advocacy for library SOA actually had much of an impact on this choice.

(You can see my blog postings in category "Service-Oriented Architecture" going back to November 2004:
http://scilib.typepad.com/science_library_pad/serviceoriented_architecture/ )

Still, one does like to be right.

implementing SOA at CISTI

My colleague Stephen Anthony has posted his author version of his peer-reviewed article for The Serials Librarian, "Implementing Service Oriented Architecture at the Canada Institute for Scientific and Technical Information".  At 14 pages, it's the most comprehensive and detailed look available at the challenges our organisation was facing, the Enterprise Architecture approach we took to help improve our technology planning, and the Service-Oriented Architecture that we are building as a result.

It's from 55 (1/2) which should show up eventually at

http://www.haworthpress.com/store/product.asp?sku=J123&detail=TOCList#TOCList

and, assuming I'm constructing the URL correctly, specifically at

http://www.haworthpress.com/store/Toc_views.asp?TOCName=J123v55n01%5FTOC&desc=Volume%3A%2055%20Issue%3A%201%2F2

Stephen reports the full citation as

Anthony, Stephen K. "Implementing Service Oriented Architecture at the Canada Institute for Scientific and Technical Information." The Serials Librarian, 55 (01-02), pp. 235 - 253.
DOI: 10.1080/03615260801970907 (doesn't yet resolve)

EA/SOA are about business process, not technology

I would claim that, by the time I left, the technical team within which I worked had a better overview of the business processes of the university than any other part of the organisation. We were able to successfully re-apply that knowledge to subsequent development, to great effect.

Paul Walk - SOA and reusable knowledge - February 26th, 2008

NLA announces Library Labs

The [National Library of Australia] has recently opened this "Library Labs" wiki space:

https://wiki.nla.gov.au/display/LABS/Home

The aim of this space is to let our colleagues know what we are doing, to invite comments, questions and feedback and to provide a space for discussion and collaboration.

We have started to redevelop our digital library services using a service-oriented architecture and open source software solutions where these are functional and robust.  We are also aiming to take a common ("single business") approach to collection management, discovery and delivery.

We are interested in forming a community of Australian business analysts and developers who are working on similar problems and who are interested in  interoperable, standards-based solutions. We are also interested in working with colleagues at an international level to provide prototypes and testbeds for new and emerging standards.

via Warwick Cathro
Assistant Director-General, Innovation
National Library of Australia

ICCBSS conference 2008 - SOA and component-based software

7th IEEE International Conference on Composition-Based Software Systems (ICCBSS) - "icecubes"
February, 25-29, 2008 - Madrid, Spain

The programme is already set, it includes NRC's Anatol Kark with SEI's Tricia Oberndorf, doing a workshop on "Research Agenda for System of Systems Interoperability" and a keynote by Alan Brown of IBM Rational Software on "Practical Approaches to Delivering Service-Oriented Solutions: The Role of Software Architects and Architecture in an SOA World".  Many of the papers/presentations cover SOA or related topics.

Enterprise Services Architectures - Chris Mackie - DLF Fall Forum 2007

I was privileged to have been paired with Chris Mackie of the Andrew W. Mellon Foundation on the DLF programme, and fortunately I think our talks were quite complementary.  Chris talked about Enterprise Services Architectures (ESA), presenting a very clear-eyed view of the sustainability challenges of IT infrastructure for higher education, and how ESA and some specific Middleware ESA (MESA) projects are addressing those challenges.  His presentation (PDF) is now available from the DLF site.

Stanford Library Cyberinfrastructure - DLF Fall Forum - Nov 7, 2007

I would really call this more a Service-Oriented Architecture to support advanced digital library activities.  Definitely some interesting work, a good contribution to library SOA.

Defining and Designing a Cyberinfrastructure for the Library of the Future
Tom Cramer, Lynn McRae, Rachel Gollub
Digital Library Systerms & Services
Stanford University

* design pattern & process

Environmental Scan at SULAIR
1. Digital preservation
2. Google Book scanning
3. Internal digitization
4. Discovery
5. Content delivery
6. Ongoing proces re-engineering

[diagram - The Digital Library: Content, Services & Infrastructure]

- content management & content infrastructure

content streams->digital library holdings->services
supported by content management & middleware plus SDR (Stanford Digital Repository)

The Challenge

Analytic Environments, Delivery Apps, Discovery Apps on top of many other components

Strategy 1: Hardware Approach - ILS (nope)
Strategy 2: Repository-centric (nope - SDR is preservation infrastructure)
Strategy 3: MacGyver It (duct tape)
Strategy 4: Library Middleware
- Content Registry
- Collections Registry
- Access Broker
supported by
Reporting, Intelligence, Monitoring
"Content & Service Middleware"

Parallel to Identity Management - in enterprise computing, we have patterns that we use

"identity management for digital content - content registry"

The process

1. Recognize need
2. Think through narratives
3. Idenity parts
- Applications
- Services
- Infrastructure
4. Assemble in an architecture
5. diversion into ESBs
6. Validate concepts & find a name

What's the Need? The Gap to fill?

what it's not
- not an ILS, nor a new catalog, nor a repository, not protocol or standard, not discovery nor delivery apps, not user-facing

what it is
* infrastructure underpinning applications
* design pattern & architecture

Narratives

1. Support for scholarly workflows
2. Personalized academic work environment
3. DIY Library/Research Environment
4. Integrated, comprehensive content discovery
5. Creating, managing, publishing dynamic digital collections
6. Extend library infrastructure & workflows to already digital content
... more

1. Scholarly workflows

Support collaboration, research and publication through a full information and creative lifecycle

[list of use cases]

extract set of capabilities from use cases

2. Personalized Academic Work Environment

Highly personalized services and resources based on persistent and intimate knowledge of the scholar's identity,
roles, background, explicity choices, and implicity preferences

* Endow all services with an awareness of personal identity, preferences and contributions

3. The DIY Library/Research Environment

Enable scholars to use the tools of their choice to incorporate library data, metadata, and complementary services
into their workspace

* APIs to content, metadata, libraryservices

4. Comprehensive content discovery

Ability to discover across all content stores

...

8. Library management, operations, reporting

internal monitoring abilties

Five themes derived from narratives
1. Identity
2. Preservation
3. Personalization
4. Access
5. Management

bound together in a common infrastructure

Services, Infrastructure

Translating function into architecutre

[diagram]

SOA on ESB

Content Management
* data/metadata stores
* data/content sources

Services -- Preservation & Management
Services -- Access/Identity/Peronalization
Infrastructure services

metadata store, "a bunch of XML files", sitting on the ESB
- including plugin service API, management services API

ESB gives you a bunch of functionality, Fedora gives you a bunch of functionality...

"Lyberstructure"

my presentation on SOA for libraries at DLF Fall Forum 2007

Overall I think it went very well, the room was packed (to my considerable surprise) and I got good questions from the audience - for once actually too many questions for my allotted time, so I unfortunately stole 10 minutes from Chris Mackie, whose presentation followed mine.

The slides are up on Slideshare, they were done in Apple Keynote, in case you're wondering.

http://www.slideshare.net/scilib/serviceoriented-architecture-for-libraries/

All of the links and background info for the presentation are available at

http://www.connotea.org/user/scilib/tag/dlf2007akerman

The following is not the actual narrative, but it gives you enough to follow the (mostly conceptual) slides:

I started with a great quote via Katherine Kott of DLF Aquifer, from the previous night's session on collaboration, stating that we need "vision translated into operational success".

Service-Oriented Architecture addresses problems in technology planning,
design and implementation.
First, we must understand what some of these problems are.
Fundamentally, technology implements the desires of the organisation.

Too often we focus on our immediate desires ("What do you want") rather
than long-term planning.
This has meant we get what we asked for (e.g. incremental catalogue
interface changes) rather than what is best for the organisation
(transformed catalogue)

In order to better align our implementations with our goals, CISTI uses
an SOA derived from our Enterprise Architecture.  In our EA methodology,
we start with business needs (Who are you?), and only then proceed to
identifying the gaps to be filled.  But you must look beyond the current
state (Why are you here?  Where are you going?)

So what are the transformational challenges we face?

In a way, we are in a fantastic world for the traditional goal of a
library - ubiquitous, zero-cost, perfect copies of information.  No more
scribes devoting their lives to copying a few books.  But this has
tremendous implications in terms of preservation, versioning, citation
etc.

Also, there are major changes in moving from a physical world where we
engineer bridges, to a virtual world where we "engineer" software.
Software doesn't have most of the constraints that apply in the physical
world, and we have to be careful with analogies to physical objects.  It
would make very little sense (except perhaps to a US Senator) to build
half a bridge, or a bridge that goes nowhere.  But in the software
world, building components, even if they are only part of an overall
goal, can be extremely useful.  And software is not location constrained
- it can be accessed from any location - there should be no "software to
nowhere".  The network is everywhere, and we need to understand how that
transforms our systems design.

So how do we apply the notions of architecture that we understand from
the physical world, and that have for centuries and continue today to
enable us to build very complex structures with many components (heat,
lighting, electrictiy, plumbing, etc.)

Well, we can use the idea of modeling.  Just as we make models of
physical objects, we can make models of business processes.  But we must
ensure that we don't get stuck in our models or frameworks, and do move
to implementation at some point.

CISTI has found that its methodology, which is focused on getting to the
implementation framework, and beyond that to design and development, has
been invaluable for turning business ideas into implemented software.
We are fortunate in that we are a library with substantial in-house
technology resources, including a dedicated architecture team (the only
library EA team in the world??)

SIDEBAR: I met the Head of eArchitecture for the British Library at my presentation, so now I think I should say "one of the very few library EA teams in the world".  END SIDEBAR

This structure may be different for
academic organisations, with faculty groups that have departmental IT,
central IT, and academic consortia.

Within this larger mehtodological framework, SOA is an approach to
identifying "servicifiable" functions in the organisation, and building
larger service offerings from various combinations of services.

In the architecture analysis phase, CISTI models all of our
technology-related activities, at a high-level this model describes all
the technology-supported activities, with functions such as "provide
library service" that then decompose down into many many lower-level
models.

Recently we have been specifically modeling a Trusted Digital
Repository, using the OAIS model and converting and adapting it.

This EA modeling work then leads to the derivation of SOA services
[depending on audience profile I may talk a lot on this slide about the
details of SOA, or not].

CISTI has used and is using SOA successfully as the foundation of
various projects - this is not just theory.

SOA is also not an agility-killer bureaucracy - services actually
provide pieces that can be used for small experiments, while
simultaneously moving the organisation in its stated general direction.

Ideally, we should all be collaborating on SOA, rather than building SOA
silos.  I have blogged a lot about this, but with little response.  Are
the individual framework groups only talking amongst themselves?  How
can we use their efforts most effectively?

SOA is certainly being explored in many areas of the library space,
including various digital library initiatives and modeling efforts (some
of which will be presented in other forum sessions).

SOA also offers the possibility of improving the catalogue by breaking
it into component services, and enabling the sort of simple layers
(widgets, mashups, javascripty stuff) by providing sustainable,
well-architected underlying service capabilities.

For academic libraries in particular, SOA offers a compelling
opportunity to connect with existing and future scientific activities,
as many Cyberinfrastructure projects have SOA at their cores.  Can we
use modelling at even the highest level (the entire scientific
communication cycle) to guide our service implementations?

Ultimately to build sustainable, non-siloed, modern information systems
to support scholars and citizens worldwide, we need to build many
bridges (this one happens to be built of lego components, in case you're
worried that giant ducks have invaded the US).  Between libraries and
scientists, between technology and business, between funders and
software developers, between computer science and library science...

We need to move beyond frameworks, using solid governance to ensure that
SOA plans become running SOA code, while avoiding getting stuck in giant
projects that don't deliver.  As well, even if you don't have the scale
or the capacity to build your own SOA, you can still participate as a
service consumer.

I'm not crazy in this (at least I hope not), as evidence for which I
present some starting points about SOA in the academic and library
sectors.

Questions...

The End

SIDEBAR 2: In case you're wondering, there's a slight science fiction theme running through the presentation, inspired by the "federation" part of the organisation name.  The Four Questions are from Babylon 5, the goodish aliens, the Vorlons, ask "Who Are You?" and represent the forces of order, the evilish aliens, the Shadows, ask "What Do You Want?" and represent the forces of chaos.  "There Are Many Copies" is from the opening sequence in the new Battlestar Galactica, using the great Creative Commons images I found worked out as a nice way of avoiding using a screencapture (plus using a screencap might have been a bit distracting as it shows a scene from BG where the many copies of Sharon Valerii/Number Eight are not so much wearing any clothes).  I had a better closing quote but I couldn't remember it, so I used a tag line from Buckaroo Banzai Across the 8th Dimension, to close out the "where are you going" questions theme.  END SIDEBAR 2

SIDEBAR 3: I was actually amazed to be able to locate the "bridge to nowhere" Google Earth image.  I had seen the unfinished highway bridge out of the airplane window as the plane was coming in to land in Philadelphia and by the time I thought to GPS-mark it or take a photo it was already past.  Incredibly, doing an image search on

"bridge to nowhere" Pennsylvania

brought it up as the first hit - Google Search, Google Earth and the Internet continue to amaze me. END SIDEBAR 3

SIDEBAR 4: I bought my carbon offsets from MyClimate.org

flight from: Ottawa, ON [Macdonald-Cartier International Airport], Canada, YOW
flight to: Philadelphia, PA [Philadelphia International Airport], USA, PHL
flight via: Toronto, ON [Lester B. Pearson International Airport], Canada, YYZ
return, economy
flight distance: 1'845 km
flight passengers: 1

CO2 Emissions: 0,511 t

Total costs for compensation of your flight: SFr. 20.00

According to Google

20 Swiss francs = 16.3543582 Canadian dollars

SOA and escience info via EDUCAUSE

Ran across a couple items on Service-Oriented Architecture provided by EDUCAUSE:

"Service-Oriented Architecture—What Is It, and How Do We Get One?" by Jim Phelps and Brian Busby in EDUCAUSE Quarterly, Vol. 30, No. 3, 2007

And from the Seminars on Academic Computing (SAC) 2007 (August 3-8, 2007)

Session on Service-Oriented Architecture
Charles F. Leonhardt (Georgetown University), Piet Niederhausen (Georgetown University), and Jens Haeusser (The University of British Columbia) - three PowerPoint presentations plus a podcast

The Institutional Challenges of Cyberinfrastructure and E-Research
Clifford A. Lynch - podcast

There is also an EDUCAUSE tag view you can follow:

http://connect.educause.edu/term_view/Service-oriented%2BArchitecture

Previously:
January 30, 2007  report on [EDUCAUSE] Library Future Roles webcast

Birkin Diana on lightweight SOA

A nice presentation on lightweight SOA (PDF).

tutorial on SOA and Web Services from ICEC06

Several NRC researchers presented a Tutorial on Service Oriented Architecture: (Semantic) Web Services, Business Process Modeling, Software Engineering last year at ICEC2006, The Eighth International Conference on Electronic Commerce.

Slides:

Introduction

Web Services

Formal Methods for Web Services Process Modelling

Semantic Web Services

Wrap-up: Semantic Service Computing (SSC) project in NRC

Globe & Mail on SOA

In a nutshell, SOA helps developers build new business applications faster, easier and cheaper by assembling them Tinkertoy-style from stand-alone software services that do things like order supplies or issue invoices.

...

SOA first emerged seven or eight years ago as the next big thing in application development, and a way to help companies become more agile and flexible. But it failed to catch on, partly because vendors found it difficult to get across to business managers why SOA was important.

According to a recent survey by research firm IDC Canada Ltd., about 40 per cent of Canadian companies have begun to invest in SOA or plan to in the next year, which puts them well behind U.S. competitors. And most are only using it in limited ways, says David Senf, IDC Canada program director for security and software research.

Globe and Mail - Unlocking the potential of SOA - October 4, 2007 (print edition)

I also discovered they had a fairly good article in July, I think it was online-only: Demystifying SOA.

SOA for government in Canada

Bill St. Arnaud points me to

Blocks of SOA: Building services with common symbols

"The cabinet system we have in Canada is built on the intentional design of silos," says Turner. "Each set of programs is enabled by legislation and statutes, and each organization supports a minister who's accountable to Parliament for that set of programs. Natural silos are designed into the system, so when you try to apply common technology across departments, it goes against the grain. It almost comes down to, do you want efficiency or accountability?"

The tendency to treat SOA as an IT project rather than a strategic architecture is a consequence of these silos, says Kuhbock. At a recent presentation he made to government IT officials, many of them said they had the same central issue: "They can do SOA in their own compartmentalized area, but they can't get it broadly driven; so they're just stuck in their silos again. It's like architecting a town, district by district, without thinking about how the whole town will interact, or mapping out the whole community."

On the other hand, I'm not sure that trying to do a government-wide SOA wouldn't get us tangled in techno-bureaucracy beyond belief.  I think the most value comes in getting individual departments and organisations to use the same SOA terminology and similar methodologies, so that we can interface well at our edges.  SOA has a lot of value, but I don't think you could successfully use it to create a Grand Unified Theory of Government IT.  My organisation sells subscriptions worldwide to scientific journals, and Revenue Canada (or whatever it's called these days) collects taxes from Canadian citizens - there are not exactly huge IT overlaps in these two business activities.

software development, staffing and new library technology

Librarian in Black responded to my article in Library Journal netConnect, and Richard Wallis' commentary thereon.  She says

What makes me sad is that both Ackerman [sic] and Wallis have missed a key point: if the future is in web services, how can libraries take advantage of that with their current staff configurations?  How many libraries in the U.S. have a honest-to-goodness computer programmer on staff?  How many have staff with Computer Science degrees?  How many staff do they have devoted to the library's hardware, software, and network?  How many staff do they have devoted to web services?

In the smallest libraries, perhaps all of these are the same one person.

My article is about modern software engineering for libraries, not on how to staff them.  However, in the very same issue of netConnect there's an article by Karen Coombs "Digital Promise and Peril" calling for library staffing to reflect the digital content environment

Many of these digital materials are in jeopardy of being lost because librarians have not yet found adequate ways to collect and manage them. In part, this is because roles and skill sets have been siloed in libraries. Materials preservation issues have typically been the purview of special collections and archives units within the library. In contrast, cataloging expertise has resided in technical services, and technology expertise has typically resided in systems. To collect and manage born digital objects adequately requires these roles and skill sets to come together.

So let me summarize some of the goals and targets of the article, as well as talk about the relationship to promising developments:

• The main focus of the article is to convey to library administrators, managers and planners that the world of networked digital content requires new ways of thinking about developing library systems, and that there are modern software engineering methods and technologies that can support new systems development.
• It may be the case that, as in Sarah's words, the library blogosphere knows that "Of course the future is in web services" but it took years of hard lobbying and education within my organisation to convince my own library leadership - that's why I was happy to have the forum of Library Journal, to reach a wider audience.  I have to believe there are many library managers who have no idea where the technology future lies, and have only a vague notion of web services.
• If I can get one library manager to be able to ask good questions about web services and service-oriented architecture, and even (dare I dream) read one of the books I recommended with more detailed information on the topic, my goal is achieved.
• I recognize that many public libraries don't have the staff for development, and never will.  That's why I have said in the past that librarians should be scripters, not coders.  Most libraries should be using technology developed externally, not trying to do their own custom internal development, or as I said in my article, they should be technology service consumers, not necessarily producers. There are only a few libraries in the world (including mine) which have the size to have a substantial development staff.
• I work at a research library, not a public library nor an academic library.  Research libraries like mine now interact with patrons almost entirely online - our walkin traffic is basically zero.  That means we have to move into the networked information space very aggressively, or basically disappear as a presence for our users, replaced by publisher web sites.
• By moving to standard APIs, using standard modern development methods, and standardizing web services, libraries can take advantage of a much broader development community, and potential staffing pool.  How many software developers do you think know Z39.50, MARC, and SRU/SRW?  That's why we have a tiny community of library hackers trying to make things work.  Now imagine that instead your library software job poster just says "developer needs to work with standard Java tools to develop software using modern methods for standardized APIs".  Getting access to a better, broader staff base is intimately connected to moving library technology into the mainstream of software development.

Where can we look?

Maybe the DLF project on ILS APIs will help.

Maybe the OASIS effort on standardising search services will be useful.

Maybe it happens by using OpenSearch and simple REST interfaces rather than custom library protocols.

UPDATE: There is another important piece, which is about libraries reaching out and speaking the right language.  That's why you need to understand how to express things in terms of SOA, Web Services, and APIs.  There is way more innovation capacity outside your walls than you can ever get inside, even if you have the perfect IT staffing policy and budget.  From your local superpatrons, highschool CS students, and local college and university computer science departments, to, basically, every CS student in the entire world.  You can reach them with contests, with collaboration requests, with invitations to improve your systems... but here's the important part... if you speak their language.  CS people love challenges and programming, but they're not going to learn obscure library jargon and usage like OPAC, Z39.50 and database (which means something completely different in CS).  You can't say "hey, can you help us improve our OPAC because the Z39.50 doesn't federate across our databases".  They're not going to know what the f*** you're asking.  Learn the CS language, and a whole world of programmers will open up to you.

To me one of the single biggest missed opportunities is in the digital library community.  Ever year, lots of computer science groups, flush with energetic grad students, toil away and produce results that are presented at JCDL and ECDL.  And, based on my experience at ECDL 2006, they then present those results entirely to a community of other computer scientists.  Where are the librarians?  Why are you all at library conferences talking to other librarians?  Come to *CDL and ask the computer scientists to build stuff you need.  Yes, it's a difficult transition from research to production, but at least join the conversation.  ENDUPDATE

I don't have the answers and I have certainly asked again and again where people see all these frameworks and groups fitting together, with no response.

The good news is that there are lots of projects out there already - I don't think it's a case that there is no activity.  The fundamental point of my article is that for these projects, we have to use enterprise architecture, service-oriented architecture, web services / standard APIs and the whole toolkit of modern network-based standard-data software development.  Because if we don't, WE WILL BUILD SILO SYSTEMS AGAIN.

That's what I said in my IATUL presentation, and in the short accompanying paper (Scribd), and what I've been saying over and over again in this blog.

[Sidebar on Scribd: be careful browsing around this document hosting site.  Many of the profiles, profile images, and documents are unfortunately very not safe for work.  Scribd really needs to put in a moderation / adult content filtering system.]

Why do I think it's important to talk about these topics?  Because there really are lots of new developments in the library catalogue and OPAC world, including:

• Scriblio - "free, open source CMS and OPAC with faceted searching and browsing features based on WordPress"
• VuFind - "The goal of VuFind is to enable your users to search and browse through all of your library's resources by replacing the traditional OPAC"
• Evergreen Open ILS - including British Columbia Pines project - "The phased implementation of the Evergreen Open ILS for all of BC ("BC PINES") be implemented over the next 5 years. We hope that eventually all public libraries in BC will join"
• eXtensible Catalog (XC) - "an open-source online system that will unify access to traditional and digital library resources"
• CERN systems redevelopment - build a complete high-energy physics (HEP) information system with full-text, data-mining and demonstrate and deploy Web 2.0 applications in the domain of sciences

There are also some great modular browser tools out there, including

• LibX - a Firefox extension that provides direct access to your library's resources [and] an open source framework from which editions for specific libraries can be built.
• Zotero - a free, easy-to-use Firefox extension to help you collect, manage, and cite your research sources

I'm very much hoping that these developments will open libraries up to be better network participants, with a broader community of developers able to build pieces, and with standards enabling libraries with limited development capability to simply plug-and-play.

Some links via The Ten Thousand Year Blog: eXtensible Catalog open source project, VuFind released as open source software.

Previously:
June 29, 2007  Casey Bisson on Scriblio and OpenLibrary

Library Journal netConnect - Web Services and the Social Catalogue

Library Journal netConnect for July is out (July 15, 2007).  The cover theme is the "Social Catalogue".  All of the content is free online as usual.

On this theme are an article I wrote about Library Web Services, advocacy by John Blyberg about the need for open APIs ("Always Pushing Information"), and an explanation of one way to Visualize Your Catalog using Grokker, by Kate Bouman et al.

I want to thank Jay Datema for this opportunity to share these concepts with a wider audience, and in particular for his patience during the extremely long genesis of my article.

I do want to add some supplementary information. 

"web services also refers to one specific technology implementation" - by this I mean the industry Web Services stack, SOAP-based services.  This shouldn't be confused with SOA, Service-Oriented Architecture, which is a conceptual framework.  There is a short version of my sidebar on SOA in the article, but with Jay's permission, the full original version is attached below.

I did a set of supplementary bookmarks for the article, but somewhere along the line I forgot to include them in the article itself, anyway, they're available at

http://www.connotea.org/user/scilib/tag/ljwebservices

UPDATE: As it happens, Jon Udell blogged a plea today for the most basic of catalogue interfaces, a standard ISBN syntax in the URL.  Via panlibus.  ENDUPDATE

I will also try to finish the blog posting of a long-overdue review of the 4 SOA / Web Services books I recommended at the end of the article.

You can of course read more as well in my (sometimes overlapping) blog categories on Service-Oriented Architecture and Web Services.

As well, Peter Murray has been advocating for library SOA, and has set up an aggregator to gather various blog postings on the topic:

http://librarysoa.dltj.org/

Finally, here's the full sidebar that I wrote:

Library SOA

In the library world, we are often bombarded with new technology terms and ideas.

In this article, I will discuss Service-Oriented Architecture (SOA) and Web Services, providing a framework for understanding their differences and complementary nature.

In this way, it should become evident which aspects will be of enduring value, and which will change as technology evolves.

Often SOA and Web Services are used interchangeably but they are actually quite different. SOA is a methodology for software architecture and for guiding software development. The library technology landscape is littered with towering monoliths of software and teetering siloed systems. We shouldn't feel too bad about this: when catalogues and digital libraries were being developed, these were reasonable implementations. It is only as the software engineering field has evolved that we have realized the problems caused by developing closed, inflexible systems.

SOA is a way of thinking about software, of breaking it down into appropriately-sized component parts, and minimizing the dependencies between those parts. It also helps to guide your thinking by emphasizing reusability. Having reusability as a core consideration reminds the developers that their code is not ephemeral - it will almost certainly be around for a long time, which means that it is critical to make it maintainable and extendable.

Web Services are a technology that can be used to implement a Service-Oriented Architecture. They have some characteristics that mesh well with the requirements of an SOA. But Web Services are just the latest technology to come, and are sure to be replaced or added to by future developments. As well, there is a potential danger of complexity creep, as more and more WS-* standards are developed.

So the important thing is to take the concept of Web Services, that of providing a well-defined interface, an API that exposes useful capabilities or data from within your organization, and select a technology as appropriate. This may be full Web Services with SOAP, it may be a simple HTTP REST interface, or both, or neither. It’s the idea that’s important, not the particular technology. In fact, what is a service, within an SOA? It’s simply a bundle of functionality with a well-defined interface.

When developing software, regardless of methodology or technology, it is important is to have the concept of sustainability in mind. There is a balance of course between long-term benefits and short-term gain. Many will argue that too much design or planning interferes with organizational agility, that it is better to rapidly deploy something, than to tilt against the windmill of perfectly engineered software.

But the fact that we now have OPAC 2.0 and “new ILS” initiatives tells us that we need to learn the lesson of sustainable software. To put it bluntly: with the library catalogue, we slowly built a software system that no longer meets all of our needs. With mashups and quickly hacked APIs, we run the risk of rapidly building software systems that will not be able to meet our needs.

Service-Oriented Architecture is the latest, and by no means perfect, method of addressing enduring challenges of software design. It is a new way of describing and communicating the concepts behind the software you build; a way of maintaining the essential conceptual integrity needed to successfully build sustainable computer systems.

SOA is a framework that sits above the development process itself. SOA does not require a waterfall model of development. We know a failed approach is trying to do all the design up front and then, perhaps after substantial time has passed in the design phase, writing the code. In fact, I believe the best results will be obtained by iterating the architecture, that is, continuously refining the framework by going back and forth between implementing defined services, and updating service definitions based on the results. I think it is important to remember that modern software engineering is about the development of software as a conversation between design and implementation, not a one-way street.

Like any methodology, there is a risk that SOA can lead to an excess of complexity, and to trying to over determine requirements that may change quickly. Be wary of elaborate SOA processes that are better matched to giant Fortune 500 companies than to the needs of basic library systems. But with all those caveats in mind, the core of SOA is quite simple: define what functions the business wants to perform, and then break those functions down into the smallest useful pieces.

This is the concept of service granularity: identify services that are large enough to provide substantial useful functionality, without being so large that they are in danger of becoming new silos. As luck would have it, we already have a fairly good idea of basic library functions, as defined by our current catalogues. We first need to break those services out of the ILS silos, and then look at adding new capabilities beyond what the ILS could offer.

There is a possibility for us to all escape the isolation of our OPAC islands, by working together to create common library service definitions, so that we move beyond the catalogue and are interoperable right from the start. I believe this is a tremendous opportunity for the library community, because standard services are the very foundations upon which innovation is built. We look to Amazon and want to capture some of the software development energy that builds upon Amazon’s services, but we have to realize that behind the scenes, Amazon has an internal SOA that helps to guide and sustain its service framework.

SOA is, in a way, like infrastructure. We benefit enormously from the fact that electrical, water, and other services in buildings are standardized, with uniform interfaces and attributes. I don’t think many people would want to work in an environment of “agile electricity”, where every wall socket had a different interface and electrical characteristics. That basic foundation of standards is what SOA can provide to the library world.

We all know that there are successful standards efforts, and there are those that languish. This is the moment, as libraries are starting to explore the world of Web Services, for us to work together as organizations to put in place the infrastructure, the simple set of basic standards, the library Service-Oriented Architecture that will form a platform for sustainable innovation for years to come.

SOA difference

I've often said that if SOA is about technology then its another minor IT improvement that won't deliver a fraction of the benefit that the hype promises.

So why do I think that SOA is different from the way most people architect IT solutions and why do I think it will have an impact?

Because simply put I think that SOA is about changing the way people think, its about making them think about Business Services and being focused on business objectives first, and technology objectives second.

Service Architecture - Why SOA is different

Library SOA for catalogues, escience and more - Richard Akerman - June 12 - IATUL 2007

I have posted my presentation to SlideShare

http://www.slideshare.net/scilib/library-serviceoriented-architecture-to-enhance-access-to-science

and you can also download the PowerPoint from there (thanks to Amazon S3 storage via SlideShare).

All of the bookmarks in the presentation along with some supplementary links are available at

http://www.connotea.org/user/scilib/tag/iatul2007akerman

I was pleased with the response to my presentation.  There was interest from people in many different roles and organisations, and I am becoming increasingly convinced that at least for national library scale IT projects, SOA is gaining increasing traction.

I will use this posting as a starting point for discussions and any further ideas I may think to add.

UPDATE 2007-09-06: My paper is available in E-LIS.

audio interview with Peter Brantley about library service collaboration

a 19 minute [audio] interview with Peter Brantley, new Executive Director of the Digital Library Federation

via EDUCAUSE Connect - gbayne's blog - An Interview with Peter Brantley at CNI's 2007 Spring Task Force Meeting

via ResourceShelf

The main topics as I heard them:

• the challenge of getting libraries to collaborate deeply and meaningfully on technology
• determining what kinds of technology services (network-level services) that libraries can usefully contribute to an already-rich discovery environment
• how to put those kinds of the library collaborations together in a way that includes the ability to sustain and support the new technology services
• the challenge of shifting staff to support new areas of activity

I wonder if I've been using the wrong terminology all this time.  I've been talking and talking about library Service-Oriented Architecture with (as far as I can tell), very little uptake.  And I will be talking about it more in an upcoming presentation and an article in Library Journal, as well one of my colleagues has an article upcoming in a different journal.  It is hard to sustain one's enthusiasm indefinitely however, in the absence of much response.

We already have framework upon framework, but I can't seem to get anyone to talk about how they might inter-relate.  Are there communication barriers?  Terminology barriers?  Resource barriers?  Culture barriers?

I wonder whether we shouldn't really be using language more like "library technology service collaboration" or "library network-level enhancement collaboration".  I think maybe when we say "digital library" or SOA, people say, "oh, that's not me".  But it is.  We're all digital libraries now.  What the whole collaborative library SOA idea is about is more libraries producing and consuming more network services, so that we can all better participate in the online experience.

Previously:
January 30, 2007  report on Library Future Roles webcast
September 26, 2005  Info Grid 2005 - Monday 26th - Deploying Services, Not Libraries

paper on library SOA and the BIBSYS system

Moving towards a service-oriented architecture (abstract) / Erlend Gutteberg (BIBSYS, Trondheim)

from European Library Automation Group - ELAG 2007: Library 2.0 - Papers and Speakers

BIBSYS "supplies library and information systems to over 100 libraries and institutions of higher education in Norway".

presenting library soa at IATUL 2007

I am pleased to announce that I will be presenting my paper "Library Service-Oriented Architecture to Enhance Access to Science" at IATUL 2007, my session is tentatively scheduled for

Tuesday June 12th 2007 at 10 am
The presentation will be about 20 minutes plus 5 minutes for questions.

UPDATE 2007-06-12: The presentation has now been completed and is available online.  ENDUPDATE

My paper is a brief summary of CISTI's EA and SOA success, based on the efforts of the rest of the architecture and application development teams.  I intend it both as a demonstration of what is possible, as well as a discussion starter, to help other libraries move beyond basic theoretical frameworks to start building a richer network of library machine-to-machine data services that we can all use for applications, websites, mashups, and whatever is coming next.

StatsCan conf on EA and SOA

John pointed out to me the Stats Canada Information Technology (IT) Conference 2007, April 4-5, 2007, Ottawa.

The conference is entirely about Enterprise Architecture and Service-Oriented Architecture,
including speakers from inside and outside of the Canadian government.

http://www.statcan.ca/english/conferences/it2007/

http://www.statcan.ca/english/conferences/it2007/all_abstracts_E.pdf

Among the new approaches, enterprise architecture has the objective of building a broad consensus within the IT community and among users. Having emerged in recent years as a central concept in the engineering of information systems, this architecture approach appears to result in a better organization of applications into discrete services.

Service Oriented Architecture (SOA) promotes an approach where automation logic is partitioned into individual units (services) that are highly interoperable, but generic enough to fulfill both immediate and future automation requirements. The major benefits associated with a successful adoption of SOA typically begin with improving an organization’s ability to share data across disparate systems and establishing an architectural model that is increasingly independent of the enabling technologies. Services are deliberately positioned as highly reusable IT tools that can be exploited over a long period of time. The end result is an increase in an organization’s overall business responsiveness (agility).

pondering SOA the UK and Aussie way

Service-Oriented Architecture (SOA) is not for the faint of heart, nor for those weak in technology expertise. 
SOA is fundamentally a methodology for systems design, informed by an overall Enterprise Architecture (EA).

If you don't have the scale of organization, or the organizational technical expertise, to sustain an EA and advanced development group, then SOA may not be for you.  You may want to focus more on consuming services and connecting services together.

Or in other words, we just have the latest in a line of IT best practices:

• If you're big enough to have an extensive software development shop, use SOA as your guiding methodology.  If you have a smaller shop, you may use or develop services defined by some other group, in their SOA.
• If you're big enough to have substantial IT infrastructure, use EA to guide the planning for that infrastructure.

Keeping that in mind, the library environment as a whole is clearly of the scale to benefit from an SOA, it's just a matter of how we put it all together.  Who defines the services?  Who will the service providers be?  What will the business model be for service consumers?

All this by way of introducing some recent reports and presentations on SOA, as well as signalling that I will be providing more of my library SOA thoughts in a few upcoming venues.

Firstly the National Library of Australia IT Architecture Project Report (PDF) has gotten some attention for its prominent mention of SOA.  (Thanks to Lorcan for pointing it out to me, DLTJ also mentioned it today.)

Second, via Brian Kelly I see that there were recent JISC conference presentations on EA and SOA.

e-Framework, SOA and EA: Using the e-Framework, Service Orientated and Enterprise architectures to develop more flexible and better aligned ICT infrastructures  Abstract

• Session commentary
• Presentation "The e-Framework, SOA & Enterprise Architecture" - JISC (PowerPoint)
• Presentation "Enterprise Architecture" - Open Group (PowerPoint)

Overall I think both SOA and EA are challenging topics to present simply.  That being said, I think both are typically presented far over to the extreme of complexity.  What was it that the rather inaccurately titled SOA for Dummies said you had to have in your SOA?  ESB, registry, broker, supervisor, BPM, yadda yadda... It's simply not true.  That's what you'll get if you let the kindly vendor hold you by the hand and take you into the twisty SOA maze.  All you actually need for SOA is to make sure you're using some basic software design principles:

autonomous, loosely-coupled and coarse-grained services with well-defined interfaces that support

the flexible reuse of application logic through composition

(The above an extraction from my 2005 presentation "Service-Oriented Architecture Methods to Develop Networked Library Services" which I am uploading to Slideshare... now.)

The same for EA.  Zachman is all well and good, but you need to be a mega-corp to fill in all the Zachman boxes, and even then the methodology doesn't help you much in translating that knowledge into action.  CISTI uses a much simpler and more practical methodology (based on InfoMajic) that focuses on translating business needs into an implementation framework.

There is lots of work and complexity in doing feasible EA and SOA without making it even more complex.

This may be why most of the published outputs available for library SOA are currently at such a very very high framework level.  I have to say, these frameworks don't do much for me.  As a technology person, all I see is some Dewey Decimal system for services, a top-level set of categories.  To actually start doing any service implementation work, you need to go much deeper, with models of how the detailed business functions inter-connect and inter-relate.

I wish I knew how to take the overall library community to that next level.  e-Framework (JISC - DEST - NZ MoE - SURF) is a piece, DLF Services Framework is a piece, Knowledge Exchange (JISC - DEFF - DFG - SURF) http://www.knowledge-exchange.info/ is as well... Maybe it's simply impossible given the scales of organizations involved to go beyond the highest-level conceptions.

All I know is we at CISTI can think of some relatively easy Web Service wins within an overall SOA - copyright / pricing services, query services - there are lots of possibilities.  Until we start getting more service providers and consumers up, we're all sort of stuck.

What do you think?  Is SOA too abstract or unachievable for your organization?  Are the service frameworks too high-level to impact on your development work?  Would you actually consume some library Web Services if more were available (keeping in mind the service interface might be simpler than full SOAP).  Are there services you're already providing that aren't being consumed?  Are the digital library people producing services that aren't being used by the rest of the library?  The Fedora repository already has Web Services, Amazon and Google already do - where do you see libraries fitting in?

Lots of questions... any answers?

I feel that I'm kind of reaching the limits of what I can do in terms of fairly abstract SOA evangelism - and I'm hoping there may be some concrete initiatives around which some services can coalesce...

SRU/SRW and library Web Services webibliography

Des Bibliothèques 2.0 blog has an extensive list of SRU/SRW and library Web Services articles, postings and documents (including some from Science Library Pad) with a brief explanation of each item in French:

SRU/SRW : Webographie / Webography

mashup platforms and SOA

Most businesses have yet to tap the potential of mashup technology. It requires an IT department that's bought into the service-oriented architecture approach, so that it's simple enough to mix, match, and change the data sources they're delivering to users in the form of mashups, says Jason Bloomberg, an analyst at ZapThink. Some sort of governance is needed to dictate who can create mashups, who has access to the data collection tools, and what mashups can contain. And governance requires technology investment in identity management systems to regulate data access.

InformationWeek - Mashups Make Their Way Into Business Apps--Slowly - February 17, 2007

Some related mashup platform sites:

IBM QEDWiki - "QEDWiki is a browser-based assembly canvas used to create simple mash-ups."

Yahoo Pipes

Microsoft Connected Services Sandbox - with a particular focus on telecom services
There is a Developers Contest, prizes are reported to range from $2500 to $25,000.