Science Library Pad

undersea cyberinfrastructure

Some interesting Canadian science sensor workflow achievements, albeit with a typically complex IBM solution

In September 2005, [VENUS] Data Management Archive System (DMAS) was awarded a grant from the CANARIE Intelligent Infrastructure Program (CIIP) for a project called: “Toward a Service Oriented Architecture and Workflow Management for VENUS and NEPTUNE”. The goals of this project were to provide these two high profile Canadian Cabled Ocean Observatories with an integrated scientific instruments management system, the capability to deliver event information to users, as well as integrated access to distributed compute and data resources through the use of innovative technologies.

The initial draft of the Service-Oriented Architecture (SOA) was proposed by IBM Canada, one of the largest R&D investors in Canada. This architecture was later refined by the DMAS team. It is worth mentioning that this is also the approach taken by Alcatel for the interaction with the NEPTUNE backbone.

This style of information systems architecture enables the NEPTUNE Canada and VENUS DMAS to be built by combining loosely coupled and interoperable services. The IBM Enterprise Service Bus (ESB) with its underlying message system WebSphere MQ provides the features such as point-to-point data delivery and message publish-and subscribe to implement the DMAS SOA.

Bill St. Arnaud - Cyber-infrastructure for undersea instruments and networks - in CAnet News - February 20, 2007

SOA video from e-Framework

A slick video (animation) explaining Service-Oriented Architecture from a university and e-Science perspective.  Available in QuickTime or Windows Media, or you can read just the raw transcript.

Introducing the Service Oriented Approach

near the end it talks about the myGrid scientific workflow project.

via David Flanders

Just to revisit a theme that continues to concern me: are major library-related IT architecture groups talking to one another?  Should they be?  We learned not to build siloed systems, are we building siloed architectures?

Initiatives I am aware of include:

• e-Framework for Education and Research
• Digital Library Federation Services Framework
• Rethinking Resource Sharing Policies Committee (buried as a subcommittee of a subcommittee, and password-protected, in typical ALA fashion)
• DEFF XML Web Services (Danish library Web Services)
• Education Commons - SOA in Higher Education
• Fedora Web Services and workflow
• DELOS Digital Library Architecture
• lots of work every year presented at the various Digital Library Conferences
• various grid and e-Science projects
• UPDATE 2007-06-02: California Digital Library (CDL) Common Framework ENDUPDATE

Previously:
December 20, 2006  my presentation on SOA and BPEL for Access 2006
July 16, 2006  library Web Services - are we making any progress

OLA 2007 - next generation ILS presentations

Ontario Library Association (OLA) Superconference 2007 presentations are up.

I haven't checked many of them, but I do highly recommend

ILS, The Next Generation: Modularity and Outward Integration (PowerPoint)

and

The Changing Nature of the Catalogue and its Integration with Other Discovery Systems (PowerPoint)

both by Karen Calhoun

UPDATE 2007-02-13: Since I find myself accidentally in the position of "recommender" (thanks LISnews), here are a couple more that I liked

* Perceptions and Expectations of the [Public] Library Brand (PDF)

An interesting examination of the challenge in extending the perception of the public library beyond books.  I liked this slide:

The term "library" cannot [be] extended to include services other than the circulation of books when there is little public trust in the brand.

The good news is that many public libraries have successfully branded themselves as knowledge providers, going far beyond books.

* Lastly, the obligatory blogwikiflickr presentation, although in 2007 I feel this is a bit like being in 1967 and talking about the marvel of the internal combustion engine, aren't blogs already retro?  Aren't we beyond the "blogs are exciting and new, come aboard, we're expecting you" stage?

Web Tools to Inspire Learning (PowerPoint)

standard IT architects

IT architect is one of the hottest jobs in tech right now, thanks in part to more companies trying to build service-oriented architectures. The problem: Not everyone agrees about what IT architects are supposed to do or know.

...

IT standards consortium Open Group is launching the industry's first professional association for IT architects. Called the Association of Open Group Enterprise Architects [AOGEA], it's looking to "elevate" the profession by promoting certification and establishing ethics standards and architecture principles.

...
A shortage of IT pros who can guide SOA development is one of the biggest hurdles for SOA adoption. Analyst firm ZapThink and staffing company Excel, seeing a market in that shortage, have launched the Architect Resource Center, a consulting service

from InformationWeek - IT Architects In High Demand - January 29, 2007

I'm all for architecture principles and standards.  I think certification is bullsh-t though (not just for IT architecture, for any kind of IT).  And if you can't figure out ethics on your own, no Association is going to help you.  AOGEA seems to have a needlessly complex structure based around certification levels.  Count me out for the time being.  Err, although the conference in Paris looks nice...

For the record, my title is Technology Architect - IT, but since that is rarely meaningful to people, I usually tell them I'm a technology planner (or if I'm feeling grand, an enterprise strategic technology planner).

report on Library Future Roles webcast

I liked the EDUCAUSE Webcast Architectures for Collaboration—Roles and Expectations for Digital Libraries.  Peter Brantley (currently of CDL, soon to be Director of the Digital Library Federation) gave a very clear-eyed assessment of the current state of library IT engagement, and proposed many areas in which the library can do better.

I have to admire the boldness of a presentation whose 4th slide is "Libraries have failed in critical ways".

He said that a key role that libraries have to embrace is to enable discovering books, and searching inside of them.

He had many well-constructed phrases, I particularly liked "technology wipes out our current understanding of how libraries engage the world".

He then moved on to explore the many roles that libraries can still play, both in the physical and digital worlds (including Second Life).

I asked a rather long question about the DLF Architectures group, their DLF Service Framework, library SOA and relationships to the JISC e-Framework.

In his response (and several other times in the presentation) he highlighted the DLF Aquifer project.

If I understand it correctly, it's about building frameworks for discovery and use of distributed digital resources.

We support scholarly discovery and access by:

    * Developing schemas, protocols and communities of practice to make digital content available to scholars and students where they do their work
    * Developing the best possible systems for finding, identifying and using digital resources in context

When asked about new DLF initiatives he has planned, he indicated a few key areas:

• mass digitization
• policy issues (e.g. rights, privacy)
• new virtual communities
• new media and discourse

He had a lot of ideas about engaging with faculty and students, as well as making better connections between the library and IT, and between libraries and publishing.

In particular, he said that libraries should explore the community aspect of dealing with faculty and students - libraries as enablers of community.

In terms of IT engagement he said libraries need to move together with IT to enable deep information discovery.

He also talked about ensuring that we take full advantage of the potential of virtual worlds, and not artificially constrain ourselves to old ways of operating in them - new libraries in new worlds, or to paraphase: No OPAC IN SECOND LIFE!

He said we need to rethink social learning, recognizing that virtual spaces are social spaces with the added aspect that they are freed from normal physical constraints.

He also talked about investigating new social media, having the library as a richer and more imaginative participant in the digital publishing process.  Again a paraphrase: "imagine what the book looks like in an interactive, networked future."

He said libraries need to re-engineer scholarly communication with faculty and with the public.

He mentioned a critical task, which is that we as libraries need to learn how to get our content harvested and ranked appropriately by search engines, perhaps by working more closely with the companies making such engines.

He identified "Collaborative problems, collaborative solutions":

• massively distributed information
• rich data
• new indexing architectures
• mining and mapping our data to build interactive linkages
• challenge of providing ubiquitous access

Successful digital libraries bridge communities, build new services, and help others discover new services to build.

Digital Libraries are the architects of collaboration.

Publishing should be increasingly online and interactive, there should be digital workflows.

Challenge of "open access" to our applications - challenge for IT and libraries.
He talked a lot about how to permit remix, mashups, and combinations/

Of course my solution to that challenge is to ensure that libraries are building on a good service-oriented framework.  I very much hope that the DLF will further pursue SOA initiatives under his guidance.

Thanks to Glen for pointing out this webcast.

In case it's not clear, you can hear the audio of the presentation (which is essential, as the slides are just very high-level talking points) by following the instructions on how to log in to the "HorizonWimba archive" on the presentation page.  Once you're in the interface, switch to the Archives tab and scroll way way down to the bottom.  (Unfortunately, I currently get 503: Service Unavailable from their QuickTime server.)

UPDATE 2007-01-31: I was able to play the QuickTime audio on both my home Mac and home Windows box, with Firefox.  On the Mac the slides didn't display though.  It would be nice if EDUCAUSE offered just the audio as an MP3 / podcast.

free download of SOA book

On the shareware model: try and if you like it, buy it.

Enterprise SOA Adoption Strategies by Steve Jones

It's a free PDF download, you have to register (also free) with InfoQ first.  (Note: Adobe Reader won't let you print the book.)

What I really like about Mr. Jones is that he insists SOA is a methodology, not a specific technology.

I like this a lot since I have been evangelizing the exact same approach.

Ajax as the face of SOA

Bill St. Arnaud's CANet News has a good posting about SOA as infrastructure and Ajax and Web 2.0 as a presentation layer on top.

Web 2.0 - SOA and AJAX - the next killer app?

He cites Service Architecture blog - There is no Web 2.0 without SOA

not only do SOA and Web 2.0 work well together its actually really hard to see how you can have enterprise grade Web 2.0 without changing the way you deliver your existing IT

...

The key here is that a lot of future value for organisations is going to be based around that form of external collaboration with suppliers, customers and partners. Enabling that is a goal of both SOA (thinking), SOA (technology), Web 2.0 (technology) and Web 2.0 (Kool-aid drinking PPT jockeys).

events on Web Services and Service Composition

5th International Conference on Web Services (ICWS)

and

1st International Workshop on Service Composition and Adaptation (WSCA-2007)

this workshop aims to bring researchers and practitioners from different domains with varying research background and building a knowledge-base cutting across the fields of distributed computing and middleware, software service com, artificial intelligence and formal methods to effectively address problems related to Web service composition and adaptation. Specific topics of interest include:

# Software infrastructure for service composition
# Case studies, and applications (e.g., in e-science and scientific workflows)

Workshop CFP deadline March 16, 2007
July 9-13, 2007. Salt Lake City, Utah.

via Glen

A slightly less technical audience may be more interested in SCC 2007 (same time, same place)

2007 IEEE International Conference on Services Computing (SCC 2007)

SCC 2007 will concentrate on the science and technology of Business/Application Services and the bridging technologies such as Business Strategy and Design, Business Process Integration and Management, Grid and Utility Computing, and SOA Services and Solutions

Second Life, Warcraft, and SOA

Yes, I can relate anything to SOA.

I was thinking about the experience that Steve and I had with Second Life, which is, we went in, we wandered around, but we couldn't quite figure out what it was for.  There was no game.  There was nothing to do.  We were homeless and impoverished.  (Well, I was impoverished.  Steve has six dollars.)

I think the answer is that it was the wrong question.

Second Life is platform, not narrative.

We were tripped up by assumptions, based on our previous experiences and background.  Computer Science people of my generation are used to being outside of programming environments.  In Google Sketchup, you also create 3D objects, but you don't create them from inside of a virtual environment, you create them from the outside, Godlike view.

Second Life is like the blinking cursor on a Commodore 64 screen.  READY, it says, and you don't say "what is the narrative" you say "what can I do with this?"  Second Life is like the Tron version of the C64, where you're inside the computer looking out.  The fact that you're inside the SL environment is what threw my thinking askew.  I'm used to programming environments where you're outside, looking in.  In my lifetime, rich visual interaction (well, as rich as was possible on the technology of the moment) has been mainly used for games.

Second Life is a hammer and nails, not a book.  You don't ask "what's the narrative of a hammer?"  Our difficulty with SL was contrasted with our immediate understanding of World of Warcraft.  WoW meets our assumptions, in that it fits into one of the visual game assumption slots, the "Dungeons and Dragons on a computer" one.  There are only a few types of computer games really.  I won't list them all, but here's resource management like in Civilization, there's first-person shooter like Doom, there's abstract shooter like Asteroids, these are all narratives that long-time computer users of my generation understand immediately.

Second Life is not a book, it's a World Processor.  It's interesting that we didn't immediately perceive Second Life as a command-line prompt, because that certainly would have made things clearer.  I don't expect any narrative from the prompt of my terminal application.

There is actually a whole set of relationships to command prompts that I think is interesting to explore.  (And yes, we are getting to SOA eventually.)

David Brin got a lot of attention in the CS community with his article Why Johnny can't code, and certainly those of us from the TRS-80 / Commodore 64 / Apple II era have been wondering how kids get into computer programming these days, without that flashing prompt of the BASIC programming language.  I had been speculating that "the new coding" was designing levels using game editors (many of the Doom / Unreal type games include editors to create new game environments).  Second Life would be a natural extension of this.  So maybe "Johnny doesn't code, he creates in SL?"

The path for my generation of programmers looks quite different.  Flashing READY prompt.  Then maybe an Adventure game (text-based).  They're related in a twisted way.  Most of the adventure games suffered from a limited and extremely specific vocabulary, which turns out to be the same "suffering" that a computer programming language has.  As we were decoding the syntax of C64 commands and lines of BASIC, we were typing stuff into adventure games like

"open the golden cage"
UNKNOWN

"unlock the gilded cage"
CAN'T PARSE THAT

"use heavy hammer on cage"
UNRECOGNIZED OBJECT "ON CAGE"

"get the ^%$^ bird out of the *%&^&*^% cage!!!"

and so on

Eventually that led to MUDs, and MUDs are a clear lineal antecedent of Second Life.  In MUDs you could also create objects and do programming.  However, I think few people embraced the platform aspect of MUDs, most people used the narrative of the D&D type quest game.  Second Life is sort of like a MUD without the game built yet.

This difficulty between the abstraction of an empty platform and the immediate understanding of
a narrative is evident to me when I talk to librarians and others about SOA (you see I told you I was getting here eventually).  They always say "that's nice, but give me concrete examples".  In the context of this posting, I guess I would say, "Web Services are a hammer, not a book, and SOA is the architecture of the house you could build with that hammer".

In a similar way, this illustrates some of the challenges between the catalogue and the idea of a general open library technology platform.  The catalogue has narrative.  I would argue it's the wrong narrative, with concepts mis-adapted from the traditional librarian-card catalogue-patron world of paper, but it is as least something one can grasp.

The attraction of narrative is probably one of the reasons that Steve and I have been spending way too many hours in WoW, while our visits to SL usually last about five minutes.

In my next posting, I will talk more about how assumptions can create challenges and barriers, specifically about how extending analogue library paper world assumptions to digital can lead to incorrect conclusions.

Tapscott: turn your organization inside-out

Ok, the way he puts it is

Part 5: Platforms for Innovation

This is a very important, key concept.
In fact, let me say it again, with colour and size:

This is a very important, key concept.

This is absolutely foundational, in fact, to understand a lot of the issues that I talk about.

The core is as simple as simple can be: you get more from sharing than from secrecy.
But you have to understand the context of "secrecy".  In many cases, anything that is not published on the public Internet, might as well be in a locked box on a high shelf.  In fact, you might as well print it out, send it to the new Library of Alexandria, and burn down the new library.

The benefits of sharing are so enormous, and the extra overhead so tiny, that I have never quite understood why people don't post more information publically on the net.  My thinking has always been "if I do this for myself, maybe there is someone else out there it might be useful for".  Considering that a page I did for myself as a firewall sysadmin gets over a million hits per year, it appears this concept is valid.

During the time of the Clinton Administration, as reported (in passing) in The River at the Center of the World by Simon Winchester, they turned classification inside out.  Instead of saying "prove this should be declassified", they said "prove this should be secret".  That led to the author of the book getting a map of China that, I suspect, he would no longer be able to get today.

This is the challenge for your organization.  Don't ask yourself "what's the justification for sharing this information on the net".  Ask "can anyone strongly justify NOT sharing this on the net".

I see the most ridiculous stuff go on.  Librarians (and many others) take written notes at conferences, then labouriously transcribe them to a written report, which is then emailed to like two other people and immediately forgotten/lost.

People don't have crystal balls to see within your organization or within your mind.  If you don't share your information, you might as well be invisible.  How's that for a slogan?  "Share transparently or be Internet invisible".  Additionally, these are not the days of late-1994, early-1995 when I first put up a website.  Then an interesting effort was highly visible, because there were so few pages.

There are now billions of pages.  The good news is, you can still rise out of those "foothills".  The bad news is, if you have a search surface, an acreage (hectarage?) if you will, that is tiny, you have very little chance of being noticed.  If your land isn't continually being expanded and improved, in fact, it will probably get little (search engine) notice.  So here's another concept: "Sharing increases your search surface".  In security, we talk about making your "attack surface" or "vulnerability surface" as small as possible.  Internet presence is the opposite: you want to make your Internet surface as big as possible.

I know this is true, because my website and my blog bring me incredible contacts and opportunities that I could never have imagined, despite the fact that both Internet surfaces are often little more than "stuff that I wanted to research and think about anyway, which I'm sharing because there's no reason not to, and someone else may have similar interests".

Anyway, this posting has grown very long, but that is because it is of such critical importance.  Think beyond container to content.  Think beyond content to sharing content.  That is where everything is going to be happening.  That is going to be the new expectation.  If you've seen the controversy in Toronto and Ottawa about private municipal contracts and secret city meetings, that thinking is mired in the past.  Citizens, young and old, are going to be demanding the end to ridiculous, needless secrecy, from their governments and from all other organizations that they deal with.

So what is library Service-Oriented Architecture about?  It is about no more (and no less) than helping libraries collectively to build an open, shared platform, that enables sharing of containers (e.g. books) as well as sharing of content (e.g. articles).

Here's how Tapscott puts it

A growing number of smart companies are learning that openness is a force for growth and competitiveness. As long as you're smart about how and when, you can blow open the windows and unlock the doors to build vast business ecosystems on top of what we call "platforms for innovation."

Because I am a prideful creature, I must say I wrote this entire posting after having read only the first few sentences of Tapscott's article today, it is therefore heartening synchronicity to read

Jeff Barr, who runs Amazon's Web services program says developers and marketplace sellers are "increasing the surface area of Amazon." They add more and more things to sell, in more and more places on the Web. All of this happens in a completely self-organizing fashion, which makes Amazon's already low overhead even lower.

So let's wrap it up with a final combination: library SOA will increase the Internet surface of all participating libraries.

in which the OPAC was wrongly provided

Perhaps you have heard this expressed already, but I haven't seen it put this way, and it helps me to frame the issue.

Libraries used to do two things:
- librarians talked to patrons
- librarians used the catalogue

then the Internet came along, and they decided patrons should use the catalogue online

MISTAKE

There are two things libraries need to support in the Internet environment
#1 patron access to library services
#2 advanced patron and external system access to library data and services

#1 is belatedly being attempted with layers on top of the catalogue, like Endeca
Personally I would have just said to Google "make me a website that will return relevant results to my patrons from my catalogue data, and link to the appropriate ordering pages", but since libraries have a hate on for Google, that's basically what they're cooking on their own with Lucene.

Anyway, #1 seems to be understandably the current number one library obsession, and I'm not going to say it's trivial, but it certainly appears technologically straightforward. Step A) rescue your data from its catalogue prison - suck it out somewhere that modern technology can be applied to it. Step B) apply modern off-the-shelf search technology to it.

I'm not even sure you need to do Step B. Step B could equally be "let every search engine in the world index it with appropriate links back to my library, and some way for patrons to identify what geographic location or specific libraries they are interested in".

#2 gets more into the Service-Oriented Architecture areas that interest me. I don't think libraries ever really thought much about machine-to-machine communication of data and provision of services. Even Z39.50 works in the library world something like a) librarian formulates search b) search sent by Z39.50 c) librarian analyzes results of search and e.g. uses for copy cataloging. That is to say: humans. Always humans.

So while I think getting data out of the catalogue is an important first step, there is much more to making a full SOA for libraries, at least one that won't be DOA. Think about

* how can you expose your data for use by other people and machines - microformats? REST interfaces? Web Services?
* how can you expose your library services ("request ILL", "please purchase this book", etc.) for similar use by other people and machines?

I think considering the above issues will lead you to conclude that either your library individually, or all libraries in general, need to do a service-oriented analysis of their business functions and data.

Previously:
December 5, 2006 but how does SOA fix my ILS?

my presentation on SOA and BPEL for Access 2006

My presentation

http://www.slideshare.net/scilib/library-web-services-for-discovery-and-delivery-of-scientific-information

Bookmarks from within the presentation, with some corrections and additions, available at

http://www.connotea.org/user/scilib/tag/access2006akerman

The audio of the presentation is at

http://www.access2006.uottawa.ca/2006-10-13-07-akerman.mp3

Overall, I thought it was well-received, but I still don't think most libraries are thinking beyond adding a few Web Services.  The main focus for library technology seems to be around the catalogue and adding layers on top of it, not breaking it apart into services.  There still seems to be limited concrete action in working on library SOA to integrate with the various frameworks that are out there.

If we can't fix the ILS, maybe we could work on a smaller group addressing issues related to journal article repositories?  I didn't even touch on the Fedora Web Services and workflow support in the above presentation.

I basically continue to be concerned about a lot of diverging wheel-reinvention activities, rather than seeing a lot of unifying activities starting to deliver good models and services.

Previously:
October 15, 2006  SOA BOF - links and thoughts

Sidebar:
This is not as-presented, this is a previous version, the reason being that somewhere between Windows PowerPoint 2000 SP3 and Mac PowerPoint X, the clickable weblinks got completely munged.  Wedged to the extent that even when I try to fix them in Win PowerPoint 2000, it won't let me save the changes.

I don't know if this is related to some bizarre Microsoft limitation, or to some Win-Mac interchange bug, or to a Mac specific bug.

Plus which I got an exciting SlideShare error when I tried to upload: "Camping Problem! SlideLoader::Controllers::Upload.POST"

thinking about SOA and Web 2.0

Some interesting thinking from FastForward blog

SOA is about enterprise efficiency. Web 2.0 is about individual empowerment.

For Web 2.0 projects, the operative term is ‘just do it.’ For SOA, build a business case first.

In a way, this tells you that if you don't have an enterprise, you don't need to be as concerned about SOA.  But that then also tells you about scale.  So SOA may not be a good match for some individual libraries, but it may be a key methodology for efficient resource sharing in the overall system of libraries.

Rethinking Resource Sharing: Get It

Somehow this managed to fly beneath my radar, it looks like because it's people talking document delivery / ILL language to other librarians, rather than people talking Web Services to library IT people.

Here's what I discovered

A browser plugin which annotates web pages with links to “getting” options for published resources held by libraries

The GET-IT plugin scans web pages for book-like resources and attempts to annotate the page with “getting” options provided by libraries, tailored where possible to the user.

from Kent Fitch's presentation A New paradigm for “getting” (PowerPoint) from Libraries Australia Forum 2006 (LAF06)

I have to express some mild annoyance, mixed with hope, to read statements like

Mission of the [Resource Sharing] Initiative

* Create a new global service framework that allows individuals to obtain what they want based on factors such as cost, time, format, and delivery.
* This framework will encompass promoting and exposing library services in a variety of environments.

from presentation Rethinking Resource Sharing (PDF) by Candy Zemon, given at NISO Workshop Discovery to Delivery, November 3, 2006.

A "global service framework", hmm, where have I heard that before, it's almost as if they want to build an architecture based on services, how shall I put it, a sort of "Service-Oriented Architecture".

No offence to librarians, but if you want to build an SOA, how about actually calling it an SOA, and working with the Web Services and SOA working groups?  Does everything have to be a separate initiative with a giant Steering Committee that uses special language ("Resource Discovery") so that no one outside knows what they're talking about?

Why can't we all get along?  And speak a common technology language?

Isn't the Get It button already used by OpenURL resolvers?  Is this going to make things simpler, or more confusing?  Are the LibX people involved?  Are we going to end up with a bunch of competing browser extensions?

Apparently you can attempt to find some information starting points at
http://www.ala.org/rusa/stars/

Here's my modest proposal:

1. Let's have a single, shared set of terminology and a single, shared library Service-Oriented Architecture

2. Let's make one high-functionality browser extension and lobby Microsoft, Firefox et al. to have it included in the default installs

 

IT directions for National Library of Australia

Libraries, IT and Everything
Mark Corbould
Assistant Director-General
Information Technology
National Library of Australia (NLA)

presented at Library and Archives Canada (LAC)
December 15, 2006

presentation notes by me, Richard Akerman

- strategic vision of the national library of australia

- function: basically, maintain a national collection and make it available

- 479 staff
- over 550,000 [physical, presumably] visitors
- 110 million page views on website, up 59%

~ 9 million items (including 2 million manuscripts and 2 million serials)

Access

- 365,000 physical items delivered
- 45,000 reference enquiries (4,000 via Ask Now)
- 13 million digital objects delivered

~80 TB of digital object storage, including Australian Web Domain Harvest
- main stuff digitized is unique cultural heritage
- 40% of digital collection is oral history

Web Site Usage

- nice chunk on PictureAustralia, new chunk on MusicAustralia

- IT Strategic Plan (3 years, annual update)

Strategic Directions [phase] 1

* for a long time it was build, describe, preserve and provide access to the physical collection

Strategic Directions [phase] 2 [in addition to phase 1]

* since 2000 "to provide rapid and easy access..."
- outcome: online federated discovery services with "no dead ends"

Looking Outwards

* Internet-enabled collaboration
- lots of freely available content
- organizations are making their content available for remixing and repurposing
- low barries to participation
- light-weight trust models, no MOUs
- sharing personal views of contemporary events
- creating social networks around areas of interest
- near enough is good enough
- loss of control
- need to be less risk averse

Strategic Directions [phase] 3 [in addition to previous]

* In 2006, "to enhance learning and knowledge creation by further simplifying and ... services..."
[slide switched before I got the rest]

Key IT Outcomes

1 ensure collecting record of Australia
2 To meet user need for rapid
5 Ensure relevance

[yes, he skipped a couple]

IT Goals

many including
* Develop and deploy new full-text search software
* Provide online spaces to support publishing, collaboration, contribution and interaction

The rest of the world wants to find stuff in Google etc.
"we've got to get our data out there"
worked with Google to get their books in Google Scholar

Get this item

* Bookshops
* Suppliers

Also A9.com - Libraries Australia is searchable by using OpenSearch lightweight protocol

working to do federated search across museums and other organizations, using OpenSearch

Libraries Australia [national union catalogue I think]

* considered replacing individual library catalogues (as a starting point) with
"Libraries Australia, as the  primary database to be searched by users"

to do this you need to be able to integrate well with all the OPACs, which of course is a problem

Expanding Borrowing

* Wake-up calls: statistics and commentary
- [Lorcan] Dempsey "Materials are not being united with users who want them" [not sure of quote,
switched to next slide]
[You can see the entire quote and more info in Kent Fitch's presentation A New paradigm for “getting” (PowerPoint) from Libraries Australia Forum 2006 (LAF06)]
- Long Tail argument

Current Fulfillment
- charge $13 for ILL, total cost actually $49

steady decline in ILL

ILL: Strong disincentives to participate
- Expensive
- Slow
- Loss of control of assets
- Inconvenient / impossible

Fulfillment

Making "Search, find, get" seamless

* Lend directly from library to reader - NetFlix model - "NetBooks"

Also get your presence into e.g. Amazon.com "Borrow this book from Libraries Australia" (using greasemonkey)

PictureAustralia
http://www.pictureaustralia.org/

includes pics from Flickr
about 6000 images harvested

  "All you need in Aussie" - by Alicia Zappier 
  Originally uploaded by desertgirl.

MusicAustralia

* bought metadata catalogue of australian contemporary music - will interface to ecommerce gateway

Model

* "metadata discovery should be free, but you may have to pay for fulfillment"
* or I would say "easy access is more important than zero cost"

AustraliaDancing

* "Take Part" - customized wiki

There's more...

* People Australia, based on using authority file
* Open Access Journals - "really cheap and easy, blurs between publishing and collecting"

Issues

* Sustaining existing services
* Managing expectations
* Supporting innovation
* Enabling rapid prototyping
* Being vibrant and relevant

Ok, but how

* Reorg IT

* Review IT Architecture
- Service Oriented [emphasis mine]
- Consolidate metadata repositories
- "single business" model

* Create an IT-aware organization
- Communicate, collaborate and train

42 IT staff, over 400 library staff

Additional tech notes
- using Lucene full-text search
- using Confluence - new IT architecture is on a wiki
- training business analysts in BPM

Q: How does IT decide on priorities for projects?
A: Through the operational plan, reviewed by Corporate IT Group that sets the priorities

If necessary, escalate to corporate management.

Q: Web Harvest?  How made accessible?  Federated discovery - what challenges?
A: Paid the Internet Archive.  Ran for 6 weeks.  About 180 million files.  On a PetaBox, installed in
Australia.  Internal access but not public.  Also the Internet Archive will put it up.

Permissions/legal issues within Australia.

Issues with robots.txt - if you follow robots, you may not get inline images or CSS.

Federate search is basically around OpenSearch, plus may need to add relevance ranking.

Q (me): How are you doing relevance ranking?
A: Currently Teratext

but going to use Lucene.

but how does SOA fix my ILS?

It just happens that I am thinking a lot in the SOA / Web Services / ILS / OPAC space and in the computer scientists (please don't call us "techies") talking to librarians space.

I guess one difference that I perceive is that CS people often don't, due to their training, start with a particular technology (Ajax, RSS feeds, whatever) when looking at solving a problem.  Similarly, Enterprise Architecture is about first abstracting the basic desired function ("provide relevance ranking") rather than examining a specific technical solution ("stick Endeca on top of my ILS").

That's why you will find me writing a lot about "what", as in, what are the underlying things you want to do, rather than "how".

So that being said, I see a lot of activity around fixing the ILS, unsucking the OPAC and such.
And I realize I haven't done a very good job of constructing the chain from SOA theory to new improved disintegrated library system.

To some extent, it is the same "you can't get there from here" pig makeover story that has been extensively discussed elsewhere.  You can't incrementally decorate your way out of the underlying problems with the ILS.  Have you seen Extreme Makeover: Home Edition?  They have a simple process for transforming people's houses:

1. Demolish the current house
2. Build a brand-new house (using the latest tools and technology, along with a lot of hard work)

What does that mean in practical terms?  It means you need to be building a brand-new library system, using the latest tools and technology, along with a lot of hard work.  Now if you've seen EM: HE, the other thing you will notice is that in the triangle of people versus time versus scope, they make the time very small, by making the number of people working on the house very large.  So: it takes a village to raise a new library system.  And the construction materials that we have at hand, the latest concrete, wood and utilities, are services (whether you call them Web Services, REST APIs, new standard interfaces, extended OpenURL - it's all services).  Services are the new pieces you need to build the new LS.

But... you're probably ahead of me here.  If you put these pieces together haphazardly, the house will fall down.  If you don't want your new LS falling apart, you need... architecture.  A sort of Service-Oriented Architecture, in fact.  An SOA agreed upon by the community (the library technology village), defining a standard set of services so that we can all build our own unique Extreme Homes.

Without a common set of services, a shared basic architecture, it's as if you're building pyramids with giant chunks of limestone, and I'm building temples with bricks and concrete: we're both using great technologies, but we can't easily build on each other's work, because our fundamental building blocks are completely different.

That's all that SOA is, a way to come to a common agreement, so that we can all use the same pieces, the same technology components.  If you agree that the way to get to a new library system is by using services, then SOA provides the framework to ensure that we all get to use the same pieces, so that no one organization has to build the entire thing themselves.  You use my relevance ranking service, and I use your book cover service.

And all of this does NOT have to be slow, with everyone waiting for some giant SOA committee consensus.  Let's agree on a few services, and combine them in different ways to make a bunch of version 1 library systems.  And then a few more services, and a few more.

Replacing the monolithic ILS with a new silo will fail.  Trying to define everything in advance in some Big Bang Waterfall will fail.  But total anarchy of services will, I believe, also fail.  We can't spend all our time pouring and re-pouring the foundation of the house, putting up a wood frame and then tearing it down and putting up a metal one.  We need to build services on top of services, layer by layer, orchestration by orchestration.

So I apologize if I'm often on about apparently esoteric debates concerning SOA reusability and similar theoretics.  What all the SOA jabber is about is making sure that we have the right methodology to sustain building more and more complex sets of services, or in other words, ever more complex yet maintainable new library systems.

If this explanation is still not doing it for you, please let me know.

two perspectives on interoperability and what it could enable

In case you're wondering, I finally managed to clear my Bloglines backlog, including the published literature, D-Lib and Ariadne.  I found two very different, but complementary views on how service standards and standard interfaces can enable an enhanced scholarly workflow or other advanced combinations of services.

In Serving Services in Web 2.0 (Ariadne issue 47, April 2006, ISSN  1361-3200), Theo van Veen of the Koninklijke Bibliotheek in the Netherlands explores and explains some fundamental concepts of Service-Oriented Architecture and standard service interfaces.

In this article I discuss the ingredients that enable users to benefit from a Service-Oriented Architecture (SOA) by combining services according to their preferences. ... This concept is an extrapolation of the use of OpenURL and goes beyond linking to an appropriate copy. Publishing and formalising these service descriptions lowers the barrier for users wishing to build their own knowledge base, makes it fun to integrate services and will contribute to the standardisation of existing non-standard services.

In An Interoperable Fabric for Scholarly Value Chains (D-Lib, October 2006, Volume 12 Number 10, ISSN 1082-9873), Herbert Van de Sompel, Carl Lagoze et al explore how you can build services using an interoperable network of digital object repositories

This article describes an interoperability fabric among a wide variety of heterogeneous repositories holding managed collections of scholarly digital objects. These digital objects are considered units of scholarly communication, and scholarly communication is seen as a global, cross-repository workflow. The proposed interoperability fabric includes a shared data model to represent digital objects, a common format to serialize those objects into network-transportable surrogates, three core repository interfaces that support surrogates (obtain, harvest, put) and some shared infrastructure. This article also describes an experiment implementing an overlay journal in which this interoperability fabric was tested across four different repository architectures (aDORe, arXiv, DSpace, Fedora).

airlines and everything with built-in carbon offset

Will SOA save the world?  We're working on it.
Jon Udell has a fascinating InfoWorld article "The carbon-adjusted supply chain: SOA-enabled optimization can help reduce businesses' impact on the global environment" and accompanying blog posting.

Does Amazon know enough about its supply chain, ... to assign a value to the atmospheric carbon attributable to the manufacturing and shipping of its products? Bezos thought that the answer was no, but he was clearly intrigued by the question. So am I.

...

Economists have a wonderful euphemism for environmental impacts. They call them “externalities,” and we can blithely ignore them until Rhode Island-size chunks of Antarctic sea ice start to vanish. Then we start to realize that, in a closed ecosystem, there are no externalities.

In order for Amazon to be able to measure and report its externalities, of course, Amazon’s suppliers would themselves have to be able to measure and report theirs. That would be a major challenge, to be sure. But it’s exactly the kind of challenge that SOA-enabled supply chain optimization prepares us to tackle.

To my surprise, there are already a number of initiatives to help people offset their carbon from air travel, although I don't know how big the idea is yet in those paragons of carbon production, the US, Canada and Australia.

Anyway, my Big Idea is quite simple and appears to be already underway at various sites: every airline should include, as one of its option fees, a pre-calculated carbon offset cost with a trusted offset vendor.  Air Canada, are you listening?

Ideally, offset providers should have calculation Web Services and payment Web Services that other sites could use.

[UK] DEFRA (Department for Environment, Food and Rural Affairs) - Carbon Offset Scheme Launched - September 12, 2005

British Airways has launched a new scheme, backed by the government, where its customers can volunteer to help to offset the carbon dioxide emissions from their flight by making a contribution to an environmental trust.

The money raised will be used by an organisation called Climate Care to invest in sustainable energy projects that tackle global warming by reducing carbon dioxide levels.

Air travellers can choose to make a donation from today (September 12) via a link from the airline's website, ba.com, for the cost of the emissions created by their journey. For example, the donation on a return flight from London Heathrow to Madrid will cost £5 and a return flight from London Heathrow to Johannesburg will cost £13.30.

You can visit http://www.britishairways.com/travel/climateimpact/public/en_gb for more info.

Silverjet, also in the UK, is building carbon neutrality into their price.

Silverjet, the first British airline to offer a low fare, exclusively business class service across the Atlantic, today announces that it will be the world’s first airline to become carbon neutral on all its flights.

Included within its ticket prices will be a mandatory carbon offset contribution, giving passengers the opportunity to reinvest the “carbon points” thus earned back into a number of climate friendly projects around the world. This still enables Silverjet to offer low fare long haul, exclusively business class, transatlantic flights from as little as £999, return.

The scheme is being set up in partnership with the leading climate change business, The CarbonNeutral Company, and it has been developed in accordance with the CarbonNeutral protocol, which is the leading standard and quality mark for action on climate change.

Silverjet PR - Silverjet is the world's first airline to go carbon neutral - November 26, 2006

In Canada, there is UNIGLOBE Travel's Green Flight Program

UNIGLOBE Travel's Green Flight Program is one of the ways you can achieve your CO2 reduction goals.

UNIGLOBE Travel's Green Flight program is getting much press for its ability to provide companies in Canada with the mechanism that has, for a long time been missing to meet carbon emission offset goals. One way to achieve these goals is to purchase carbon offset credits from UNIGLOBE Travel that are calculated on the number of miles flown in an aircraft generated by companies business, conference or employee incentive travel.

The monies collected from the credits are then invested into environmental programs supported by Environment Canada.

UNIGLOBE's offset provider is Baseline Emissions Management Inc., and the specific site provided by Baseline is

http://www.greenmyflight.com/

Also, via The Great Canadian Carbon Offset 2006, I find a rather roundabout offset arrangement - buy a WestJet ticket via offsetters.ca and offsetters will use their affiliate income to offset some of the carbon. (GCCO2006 link via Treehugger.)

lastminute.com UK also offers this option

Customers using lastminute.com are to be given the option to offset carbon dioxide emissions from flights.

The online retailer claims to have become the first major travel company to make CO2 offsetting an integral part of buying a flight in a Government-backed initiative.

Payments will be invested in sustainable energy projects to reduce the damage done to the environment in a partnership with Climate Care, an organisation which tackles global warming.

TravelMole - Lastminute.com to offer emissions offset option - November 24, 2006

and indeed the option appears prominently at the top of their UK site, with accompanying web page

and not at all on their US site

cheapflights.co.uk has a variety of information - Air passengers can offset CO2 at Treeflights

Passengers who want to offset the carbon dioxide emitted from their flights have the option of using a new carbon-offsetting website where they can pay to plant a tree.

Treeflights, whose motto is "You fly – we plant", will plant one tree in a Welsh forest for each flight taken, at a cost of £10 (or $19/€15).

as well they list other options on their Carbon Emissions page

Al Gore uses Native Energy to offset all of his travel, but unfortunately it's a bit US-centric (miles? map of the USA?)

http://www.nativeenergy.com/travel/

There are lots of competing carbon offset providers, unfortunately I have neither the skills nor the time to evaluate them all.  I may expand this posting as I learn more.

Wikipedia - Carbon offset is probably another good starting point, and they'd take away my Canadian environment blazer badge if I didn't point to David Suzuki.org - Go Carbon Neutral.

little SOA and the reusability debate

There's quite a wide debate going on about SOA.
Some issues include

• do you need it at all?
• how big (how complex) does it need to be?
• will it actually give the promised reusability?

I found some interesting postings, good discussion starters:

SOA What? Big vs. Little vs. Nothing at All - Josh Greenbaum - Enterprise Anti-Matter blog

the relative paucity of companies that today are betting their business on a SOA strategy — in an enterprise-wide way — has brought the market to an important inflection point. The need for comprehensive ROI data — as well as some good info on total cost of ownership — has never been greater. And until those data — that elusive business case — are readily available, the uptake of what I call "Big SOA" is going to be slower than most of its proponents would like. "Little SOA" seems to be where the action is for now — pilot projects, smaller, discrete implementations, and custom-built composite applications that deliver some immediate value to a specific line of business in need.

The ROI of Little SOA is easy to establish — these discrete little projects have a lot of potential payback, as they are addressing real pain with a real solution. The problem with Big SOA is that the pain it addresses is one we've gotten much too used to over the years. Costs associated with bad integration, lack of re-use, overall inefficiencies and the inability to truly address business needs with technology solutions have been with us so long that we hardly know they are there.

...

The alternative to Little SOA is none at all, and I think that's a major disadvantage for all concerned. Service architectures and web services are as inevitable as Moore's Law. What's needed is just a little more time to let the big ROI of Big SOA get established. In the meantime, there's Little SOA to keep us all happy, and that's not a bad thing at all. 

I think that's a pretty reasonable position, for large organizations.  Doing a true, "Big" SOA in an organization with substantial infrastructure and processes will be enormously disruptive if you do it properly.  I do however feel, that if your IT is relatively simple (say, a single catalogue system and some web pages), you can get most of the Big SOA advantages.  As long as libraries are collectively working on a bigger SOA then at most libraries, the actual activities will look a lot like Little SOA.

I do get worried when we talk up the merits of "discrete little projects".  Isn't that how we got ourselves into the silo mess in the first place?  One little app here, one little app there... Yes, I know, Enterprise Architects generally say "you always need an architecture".  But I'm worried about a slippery slope that leads to hundreds of un-coordinated Web Services.  That's no path to reuse.

In SOA in enterprises or Hype 2.0, Vilas asserts that we won't get real reuse from SOA, but that we should do it anyway.

[Hype] 1. Myth of reusable services

In my experience as architect I have never seen as-is reuse of a business service implementation. Some amount of refactoring is needed for it to be reused. The refactored business service actually harbours multiple services under a common facade.

...

[benefit] 1. Manageability of software with business alignment

The single most important contribution of SOA is that it connects software with business. In an SOA approach, one can make sure that all software is aligned with business needs, because all software is traceable to their business needs.

...

[benefit] 2. Ability to seperate concepts from implementation

The next important contribution of SOA approach is the focus it brings on seperating interface from implementation.

I think that's an interesting perspective.  We do need to be realistic about the "Holy Grail" of reusability, and not oversell it, as well as not overselling Big SOA.  But I nevertheless feel that some of us have to be trying big scale SOA, because quite frankly, how else do we move the software engineering framework forward?  The only alternative right now seems to be to have no architecture, to do some sort of rapid mashup stuff, which I think can't possibly be sustainable.  (I'm not saying rapid mashups don't have their place, and they are way better than spending a year waterfalling something that won't deliver, but you have to remember, An Application Is Forever.  Humans are terrible at stopping things once they start.  That's why you need a framework to make sure you're starting the right things.)

crossing categories

There's a good presentation about library SOA http://info.drc.ohiolink.edu/presentations/200611-uwindsor-soa/ that points to my Web Services category, you may also want to check my SOA - Service-Oriented Architecture category.

my review of SOA for Dummies

Summary

I am reviewing this book primarily in the context of Service-Oriented Architecture (SOA) for Libraries, which is a rather dramatically different audience than it is intended for. 

I recommend the first three chapters of this book, along with the "loosely coupled" section of chapter 5 as a good introduction to this topic.  As well, Chapter 11, on SOA Governance, is excellent.  However, its main audience is large organizations (who are the main current adopters of SOA), so much of the content may not apply to libraries, particular those with limited technical resources.

Although the Dummies books proclaim "A Reference for the Rest of Us", this book would really more accurately be titled "SOA for Dummies... Who Work at Giant Fortune 500 Companies".  Much of the technology and processes they present are beyond the reach of all but the largest organizations.

To my surprise and dismay, there isn't a single mention of Enterprise Architecture (EA) in this book, whereas I feel that for the scale of organization this book targets, EA is absolutely essential.  Without EA, the scale of activities and technology described in this book is impossible.

There is also a very important discussion that needs to be had about agility versus structure.  I am not convinced you can maintain organizational agility as a goal, with the complexity of the management and technology structures this book describes.  The book does not address this issue at all.

Overall the flow of the book is a bit uneven, both in terms of writing style and in content, and it doesn't present a coherent "SOA story".  To some extent this is due to the nature of the Dummies books, where they want each chapter to stand alone.  I found it made for a bit of a "here's an important concept" ... "and here's another one, with some vague undefined relationship to the previous one".

This book will mostly give you a lot of "why" and "what" about SOA, not a lot of "how".  That is the nature of the beast, however - I don't think there's any other way to present it, with out it being a 3000 page book.

The companion website www.dummies.com/go/soafordummies is highly promoted on the front and back covers of the book but it does not seem to exist, which seems like a bit of rather bad planning.

Sidebar

UPDATE 2007-08-09:

If you're coming here from SOA Center, you may be expecting Beth Gold-Bernstein's review.
The SOA Center link is currently wrong.  She blogs at ebizq:

July 24, 2007  Beth Gold-Bernstein  SOA for Dummies
July 31, 2007  Beth Gold-Bernstein  Reactions to SOA for Dummies Book Review (in which she refers to my review as having some similar points to hers)

I did leave a comment on SOA Center correcting the link, but it has not been posted.

ENDUPDATE

Details

I don't come to this book with a blank slate, quite the opposite, I've been trying to find ways of clearly explaining SOA to both business people and technical people for the past few years.

Just to stake my prior claim, I think I did a pretty good job in my December 9, 2005 presentation "Service-Oriented Architecture Methods to develop Networked Library Services" (video, PowerPoint) which predates this book by about a year.

It is a difficult story to tell to be sure.  To my mind, SOA sits at an intersection of Enterprise Architecture, software engineering, Web Services, business and technology.  So in a way, the challenge is "let's take a bunch of things that people don't understand, and try to explain a layer on top of them, that depends on all of them".

Also, in the interests of transparency, the informal style used in this book is not one that I enjoy for learning.  My ideal informational book is just facts piled upon facts - I don't need a story, or to be put at ease. 

I tried in my analysis to ignore that stylistic aspect, as well as trying to come with as blank a slate as possible, keeping in mind I've been deep into SOA for the past couple years.

Here are my thoughts, chapter by chapter.

Chapter 1 - SOA What? (this makes more sense if you pronounce it so-a, as the authors do, rather than S-O-A)

All of Chapter 1 (PDF) is available free online from Wiley.

They talk about their audience, but I do think they make a mistake in not scoping the book.  The full SOA they describe in the book's 300+ pages is one that could only be used by a large organization.  Think, General Motors revising their information technology, not Bob's Corner Store.  They should have made it clear that you can adopt some SOA concepts, without going to the full, complex, vendor-rich (and vendor-enriching) extent that they document.

For example, they start out with some clear explanations about how SOA is an enabler for business to work better with the IT department.  You may at that point be thinking, "my problem is not working with the IT department, my problem is I don't have an IT department".

So in reading the book, be mindful of their audience.  And also, I suggest you think of how SOA could help the entire community of libraries, collectively, if the scale of your individual organization is not appropriate for SOA.

They do extract good key points, like the fact that reuse is at the core of the SOA goals and methodology.  At this level, they face a challenge that I don't think there is a good solution for, which is that you have to talk a lot at the beginning about SOA "why" and "what", without being able to easily walk through a lot of SOA "how".

I also think that their definition of SOA is a bit over-broad.  Some of what they describe as SOA is what I would consider Enterprise Architecture (EA).  Now, I certainly think SOA works best within an EA context, but there doesn't seem to be any mention of EA at all.  So I disagree when they say

1. The IT organization will partner with the line of business managers to create a high-level map of what the business will look like.

That surely needs to be done, but that's EA, not SOA.  To me, Service-Oriented Architecture is just the architecture of the services, not all the processes of business process analysis as well.

Chapter 2 - Noah's Architecture (oh I get it -- literally, just now - it's a play on words, see.  sigh.  I spent the entire chapter wondering, what does Noah have to do with it?)

So at this point, the conceptual assault really gets underway.  To get to the heights of SOA, you have to build concept upon concept.  Unfortunately, each one of those concepts could be a book in itself, and here they get a couple pages at best.  I'm not being critical - I don't see any other way you could do this, and it's the same approach I have used, but I can imagine it being overwhelming reading for someone coming from outside a software engineering background.

They give an ok explanation of software architecture, and hit the high points of the history of software engineering that got us here.  I liked their mentions of business logic and the key concept of separation of concerns.  Again, they are talking about big organizations, for example, on page 16, they state

In the Good Old Days, a business unit would ask the IT department to create an application to solve a specific problem.  To accomplish this goal, the IT department would write a set of customized programs.

This is absolutely, 100% true.  For a certain scale of organization.  If you're thinking instead, as I mentioned before, "I wish I had business units.  Or an IT department." then SOA may not be a good match for you.

They then take a bit of a side trip into components.  I'm not sure I would have used this terminology.  It's certainly true that we tend to run out of useful worlds to describe certain concepts, but to me components are about a specific subfield of software engineering, Component Based Design.  Software components have a very specific meaning, a meaning which is different from the one given in this book.

This is important, because if you read this book and say "component" to an IT person, they may not have the same understanding that you got from the book.  In a similar channel, the use of "plumbing layer", which is continued throughout the book, is most unfortunate.  "Infrastructure" would have been a much better choice.  Whatever dubious merit there is in the informal term of "plumbing" is going to be lost the first time a non-tech person uses the word to a technical person, and the technical person thinks "Plumbing? What the heck?  Is this guy an idiot?"

At this level, there does have to be a bit of "hand-waving" over the explanations, this is particularly egregious (although the authors themselves acknowledge this somewhat) in the "simple adapter" explanation.  Adapters are probably some of the most complex challenges an organization faces in transitioning to SOA.

I did enjoy the brief excursion into "application archaeology", particularly since my original SOA presentation on SOA had a slide or two on this concept, with a nice quote from Vernor Vinge's writings, but I had to take the whole segment out because people said "application archaeology, what the heck were you talking about there?" (I'll try to dig up the quote, it's excellent.)

I'm also not a super big fan of the "black box" terminology used, but it does capture the concept well enough I suppose.

Chapter 3 - Not So Simple SOA

A bit of a jump from components to Web Services, with a bit more handwaving.  I also didn't like the strong link between Web Services and SOA.  While their statement that SOA is not Web Services enhanced (or I guess I should say "is not Web Services 2.0") is certainly true, it's also true that you can do SOA without Web Services at all.  The two are, as we say, loosely coupled.

There's also a bit of a jumble about business processes, I'm not sure it fits well within this chapter.

Finally, while silos are mentioned, I think they really missed an opportunity to emphasize a key failure of siloed software - that it locks up useful services deep inside the silo, where no one can get at them.  This aspect is mentioned only in passing.

All those caveats aside, I did feel that Chapters 1-3 are about as good a clear explanation of this extremely layered concept that you can get (well, that is to say, almost as good as my explanation :)

You should really jump from Chapter 3 right to Chapter 5, I'm not sure why they placed Chapter 4 here.

Chapter 4 - SOA Sophistication

Woo boy.  Chapter 4.  If you are coming from a small organization, or a non-technology background, your service experience of chapter 4 will be akin to hitting a brick wall.  Chapter 4 layers on the SOA "essentials" at a breakneck pace, presenting as essential elements of SOA things that are, each of them alone, HUGE complex business activities.  We are definitely into the General Motors scale here.  The Galactic Motors scale, in fact.

Let me make it immediately clear: I do not personally believe you need all of the stuff they describe in order to use SOA, and to participate usefully in an SOA.  The basic SOA methodology can be used by almost any organization that has at least some idea of its basic business functions (and some business decisionmakers) and a small IT group to design and build services.

So, what are the bricks in the giant Chapter 4 wall?

• Enterprise Service Bus (ESB)
• SOA Registry
• Workflow Engine
• Service Broker
• SOA Supervisor (about which, I have more to say)
• Business Process Management (BPM)
• Service Level Agreements (SLAs)
• Automated Service Monitoring (related to SOA Supervisor)

In looking at that list, my first reaction is "hmm, are there any 30 million dollar cheques I haven't cashed yet?" and secondly, that almost anyone, even someone from a large organization, who reads this chapter is at great danger of rejecting SOA out of hand.  You're telling me that SOA is going to improve my organization by adding re-use and handling complexity, and you want me to add a gigantic laundry list of complex technologies (many of which don't yet fully exist) to my organization?  No thanks.  Call me in 10 years when you've got it all figured out.

Now I want to be clear: all of these are important aspects to consider, and many of them will need to be adopted, by very large organizations with huge numbers of services.  But I imagine that most libraries will have a handful of services - the basic functions that the ILS performed, plus a few others.  That doesn't require anywhere near the scale described in Chapter 4. 

I have certainly advocated that the most important piece I think will need to be layered on the base SOA is workflow, the orchestration server (you can listen to my presentation - yes, I will put the slides online eventually too).

I also want to address the SOA Supervisor, and the whole idea of automatically monitoring Service-Level Agreements.  I think this is very deep voodoo.  I am quite happy to be corrected, but I think this capability does not exist in any meaningful, easy-to-use, standardized way today.  To me, for SOA Supervisor, you might as well read SOA Magician, or SOA Magic Box.  It simply is not possible to perform most of the supervisory functions they describe, and even attempting to do so would be hugely complex and vastly expensive.

Chapter 5 - Loose Coupling and Federation

The loose coupling part of this chapter is good, but it really should have been separated out and put after chapter 3.  The Federation part only applies, as is often the case in this book, to very large organizations, that are so complex and diverse that they need to have multiple, federated, SOA domains, with a different set of rules applying in each one - think IBM HR SOA domain and IBM Supercomputing SOA domain.

(Note to CISTI architects: not the same as our domain concept.)

There is a bit of an overlong excursion into software-as-a-service (SAAS).  I think it could have been covered quite well by just saying "you may want to use external services, as well as your own internal ones".

I disagree quite strongly with the concluding section, which asserts that SOA is part of the industrialization of software development.  I certainly agree, and I have in fact asserted that SOA is simply our latest understanding of how best to do software development, but industrialization is way, way too strong a term.  Software is not constrained by the physical world, that's why you can't build software the way you build a bridge or a car.  I'm certain that Toyota knows, second-by-second and gram-by-gram, what is happening as a vehicle moves along its assembly line.  That is simply impossible for software, and it will probably never be possible.

Chapter 6 -  XML

To be quite honest, I don't really understand the emphasis placed on XML within SOA.  Thomas Erl certainly uses it as the keystone of the explanations he gives in his talks and books.  To me, you could get by with two sentences: "Software design works best when you can use standards.  XML provides the ability to create data standards."

This chapter goes into way way more than I think you need to know (for SOA purposes) about XML, SOAP, WSDL, UDDI and even (oddly thrown in) HTTP.  It also seems to express a view of XML as a sort of "magic glue" that ties different programming languages and systems together.  Um, we were able to connect systems together in the bad old pre-XML days.  We called it "using APIs".

Chapter 7 - Dealing with Adapters

This chapter starts out with a strong statement, unfortunately, one I disagree with

Adapters make SOA possible.  No adapters, no SOA -- it's as simple as that.

Um.  No.  I recognize that not all is green fields, but not all is layers and layers of legacy either.  This is coming from big corporation, banking thinking.  It is certainly true that when I enter a number on my bank's customer self-service website, it is going through layer upon layer of adapters, ending up at a mainframe Which Must Never Be Touched.

But for example, library catalogues?  You'd have to be mad to build adapters to catalogues.  Blow them up, replace them with a green field of services.

This chapter also creates a dependency between adapters and service registry and service brokers.  This is unnecessary.  It also continues the early "simple adapter" nonsense, on page 93

Technically, building an adapter isn't too difficult.

Um.  Yes it is.  What they're talking about is API Adapters.  My mainframe interface takes some ancient binary IBM format, and I wrap it with an XML adapter interface.  That isn't too difficult, yes.

But most of the problems facing libraries, at least, is that there is no interface.  There's nothing to easily adapt.  You have to write an entire interface layer.  That is hugely complex.  I have to imagine this also applies widely outside the library technology context.

I have to admit my final written note on this chapter reads, simply: this chapter is baloney.

Chapter 8 - The Registry and the Broker

We're now getting into more and more complex technology, and I'm getting more and more dubious about its actual existence and workability.  To be quite frank, unless I see lots and lots of demonstrations in the real world of this stuff working, I think it is a lot of wishful technothinking.  And I'm not convinced even if it was in place, it wouldn't create a huge agility barrier.

I'm in agreement with the people who say a UDDI service registry simply isn't going to be maintainable in the long term.  You might as well call it "the perpetually out-of-date service registry".

Chapter 9 - Enterprise Service Bus (ESB)

I'll willingly admit to some ignorance here.  We have certainly been discussion ESB at my organization, but we have yet to be convinced that it is necessary.

Again, there is a scale issue.  ESB is basically the reliable message queue idea.  You need a reliable message queue if every (service) message in your organization potentially has huge financial impact - that's the only thing that can justify the cost and complexity of an ESB.  For a big org, or one doing heavy financial stuff, this certainly applies.

To a library, loaning books for free?  If the occasional message drops, I think we can survive.

Chapter 10 - SOA Supervisor

This is currently indistinguishable from magic, which puts the lie to Arthur C. Clarke's famous maxim, because this is certainly far from being an advanced technology.

I'm not even convinced this is an existing technology.

Chapter 11 - SOA Governance

Excellent coverage of the transformational issues raised by moving to an SOA.  A must read for anyone doing a serious SOA initiative.

Chapter 12 - SOA Security

I'm not entirely convinced security is part of SOA, it's a separate service or services to be defined within the architecture.  But the kind of security they're talking about in this chapter is more "organization-wide complexity protection".  Again, the heavy-weight approaches they indicate are appropriate only to large-scale organizations. 

To get a taste, the essential pieces they recommend are:

• identity management
• software and data authentication
• audit trails

If you're not the scale or type of organization that has the above three concerns, you probably don't need to think too much about the type of security they are describing.

Chapter 13 - Where's the Data?

I have to admit a bias - I don't care about data.  I know however, that many EA people, such as Jane Carbone, consider data the essential starting point of analysis.  It is of course important that someone in the organization cares about the data, and this chapter has some useful guidance about the topic.

Chapter 14 - SOA Software Development

In this chapter, we get a little bit of "how".  But it's not really entirely about software development.

For example, I was again surprised to see yet another way to describe Enterprise Architecture, without saying "Enterprise Architecture".  This time they call it "building a business process map".

It then dives into iterative prototype development, which is all well and good, but not essential for SOA.  You certainly could develop services using the waterfall model.  Iterative prototype is certainly a good software development practice, but it doesn't have any strong connection to SOA in particular.

The last part of the chapter discusses Web Services and SOA testing, and this is an important topic which they cover well.

Chapter 15 - The Repository and the Registry

My concerns are as expressed about all these additional layers before:

• this is an important consideration for large organizations
• I'm not convinced anyone has technology that actually does this well in the real world
• I'm not convinced you can use it and still be agile

Chapter 16 - Do You Need a SOA?

Some relevant items for any organization, but some of their questions, e.g. #1 "Is Your Business Ecosystem Broad and Complex" apply again only to large organizations.  I think you can usefully have an SOA even if your business is simple and narrow.

Chapter 17 - Making Sure SOA Happens

Useful information, if a bit generically applicable to any type of major organizational change.

Chapter 18 - SOA Entry Points

Some reasonable guidance.

Chapters 19-26 are supposedly under the rubric "Real Life with SOA".  But I think more accurate would be "here are a series of chapters about major SOA vendors".  I was hoping for real organizational examples, but this section (which runs from page 225 to page 308) is mostly just "here is the SOA stuff from IBM / HP / BEA / Progress / Oracle / Microsoft / SAP / JBoss".  To me, it was a waste of almost 100 pages.  A single page pointing to the vendor SOA websites, with perhaps some of their key articles or white papers indicated, would have sufficed.

I think you've already seen from this review that SOA is a complex enough topic that 80 pages could have been usefully deployed for more explanation and for more discussion of the "how" of SOA.

We wrap up with Chapter 27 to 29, where we get some "Part of Tens" guidance and links, including even more vendor links.  My written notes conclude this part is "vaguely useful".

Overall, the book started very strong, and really could have been for the Rest of Us if they had continued in that vein, but instead it ends up being SOA for Really Big Business.

If that sounds like a good match for you, or if my review has piqued your interest, you can buy the book from my

SOA Book Store (powered by Amazon.com)
or
SOA Book Store (powered by Amazon.ca)

Wikipedia - Book Sources - ISBN 0470054352

review of Lib Tech Reports: Web Services and the SOA

I am always looking for information about technology that is library-specific or that has a library perspective.  For Web Services and Service-Oriented Architecture (SOA), there are not many resources of that type.  I was interested therefore to read

Library Technology Reports, May/June 2006, Vol. 42, No. 3, ISSN 0024-2586
Web Services and the Service-Oriented Architecture by Marshall Breeding

There are three main sections in this 52-page report.  The first provides Basic Concepts, the middle provides a brief intro to Google and Amazon Web Services and how to write Perl code to use them, and at the end Breeding looks at what library groups and technology vendors are doing with Web Services.  Although there a brief mention of SOA, I wouldn't consider this an intro to SOA; it's basically entirely about WS.

It starts off strong, the Basic Concepts description of Web Services from pages 6-9 is very clear and gave me a lot of useful ideas to use for explaining WS.  After that it starts to get quite technical, providing brief intros to XML, SOAP, WSDL, UDDI and so forth.

For the programming section, I'm not so enthusiastic.  These concepts are so complex that they really need a combination of Internet and book-length resources.  This report will give you at best a tiny and perhaps confusing taste of Web Services coding.

The last section tries to cover Web Services in the Library Environment, but hits some challenges because the topic is so new.  There is a discussion of VIEWS and the NISO working group it turned into.  Then there's a problem because there aren't really many Web Services running - he covers SRU/SRW and SUSHI, but there's also a bunch of non-WS library tech thrown in.  He ends by surveying library technology vendors on their use of Web Services.

(Note: I will review the NISO Web Service guidelines in my next post.)

My opinion overall is that it was a worthy effort, but it tries to reach too many different communities.  The mix of basic concepts, coding, and industry survey ends up delivering three quite disparate pieces, each of them to some extent receiving short shrift.  I could see a CIO benefiting from the basic concepts and library environment overviews.  For most other staff, the basic concepts gets way too technical.

The coding section I think is the weakest, simply because there is nowhere near enough space to devote to this topic.  A programmer would be much better off getting some O'Reilly books (and/or a Safari account) along with using web examples.

The section on Library Web Services is ok, but it wasn't much use to me since I already independently gathered much of the same information, and more.  (See library Web Services - are we making any progress?)  I was surprised that there was no mention of the Digital Library Federation services initiatives.  The vendor survey was a bit useful if you're vendor-bound, but I don't think we can look to most vendors to innovate in this area.

I think it would have been better to spend more time on the basic concepts, and talk a lot more about SOA.  This guide really doesn't provide any intro to SOA, it's all about Web Services.  So I would have cut out the coding part entirely, except for maybe a brief intro with pointers to web resources, and expanded the basic concepts section, perhaps with some diagrams (this report has only text).

Overall I was a bit disappointed, at US$63 for 52 pages, this is a pretty expensive document.

SOA BOF - links and thoughts

I had a small but useful BOF on SOA at Access 2006.  Thanks to Glen Newton for posting the BOF, via the magic of blog communication, and thanks to Alison for trying out the nano-BOF-introduction idea I suggested.

Anyway, here are some SOA starting points, from my participation in an SOA Symposium last year

• bookmarks - http://www.connotea.org/user/scilib/tag/soasym2005
• presentation - Service-Oriented Architecture Methods to develop Networked Library Services (PowerPoint)
• video - me doing the presentation (RealVideo)

In terms of my thoughts, I think it's great that Access brings together people from so many different perspectives and backgrounds, all with some connection to library technology - this is silo-breaking as it's meant to happen, but that is also a tremendous challenge.  We have people who come to coding through a formal CS background, people who come to it from a lifelong interest in hacking, people who arrive at it through scripting, and people who manage coders without having ever written code.  That's a really diverse range of backgrounds to try to address.  I think there is a balance between "just getting it done" and some of the higher-level structures and concepts that can be provided by software engineering.  I will have more to say about this when I do a posting about my Friday afternoon "SOA / Web Services / Lego / BPEL / Framework" presentation.

My colleague Stephen Anthony was as always helpful during the BOF and during other discussions, in particular he pointed out to me that I need to find some better terminology for "rapid but non-architecturally-guided" development.  I had been using "Agile" as a shorthand, but to most people that means the (excellent and used by CISTI) concepts of Agile software development.

The following previous posting will lead you to links for all the SOA Symposium videos and presentations: March 13, 2006 - video from Manitoba SOA Symposium.

BOF on SOA at Access 2006?

Anyone interested in a BOF on Service-Oriented Architecture at Access 2006?

I don't know how the Access BOFs work, but I do see there is a slot

Thursday, October 12, 2006

5:00pm - 5:45pm     Birds of a Feather
7:30pm -     Improv Night at the Velvet Room

With so much excellent thinkage on SOA, it would be nice to see what info can be exchanged.
In case you're wondering, CISTI is already developing its services within an SOA today.

You can read my Service-Oriented Architecture category for more info, it has the past two years of my links and thoughts on the topic.

UPDATE 2006-10-15: I held the BOF, you can read my report.

ECDL 2006 - DLSci06 - DILIGENT grid-based DL

Leonardo Candela, Donatella Castelli, Christoph Langguth, Pasquale Pagano, Heiko Schuldt, Manuele Simi and Laura Voicu
On-Demand Service Deployment and Process Support in e-Science DLs: the DILIGENT Experience

http://www.diligentproject.org/

(Also see notes from tutorial Distributed Infrastructures for Digital Libraries.)

Motivation

* research is multidisciplinary and co-operative effort
* may use a virtual resource organization that doesn't last a long time or have DL expertise
* but the DL is an important tool

DELOS view: from DL to Knowledge Commons
* from content-centric to person-centric
* from info storage to communication and collaboration support
* from centrally-located text to distributed and heterogenous data sources

New DL development model
* DL built by dynamically aggregating the needed resources
* new functionality combined in user-defined workflows

Service-Oriented Architecture over Grid Framework

[complex diagram]

* New functionality delivered by workflows of services

Services Overview

* Mediation
* Information Space Management
* Access
* User and Resource Management
* Presentation
  - user-oriented access point to the DL
  - plug and play community-specific tools
  - [something about JSR168 portlets and other stuff]
* Enabling
  - monitoring, other operations domain functions

Service Detail

* the Keeper Service
  - deploy and monitor user-defined virtual DLs

* the Information Service
  - gathers, stores and supplies information about the resources constituting DILIGENT
    and needed to the other services
  - XML-based resource profiles
  - push and pull modalities i.e. query and subscribe/notify

Process Design and Validation

Implement complex services by combining existing ones, a.k.a. "programming in the large"
* control flow
* data flow
* transaction behavior and execution guartees for concurrency and failure handling
* XML, SOAP and WSDL as technologies
* BPEL as a foundation for process specification

[diagram of tool]

Process Execution

* built on top of OSIRIS
* runs BPEL tasks?