Science Library Pad

Enterprise Services Architectures - Chris Mackie - DLF Fall Forum 2007

I was privileged to have been paired with Chris Mackie of the Andrew W. Mellon Foundation on the DLF programme, and fortunately I think our talks were quite complementary.  Chris talked about Enterprise Services Architectures (ESA), presenting a very clear-eyed view of the sustainability challenges of IT infrastructure for higher education, and how ESA and some specific Middleware ESA (MESA) projects are addressing those challenges.  His presentation (PDF) is now available from the DLF site.

presentations from DLF Fall Forum 2007

Some of the presentations are posted on

http://www.diglib.org/forums/fall2007/2007fallprogram.htm

If you want to locate them, just search for [PRESENTATION]

Stanford Library Cyberinfrastructure - DLF Fall Forum - Nov 7, 2007

I would really call this more a Service-Oriented Architecture to support advanced digital library activities.  Definitely some interesting work, a good contribution to library SOA.

Defining and Designing a Cyberinfrastructure for the Library of the Future
Tom Cramer, Lynn McRae, Rachel Gollub
Digital Library Systerms & Services
Stanford University

* design pattern & process

Environmental Scan at SULAIR
1. Digital preservation
2. Google Book scanning
3. Internal digitization
4. Discovery
5. Content delivery
6. Ongoing proces re-engineering

[diagram - The Digital Library: Content, Services & Infrastructure]

- content management & content infrastructure

content streams->digital library holdings->services
supported by content management & middleware plus SDR (Stanford Digital Repository)

The Challenge

Analytic Environments, Delivery Apps, Discovery Apps on top of many other components

Strategy 1: Hardware Approach - ILS (nope)
Strategy 2: Repository-centric (nope - SDR is preservation infrastructure)
Strategy 3: MacGyver It (duct tape)
Strategy 4: Library Middleware
- Content Registry
- Collections Registry
- Access Broker
supported by
Reporting, Intelligence, Monitoring
"Content & Service Middleware"

Parallel to Identity Management - in enterprise computing, we have patterns that we use

"identity management for digital content - content registry"

The process

1. Recognize need
2. Think through narratives
3. Idenity parts
- Applications
- Services
- Infrastructure
4. Assemble in an architecture
5. diversion into ESBs
6. Validate concepts & find a name

What's the Need? The Gap to fill?

what it's not
- not an ILS, nor a new catalog, nor a repository, not protocol or standard, not discovery nor delivery apps, not user-facing

what it is
* infrastructure underpinning applications
* design pattern & architecture

Narratives

1. Support for scholarly workflows
2. Personalized academic work environment
3. DIY Library/Research Environment
4. Integrated, comprehensive content discovery
5. Creating, managing, publishing dynamic digital collections
6. Extend library infrastructure & workflows to already digital content
... more

1. Scholarly workflows

Support collaboration, research and publication through a full information and creative lifecycle

[list of use cases]

extract set of capabilities from use cases

2. Personalized Academic Work Environment

Highly personalized services and resources based on persistent and intimate knowledge of the scholar's identity,
roles, background, explicity choices, and implicity preferences

* Endow all services with an awareness of personal identity, preferences and contributions

3. The DIY Library/Research Environment

Enable scholars to use the tools of their choice to incorporate library data, metadata, and complementary services
into their workspace

* APIs to content, metadata, libraryservices

4. Comprehensive content discovery

Ability to discover across all content stores

...

8. Library management, operations, reporting

internal monitoring abilties

Five themes derived from narratives
1. Identity
2. Preservation
3. Personalization
4. Access
5. Management

bound together in a common infrastructure

Services, Infrastructure

Translating function into architecutre

[diagram]

SOA on ESB

Content Management
* data/metadata stores
* data/content sources

Services -- Preservation & Management
Services -- Access/Identity/Peronalization
Infrastructure services

metadata store, "a bunch of XML files", sitting on the ESB
- including plugin service API, management services API

ESB gives you a bunch of functionality, Fedora gives you a bunch of functionality...

"Lyberstructure"

ILS Discovery Interface - DLF Fall Forum - Nov 6, 2007

I'm not sure I really grok this idea of bindings, it sounds like they run the risk of ending up with a catalog of specialised implementations, rather than a unified API.  But it is very much needed work, and we do have to be practical.  There was a follow-up discussion session, but I missed it.

ILS Discovery Interface (ILS-DI)

http://project.library.upenn.edu/confluence/display/ilsapi/

* We need Integrating Library Systems
* We need practical solutions soon
- The ILS may undergo radical redesign... but our users won't wait for that, and we can't

Scope

Out-of-scope
- acquisitions
- cataloging integration
- item management

in-scope
- patron-driven discovery from search to use
  -- finding relevant resources (discovery)
  -- acquiring them (delvery)
  -- managing their usage (patron info and account)

You are here: Rough draft recommendation being presented and discussed

Early 2008: Formal recommendation to be released

Beyond? Recommendation will be updated as new technologies, functions, tools emerge

Survey on Current Use of OPAC and Beyond

Current use
- majority considering replacing ILS in next 2 years
- widespread frustration with OPAC interfaces, metadata schemes, resource scope
- generally ok with ILS inventory management functions

Beyond the OPAC
- 3/4 using supplementary discovery apps
  -- many locally developed
- wide variety of interactions with OPAC
  -- data export most common

a lot of duplicate work being done on extracting info from ILS

1. Improve discovery by supporting interfaces to ILS
2. Articulate a clear set of interaction expectations for ILS and app developers
3. Making recommendations applicable to a wide variety of systems and technologies
4. Make recommendations that are feasible to implement
5. Work with applications beyond "traditional library" domain
6. Be responsive to the user and developer community

Functions
- areas of interest: data aggregation, real-time search/query, delivery, patron info & services,
OPAC embed / escape / entry

Bindings
* Specific technologies that implement desired functions

Data Aggregation (getting the data out of your ILS)
- GetBibliographicRecords
* want to have selective export (by date or record type)

Real-time search/query
- GetAvailability
- Search
- Scan
- SearchAuthorityRecords
- ListCourseReserves
- ID-based record retrieval

Delivery Functions
* Holds
* RecallItem
* Security, policy issues can be complex for these functions

Patron information and Service Functions
- RenewLoan
- CancelHold

...

OPAC embed/escape/entry

* OutputRewritablePage
* OutputIntermediateFormat

Getting to an official recommendation

* Make sure we're not missing any essential functions
* Get more specific on functions
* Also find or point to specific bindings of the functions
* Try to finish in reasonable time
* Now is the time to encorage implementor involvement

How can my solution be incorporated?

- show the group how your solution works as a binding for an abstract function we've specified

- it has to be openly and fully specified

* it ideally has service and client implementations in product

- it also helps to have
  -- no IP encumbrances

Beyond the recommendation

* periodically update
  -- is there a sustainability model?

my presentation on SOA for libraries at DLF Fall Forum 2007

Overall I think it went very well, the room was packed (to my considerable surprise) and I got good questions from the audience - for once actually too many questions for my allotted time, so I unfortunately stole 10 minutes from Chris Mackie, whose presentation followed mine.

The slides are up on Slideshare, they were done in Apple Keynote, in case you're wondering.

http://www.slideshare.net/scilib/serviceoriented-architecture-for-libraries/

All of the links and background info for the presentation are available at

http://www.connotea.org/user/scilib/tag/dlf2007akerman

The following is not the actual narrative, but it gives you enough to follow the (mostly conceptual) slides:

I started with a great quote via Katherine Kott of DLF Aquifer, from the previous night's session on collaboration, stating that we need "vision translated into operational success".

Service-Oriented Architecture addresses problems in technology planning,
design and implementation.
First, we must understand what some of these problems are.
Fundamentally, technology implements the desires of the organisation.

Too often we focus on our immediate desires ("What do you want") rather
than long-term planning.
This has meant we get what we asked for (e.g. incremental catalogue
interface changes) rather than what is best for the organisation
(transformed catalogue)

In order to better align our implementations with our goals, CISTI uses
an SOA derived from our Enterprise Architecture.  In our EA methodology,
we start with business needs (Who are you?), and only then proceed to
identifying the gaps to be filled.  But you must look beyond the current
state (Why are you here?  Where are you going?)

So what are the transformational challenges we face?

In a way, we are in a fantastic world for the traditional goal of a
library - ubiquitous, zero-cost, perfect copies of information.  No more
scribes devoting their lives to copying a few books.  But this has
tremendous implications in terms of preservation, versioning, citation
etc.

Also, there are major changes in moving from a physical world where we
engineer bridges, to a virtual world where we "engineer" software.
Software doesn't have most of the constraints that apply in the physical
world, and we have to be careful with analogies to physical objects.  It
would make very little sense (except perhaps to a US Senator) to build
half a bridge, or a bridge that goes nowhere.  But in the software
world, building components, even if they are only part of an overall
goal, can be extremely useful.  And software is not location constrained
- it can be accessed from any location - there should be no "software to
nowhere".  The network is everywhere, and we need to understand how that
transforms our systems design.

So how do we apply the notions of architecture that we understand from
the physical world, and that have for centuries and continue today to
enable us to build very complex structures with many components (heat,
lighting, electrictiy, plumbing, etc.)

Well, we can use the idea of modeling.  Just as we make models of
physical objects, we can make models of business processes.  But we must
ensure that we don't get stuck in our models or frameworks, and do move
to implementation at some point.

CISTI has found that its methodology, which is focused on getting to the
implementation framework, and beyond that to design and development, has
been invaluable for turning business ideas into implemented software.
We are fortunate in that we are a library with substantial in-house
technology resources, including a dedicated architecture team (the only
library EA team in the world??)

SIDEBAR: I met the Head of eArchitecture for the British Library at my presentation, so now I think I should say "one of the very few library EA teams in the world".  END SIDEBAR

This structure may be different for
academic organisations, with faculty groups that have departmental IT,
central IT, and academic consortia.

Within this larger mehtodological framework, SOA is an approach to
identifying "servicifiable" functions in the organisation, and building
larger service offerings from various combinations of services.

In the architecture analysis phase, CISTI models all of our
technology-related activities, at a high-level this model describes all
the technology-supported activities, with functions such as "provide
library service" that then decompose down into many many lower-level
models.

Recently we have been specifically modeling a Trusted Digital
Repository, using the OAIS model and converting and adapting it.

This EA modeling work then leads to the derivation of SOA services
[depending on audience profile I may talk a lot on this slide about the
details of SOA, or not].

CISTI has used and is using SOA successfully as the foundation of
various projects - this is not just theory.

SOA is also not an agility-killer bureaucracy - services actually
provide pieces that can be used for small experiments, while
simultaneously moving the organisation in its stated general direction.

Ideally, we should all be collaborating on SOA, rather than building SOA
silos.  I have blogged a lot about this, but with little response.  Are
the individual framework groups only talking amongst themselves?  How
can we use their efforts most effectively?

SOA is certainly being explored in many areas of the library space,
including various digital library initiatives and modeling efforts (some
of which will be presented in other forum sessions).

SOA also offers the possibility of improving the catalogue by breaking
it into component services, and enabling the sort of simple layers
(widgets, mashups, javascripty stuff) by providing sustainable,
well-architected underlying service capabilities.

For academic libraries in particular, SOA offers a compelling
opportunity to connect with existing and future scientific activities,
as many Cyberinfrastructure projects have SOA at their cores.  Can we
use modelling at even the highest level (the entire scientific
communication cycle) to guide our service implementations?

Ultimately to build sustainable, non-siloed, modern information systems
to support scholars and citizens worldwide, we need to build many
bridges (this one happens to be built of lego components, in case you're
worried that giant ducks have invaded the US).  Between libraries and
scientists, between technology and business, between funders and
software developers, between computer science and library science...

We need to move beyond frameworks, using solid governance to ensure that
SOA plans become running SOA code, while avoiding getting stuck in giant
projects that don't deliver.  As well, even if you don't have the scale
or the capacity to build your own SOA, you can still participate as a
service consumer.

I'm not crazy in this (at least I hope not), as evidence for which I
present some starting points about SOA in the academic and library
sectors.

Questions...

The End

SIDEBAR 2: In case you're wondering, there's a slight science fiction theme running through the presentation, inspired by the "federation" part of the organisation name.  The Four Questions are from Babylon 5, the goodish aliens, the Vorlons, ask "Who Are You?" and represent the forces of order, the evilish aliens, the Shadows, ask "What Do You Want?" and represent the forces of chaos.  "There Are Many Copies" is from the opening sequence in the new Battlestar Galactica, using the great Creative Commons images I found worked out as a nice way of avoiding using a screencapture (plus using a screencap might have been a bit distracting as it shows a scene from BG where the many copies of Sharon Valerii/Number Eight are not so much wearing any clothes).  I had a better closing quote but I couldn't remember it, so I used a tag line from Buckaroo Banzai Across the 8th Dimension, to close out the "where are you going" questions theme.  END SIDEBAR 2

SIDEBAR 3: I was actually amazed to be able to locate the "bridge to nowhere" Google Earth image.  I had seen the unfinished highway bridge out of the airplane window as the plane was coming in to land in Philadelphia and by the time I thought to GPS-mark it or take a photo it was already past.  Incredibly, doing an image search on

"bridge to nowhere" Pennsylvania

brought it up as the first hit - Google Search, Google Earth and the Internet continue to amaze me. END SIDEBAR 3

SIDEBAR 4: I bought my carbon offsets from MyClimate.org

flight from: Ottawa, ON [Macdonald-Cartier International Airport], Canada, YOW
flight to: Philadelphia, PA [Philadelphia International Airport], USA, PHL
flight via: Toronto, ON [Lester B. Pearson International Airport], Canada, YYZ
return, economy
flight distance: 1'845 km
flight passengers: 1

CO2 Emissions: 0,511 t

Total costs for compensation of your flight: SFr. 20.00

According to Google

20 Swiss francs = 16.3543582 Canadian dollars

Pennvibes - DLF Fall Forum - Nov 6, 2007

PennVibes

Currently the production & maintenance of library web pages is labor intesive, involving:
* content specialists
* web production staff
* IT staff
* hand-coded HTML
* adding any kind of tools to a page requires special code

This results in a very limiting economy of production:
* we can't modify our pages too often
* we can't have to many pages

Enter Web 2.0

* Netvibes, PageFlakes, iGoogle

* pages are custom-tailored to the user's needs

Can this Web 2.0 model be of use in a research library context?

* yes if
- we develop library-oriented widgets
- if we build a framework that is community oriented (e.g. for a librarian to build pages for
patron use)

Enable a new level of service
- e.g. microsubject pages

[PennVibes Demo]

main focus is on rapid web page development - even to a web page developed to respond to a
specific patron request

Next Steps:

* Release Into Production
* Goal: Progressively replace core library web pages with Pennvibes pages

Longer term:
* more customizable pages
* group maintenance of pages
(but these customizable & maintenance features requires significantly more infrastructure)

* widget interoperability
e.g. Librarian adds a PageFlakes widget to a Pennvibes page and vice-versa

* need for industry standards

* a pool of library widgets from many organisations?

Zotero 2.0 - DLF Fall Forum - Nov 6, 2007

raw presentation notes

Zotero

Trevor Owens

"The Fluidity of Bibliography"

Smart Bib: easy metadata embedding
(coins embedded)

Read Write Bib (using Wikipedia exmaple)

Visualizing Bib (using Simile Timeline)

Smart Bib:
demo of grabbing embedded bib info from a web page

Zotero 2.0 - Zotero Server
* shared collections and notes
* scholarly groups in macro- and micro- disciplines, official groups
* recommendations
* bibliographic feeds
* APIs

Q: Where is the server
A: One server, at Zotero (Center for Media), closed source

Q (me, paraphrased): Will there be server API usage limitations?  We might like to send queries to retrieve recommendations every time someone does a search, that could be a lot of traffic.
A: TBD... it will be mostly constrained by server performance and capacity

NCSU CatalogWS API - DLF Fall Forum 2007 - Nov. 6, 2007

Very interesting presentation by NCSU on their CatalogWS Web Services API they have build on top of their Endeca layer.

Here are my raw notes

Library Catalog as Versatile Discovery Platform

UPDATE 2007-11-08: [PRESENTATION] ENDUPDATE

Next Generation Catalogs

Library Technology Reports - Next Generation Library Catalogs

Examples
* Endeca (NCSU Libraries)
* AquaBrowser
* Encore (UQueensland)
* ExLibris Primo
* WorldCat Local
* Solr (UVirginia Project Blacklight)
* vufind (also Solr powered)

Modern search
- relevance
- faceted
- tag clouds
New content
- user contributed content
? enriched content
? content or context awareness?

Focused on optimizing a single discovery context... the OPAC

Question

Why should the discovery of catalogued library collections be limited to user interaction
with a single catalog application?

Catalog as Discovery Platform, beyond the OPAC

Web Services: CatalogWS API

Platform

"a platform is a system that can be reprogrammed and therefore customized by outside developers..."

- quote from Marc Andreesssen

Discovery Happens Elsewhere

"No single website is the sole focus of a user's attention."

- quaote from Lorcan Dempsey

Platform Motivation

* move beyond the one-size0-fits all approach
make it easer to reuse and repurpose catalogue data outside the opac
* build catalog interaces optimised for different use contexts

CatalogWS API

Goals
- can we have RSS feeds for the catalogue
- can we integrate catalog results into library website quick search

Final result
* rich API

[diagram of architecture]

API implemented as layer on top of Endeca

Limitations
- subset of catalog data
- read-only
- not real-time

Technical Design
- RESTful
- Java, Tomcat, XOM, Saxon 8.8, JSON

http://www.lib.ncsu.edu/catalog/ws/

* discovery-oriented
* catalog availability (known item lookup isbn)
* catalog search (both known and exploratory searching)

can pass a style parameter (URL of XSL), for server-side XML transformation, or XML to XHTML

Why new XML schema?
- include as much of data as possible in reponse
- MARC XML and MODS didn't appear capable of capturing the varied data
- XML response includes links

Demo of Current CatalogWS Apps

* integration with external apps
- Quick Search (NCSU library website search)
- iGoogle Widget

* alternative catalog interfaces
- mobiLIB catalog
- facetbrowser

Collection Promotion
- FacetBrowser - ability to easily create blog entries for selected items
- automatically generated "bookwalls"
- RSS feeds for new titles

http://www.lib.ncsu.edu/dli/projects/catalogwsapps/

DLF Fall 2007 blogs and bookmarks

Since unfortunately our search engines still tend to fail us for rapid indexing (I'm looking at you, Technorati) I've bookmarked a few blogs I managed to find of people at DLF Fall 2007.

http://www.connotea.org/tag/dlffall2007

Incidentally, speaking of bookmarking, Ian Mulvaney of Nature will be talking about Connotea tomorrow (Wed. Nov. 7, 2007) in the Grand Ballroom as the 2nd presentation in the 11-12:30 slot.

DLF in Second Life

There will be a session on the Digital Library Federation in Second Life at the forum today (Tues. Nov. 6, 2007), it's not in the programme, it will be at 5 PM, I'm not sure where in the Pine Room on the 3rd floor.

You may be able to join it within SL as well, but I couldn't find the DLF in SL, anyone know where it is? (Remember that for SL you have to give both the island name and the x,y,z as coordinates are not globally unique, they're per-island.)  You can send me a teleport (Rakerman Yellowjacket) or use that 20th century technology, email.

UPDATE: The island is called Entropia, you can't find it yet through a places search, but if you bring up the Map and search within that interface, it will come up.  ENDUPDATE

You can see Peter Brantley's posting DLF moving into Second Life for some background.

DLF twittering

For what it's worth, I am as usual twittering the event (DLF Fall Forum 2007), at

http://twitter.com/scilib

Internet Archive 20th Century Search - DLF developers' preconference - Nov 3, 2007

Before the main event there was a preconference (with a fraction of the main attendees) exploring technical challenges and possible collaborations.  There were also a couple brief presentations, here are my raw notes on Kris Carpenter Negulescu, the Director of the Internet Archive Web Group talking about

"20th Century Find" using Amazon S3 & EC2

Internet Archive stats

3.5 PB
1.5 million downloads/day

this project is about providing full-text search of their web archive for the 20th century 1996-2000, ~22TB

NutchWAX = Nutch + Hadoop
focus moving to Nutch with plugins

Amazon S3
Amazon EC2 (beta)

there are now more EC2 node options: small (default), large, extra large 8 times small performance and better I/O ($0.80/cpu hour)

indexing began in October 2006
1996: indexed via 20 EC2 in ~36 hours
1997: 100+ EC2 nodes
1998: 300+ EC2 nodes

1999 was attempted in September 2007 using cluster of ~270 EC2 nodes but halted due to lack of
consistent CPU/IO across nodes.

deployed (alpha) index is 1.35TB in size, no compression, ~600 mill docs

Enhancements
* multiple instances of a page
* improved ranking of results
* handle dimension of time
* easy UI

Why Amazon Web Services
* pay as you go
* simple to provision
* committed to support
* indeal for indexing Web pages, providing offsite storage, reliable hosting
* great platform for experimentation, iteration
* geographically disperse from Internet Archive ?Data Repository?

Cost Effective, budgeted $20k

* Note: fees can add up fast if not vigilant

Working Well

* APIs
* tech support
* S3
* fee structure
* speed of provisioning
* S3 uniformity of nodes

Challenges - S3

Oct 2006 - June 2007

* (internal?) bandwidth availability
* no specific guarantees for data preservation
* issues related to popularity of the service

Fall 2007

* available bandwidth consistent (~4h to move 7.5TB into EC2)

Challenges - EC2

lots of issues Oct 2006 - June 2007
* location of S3 nodes relative to EC2 was a significant factor for large-scale data processing

July 2007 - present

* working well but hitting IO and CPU constraints on small (basic, default) nodes;
however will continue to use these small nodes

Consider Using AWS When

S3
* need cost-effective backup for data
* multi-provider preservation, geographically diverse

EC2
* if you have spiky computing needs (e.g. spikes in demand)
* you have available R&D resources

Will experiment with AWS for crawling and harvesting, starting Jan 2008, Heritrix/AWS.

DLF Fall Forum 2007

The Digital Library Federation (DLF) Fall Forum 2007 will be November 5-7, 2007 in Philadelphia.  I just realised I haven't written anything about it.

I am scheduled to present "Service-Oriented Architecture for Libraries" on Tuesday 6th.

For blogging etc. I will be using tag/category DLFfall2007.

I'm interested to go because although I have blogged a number of times about the DLF, I haven't actually been to a DLF event before this.

Previously:
July 24, 2007  software development, staffing and new library technology
March 30, 2007  pondering SOA the UK and Aussie way
January 30, 2007  report on Library Future Roles webcast - in which I ask Peter Brantley a question
April 3, 2006  standards and discoverability for library Web Services - DSLR Workshop
September 6, 2005  library service frameworks: avoiding wheel reinvention