Science Library Pad

NRC jobs - Manager, Enterprise Architecture

National Research Council Canada

Ottawa, Ontario

CS-4, bilingual imperative CBC

closes December 20, 2016

This is a continuing position.

Apply at the bottom of the job poster http://www.nrc-cnrc.gc.ca/careers/jobpost.nsf/EnglishAll/7E62A0D17E2BC2758525804F006C15CC

Education

Bachelor degree in Computer Science, Computer Engineering, Information Technology or Information Systems.

Experience

Extensive experience in the IM/IT field, including areas such as systems analysis, systems implementation and software development life cycle.

Significant experience in identifying a Department’s IT needs and technologies that could address policy, program and organization needs.

Significant experience in designing and putting into effect the IM/IT architecture for a department.

Significant experience in advising senior management on the strategic planning of IM/IT architecture.

Significant experience in the supervision and direction of staff.

Preference will be given to candidates having provided services in a decentralized environment and within the context of the federal government.

20th century organisations with 21st century challenges

In general, people want their organisations to do wonderful things. Yet even when organisations are staffed with individually excellent employees, often the result is less than the sum of its parts. Or in other words, "Why can't my organisation do x?"

You can think of this as the lifecycle of an idea.

idea -> implementation -> operations

The above of course only works if you have infinite resources, to take the vast number of ideas, implement them all, and maintain them forever.  Since this is impossible, instead we try to talk about this using an analogy to living things, e.g. from seed to growing tree to mature tree to chopped down tree.

There are many different angles on this, in technology commonly we talk about the Software Development Life Cycle.  Unfortunately in most organisations, this is actually the Software Development Lie Cycle, for many reasons.

First is that the above line from idea to operations is woefully oversimplified.  It's actually a loop with current state turning into future state (which becomes the new current state)

(entire current organisation and current technology and idea environment) -> new ideas and improvements on current systems -> gateway -> prototypes and upgrade projects -> gateway -> new and existing operations (future state) -> death gateway -> decommisioned systems

There are at least three problems that organisations encounter as they go around this loop

1. The processes are bad at each step

2. The staff and management are unable to properly execute the processes, even if the processes themselves are good

3. The entire world changes while you're trying to move from your fixed current state to a wishfully-fixed future state

Specifically when it comes to information technology, big organisations are bad at IT.  Also small organisations.  Also software engineers.  Also, well, basically everyone.

The good news is we have a handle on some of the main problems.  In fact we've been trying to find solutions to these problems for decades.  The bad news is that solving these problems is very hard, and so quick fixes rather than real fixes are very tempting.

One of the underlying issues is the overall factory model we have both for producing and managing.  This is very deeply embedded both in every organisation as well as in the assumptions of every employee and manager.  It is hard to break out of a century-long mindset; in fact it's hard to realise you even have that mindset.

Software engineers have tried to attack this problem from all fronts: better languages (e.g. Ada from the US Defence Department), better methodologies (e.g. agile instead of waterfall), better overall context (e.g. Service-Oriented Architecture), better component sharing (e.g. github).

All of these are challenging because of the underlying factory organisation model.  For large organisations in particular, certain approaches pull the entire organisation as if by magnetism.  In particular, organisations will tend to do

* big projects (in people resources, dollars, and time elapsed)

* complex projects (e.g. many different technologies to be integrated)

* ambitious projects (e.g. trying to fix many problems at once)

* business-incremental projects (i.e. to add new features to a current process or product offering)

* perceived low-risk projects (e.g. with elaborate cost analysis and justifications)

* poorly-communicated projects (management doesn't communicate their intentions "down", IT doesn't communicate complexities "up" and comms doesn't talk to EA which doesn't talk to the coders who don't talk to... etc.)

Also business tends to measure certain costs while ignoring others (typically capital expenditures and coding time are carefully tracked, but having endless meetings is often considered to have zero cost).

This force pulls on all initiatives, pulling them towards having more stakeholders, more committees, more senior-level buy-in, more analysis, more consultations.  And once substantial effort is invested, the "sunk cost" psychology means that it is almost impossible to stop a running initiative, or to turn off a running system.

This force pulls towards bigness and complexity in general - towards big bang Enterprise Architecture, elaborate Service-Oriented Architecture, big-iron computers and networking hardware, "enterprise-class" heavyweight software backed by giant consulting firms... organisations somehow believe if they can just do big enough things with big enough technology, their problems will be solved.

There are many, many reports over decades that have documented this.  A recent one is System Error: fixing the flaws in government IT from the Institute for Government in the UK.

Pretty much everyone wants to fix this.   (Of course consultants may in some cases have perverse incentives to perpetuate dysfunctional processes that generate a lot of money for them, and managers may want to retain processes that give them power, but in general people want to do good work.)

One key problem is that usually people don't take a systems view.  They identify one problem area in the cycle and want to attack just that, thinking that will solve the entire problem.  For example:

* if we could just generate more / better ideas, everything else would work

* if we had a better planning process, everything else would work

* if we did more prototypes, everything else would work

* if we had more hackers, everything else would work

* if we changed development methodologies (e.g. to agile development), everything else would work

The reality of course is that unless you attack ALL steps in this cycle, the end results will always be, as they say, hashtag #fail

And even if you do each individual step well, you may still be choosing to do the wrong things.

Let's take a typical example.  Your organisation has an existing telephone system (PBX with voicemail) and knows to the penny most of the costs (in reality it probably knows most of the clear expenditures but has a more hazy idea of systems maintenance costs and opportunity costs).  Someone, probably at a senior level, has read about Voice over IP (VoIP).  Investigation initiated.  RFP written.  Consultants consulted.  ROI laid out.  Network upgrades needed.  Hardware to purchase.  Desktop software to integrate.  Options to evaluation.  Choices are made, purchasing, coding, implementing... a nice project that takes a year+ to replace a telephone on an employee's desk with... a telephone on an employee's desk.  Everyone happy.

Except in all this process no one stepped back to say: do we need the telephone at all?  Does anyone like using the telephone?  Are there other cheaper solutions?  What if we got rid of telephones altogether?  What do telephones do to knowledge work?

In fact, you can find whole threads of productivity literature that will tell you that telephone interruptions disrupt knowledge work, and that other methods of communication may be more effective.  In parallel the trends are clearly towards a global decline in voice traffic and a huge increase in data traffic.  People are not talking on their cellphones, they're texting and Facebooking.  Cartoons are appearing about the "phone gap", where texting users expect to receive a text asking whether a call can be made, rather than the phone just ringing out of the blue.

Over a decade ago in The Time Trap, Telephone Interruptions get an entire chapter

There is something wholly irresistible about a ringing telephone. ... Yet tune it out you must, for telephone interruptions can shatter your concentration and fracture your productivity as nothing else can.

(In fact the solution to this is quite simple, it just requires throwing out 20th Century convention and turning the ringer on your phone off, which is what I do.  My work phone and Blackberry never ring.  Although mostly these days they never ring because no one uses the phone any more.)

All this to say, even if somehow every stage of the giant VoIP project was perfectly executed, they still would have done a good job building the wrong thing.  But in general of course, every single stage is badly executed.

So you need a plan of attack for every stage.  At every stage you need BOTH better process and better-trained capable people.  Like a bike lane system which is barely used at 98% complete but starts to hum when every pathway is connected, you only get the big improvements when every stage of your ideas cycle works well.  In addition, we need the ability to continuously step back to the big picture and decide whether an entire initative still makes sense in light of changing conditions.

So you must

* know your current state well (most organisations have no idea what their actual processes are and are often shocked to see ridiculous legacy processes once they're diagrammed on a whiteboard)

* have a good process for gathering and generating ideas (often perceived as a major barrier, this is actually quite straightforward - the issue is rarely a shortage of ideas)

* have a very good process for gatewaying ideas to the investigation stage (making sure to let through both some high-risk prototype ideas and some improve-the-business ideas)

* be very very fast and agile at cranking out high-risk prototypes

* have a very good and very strict process for gatewaying prototypes into full implementation (sunk cost and idea attachment tend to make it hard to kill a running system no matter how many times it was emphasized that it was temporary)

* have a short process to take very focused features to production (e.g. a few months for a few very specific features)

* be adaptable as everything changes along the way

* have a process to kill things from production once they're no longer relevant (this is like the prototype killing problem, except multiplied by 100 as there may be decades of sunk cost)

* have a process to continuously review running initiatives and kill them (or modify them dramatically) if the world has changed while they were running

* Start tracking real employee costs not just specific activities and expenditures.  If you have to have a committee and burn $5000 of people time to decide to spend $500 on an Internet tool license, you have a costing misallocation problem.

At every stage just the process alone won't do it, you also need people who are creative, well-trained, adaptable, communicative, sharing their knowledge and documenting what they're doing as they go along.

So for example you might:

* immediately document your current systems (enterprise architecture)

* introduce an environment scanning process (and/or partner with existing horizon-scanning initiatives)

* devise a ruthless gating process that senior management can implement with input from all staff

* reward success and failure - reward successful products but also celebrate the lessons from prototypes that don't pass the gates

* create a prototyping group (innovation team, tiger team, skunkworks etc.) but ensure it has a close working relationship (or is simply a role within) the main implementation/operations team

* Ensure everyone is getting continuous training.  Including senior management.  Create time for reverse mentoring and innovation.

* Commit to FULLY FUNDING THE OPERATIONS of prototypes which pass the gate into full implementation

* Commit to annually review the relevance of running systems and to plan to kill the ones that need to retire entirely or be replaced with new approches

* If you are doing technology projects, ensure you aren't a technology island.  That means many different things including:

** the teams need full, unblocked Internet access, because that's where the information and the tools are

** the teams need to connect out to modern approaches and other developers - this means participating in local hackfests, sharing code on github, etc.

** the teams need to be able to share, publically, their challenges, the work they're doing - through blogs, presentations etc.

This is all a long, long way from our current factory model.  In the factory model, interchangeable employees toil on the factory floor, silently, while high above the managers gaze down benevolently from their glass-encased offices, and every project is initiated from senior management, for senior management.  It is a long way from a hierarchical 20th century factory organisation to a 21st century agile adhocracy that is both well-connected internally and well-connected out to the entire world of agile innovation.

Plus which, even if you have all of this top of mind, you can still fail.  Microsoft and Google hire the top computer scientists (and PhDs in many other areas) every year, AND have deliberate initiatives to attack every stage of this process, to try to ensure they don't fall into the big organisation trap... and yet both Microsoft and Google continue to fall into this trap in various ways.  The point being, even organisations that set out to build agility into their DNA like Google can fail, which is why you get endless "why Google can't build (small startup x)" or "why Yahoo Flickr didn't invent (new photo sharing approach y)".  In fact to prove my point here's both Why Google Can't Build Instagram AND Flickr Should Have Built Instagram. But They Didn’t. Here’s Why.

The fact of the matter is, whether your organisation is 300, 3000, or 300,000 it is never going to be the same as 8 people in a room with a coffeepot.  The best you can do is be mindful of that, and have a mix of internal improvements and standalone innovation, spinning in and out of your organisation continuously.

And that is my answer to @dbast's question

@sdkibb @jesgood @s_dav @mrpwalt @jeromefb how can I convince you to organize a full-day #goc "hackothon", besides that it would be awesome?

Our organisations are not incapable of coming together to create awesome prototypes.  The data.gc.ca prototype was done with a mix of dedicated resources and corner-of-desk, in 7 weeks.  It would be another year before it was made publically available.  GCPEDIA was done in months and has run for over two years, but it has yet to be operationally integrated or sustainably funded.  Hacking for hacking's sake will produce more orphaned prototypes that no one wants to sustain but no one is willing to shut down.

Hacking for hacking's sake also won't help if at least at a minimum the organisation itself has not recognized the problems of big projects and backed up that recognition with both public statements and effective internal initiatives.

I'm not saying hacking to demonstrate the art of the possible isn't great - it is.  Hackathons and apps contests have in many cases opened the eyes of CIOs to what can be done with openness combined with outreach and enthusiasm.  But unless you're willing to turn everything off the day after the demo, you still need to have an organisation-wide transformation in order to have a repeatable process to get from innovation through a gateway to full implementation and sustainable operations.

Basically the situation sucks.  This is not a factory, Taylorism doesn't apply to solve the entire problem.  We can't chop this up into smaller and smaller problems and make each piece more and more efficient.  If you have The Innovation Group and The Gateway Group and The Operations Group you just end up with efficient silos and an inefficient organisation.  This problem needs both bottom-up attacks to demonstrate local efficiencies and possibilities as well as systems thinking, a systematic organisation-wide attack on the problem as a whole.  Taylorism is easy and high-achievers get to be individual Linchpin superheros.  Systems change is hard.  But systems change is what we need.

authentication architecture in a federated enterprise

One of the challenges for authenticated services provided across your entire enterprise is identifying the appropriate credentials to use, for both ease-of-use for users as well as for sustainable manageability.

For organisations with a single central user directory, such as an enterprise-wide ActiveDirectory, this can be relatively straightforward.  In other organisations, there may be separate divisional authentication systems, without a single global user directory.

This means there are a few options for providing a global service within the enterprise, all with associated complexity and tradeoffs:

* One can create a new central directory, and either have users create accounts, or pre-populate it from existing stores.  However, if the existing divisional authentication systems continue to exist, this creates both a potential source of confusion (as users now have at least two accounts) as well as a possible synchronisation issue as both divisional and central user accounts now need to be maintained.

* One can federate across the various authentication systems, using e.g. Active Directory Federation Services or various LDAP federation approaches.

* One can use Shibboleth or other distributed authentication systems, where trust relationships allow the individual divisions to continue to maintain the credentials, but act as trusted authentication providers to other parts of the enterprise.

Within the Government of Canada, there are a number of potential approaches that could be taken to solve this problem.  For example, there is a PKI credential that can be created, attached to the globally-unique employee identifer (the PRI).  This is called myKEY... except it seems this branding is also used to describe a variety of legacy PKI systems within the government as well, so the situation is somewhat muddied.

Where this starts to get really complicated is when you want to have different levels of user rights, with e.g. closed groups with administrators, managers and contributors, and when you want to make this work in proprietary systems (which often have robust rights management, but using proprietary internal user account management) or across organisation boundaries, e.g. an extranet site that allows both internal users and external users.

As well, a federated solution means one has to deal with all of the varying privacy, security, and personal information standards of all the different divisions.  In such a situation, "can we use these credentials for X?" can become a difficult question to answer.

Beyond that, ideally we would like federated search to be able to return results for all content that a user has access to (including closed groups that they participate in).  I'm not sure how to manage this, effectively it means the central search service has to be able to act as an authenticated proxy-user for every single user.

For external collaboration, do we need to consider something like the (controversial) Identity Ecosystem proposed by the US government?  How do we allow Canadian citizens to participate in a trusted way in online government consultations, while ensuring that both their privacy is protected (so we probably don't want to share credentials with health or tax systems) and that they actually are Canadians?

Another issue is internal sustainability, for example, tightly binding to myKEY means that there needs to be a clear sustainability and evolution plan for those credentials.  If not, we run a danger of locking ourselves into a particular authentication solution.  To avoid this, one needs to find a way to loosely couple to an authentication service, but this can be difficult if you are seeking very granular user permissions and identification information.

For the moment, we are simply having users create separate accounts for new cross-division (enterprise global) systems but this is not sustainable.  Even attaching accounts to approved government email addresses is not sustainable, as currently the email address changes when users change departments.  I would be interested in hearing how other organisations are managing this (particularly for internal Government 2.0 services, such as government-internal wikis and discussion fora).  Is GovDex using user-created accounts attached to government email addresses?  Are there any best practices?

RSS and enterprise notification architecture

One aspect of Enterprise 2.0 is the ability to monitor, discover and share content from a variety of sources.

We don't have a lot of "glue" to enable this from the user perspective - basically you can search (Google or enterprise search) or follow (email, Twitter, Facebook, Yammer, RSS).

There is a line that breaks this approach, which is the line (actually usually multiple layers of firewalls and policies) between the intranet and the Internet.  In general, many of the rapidly evolving, inexpensive tools for web and mobile "live" on the Internet - they can't see intranet content.  Which means you can't get a search alert about your internal blog, nor can you follow your internal blog's RSS feed on Google Reader, or on Reeder on your external-network iPhone, or on an external-network Blackberry RSS feed reader.

The stiles that will get you over the intranet fence are pretty limited: email or authenticated RSS (if there are other options, I'd be interested to know).  So there are basically three options:

* have two completely separate working environments, one with Internet tools for public content, and one with intranet tools for internal content

* open access so that Internet tools can receive internal content, whether by allowing emails of content e.g. from internal blogs or (preferably) allowing authenticated RSS

* provide an internal feed reader service (website) which can see both internal and external feeds, allow it to be used through e.g. user/pass login from outside the intranet

Unfortunately, this usually breaks down in multiple ways:

1) Internal environments often have multiple levels of silos, rather than a single unified intranet. This means authentication challenges (simply being within an organisation may not provide the credentials needed to access a particular departmental feed) as well as policy challenges, as different sub-organisations may have different access policies.

2) The internal environment often lacks a notification strategy (e.g. a policy that all sites should provide RSS), as well as lacking tools and training for internal monitoring of RSS (desktop apps, Outlook, browser plugins etc.)  Let alone a set of OPML lists of organisations feeds, or activity aggregators like FriendFeed can do externally, or PlanetPlanet can do internally.  Even when the tool itself has support for RSS (e.g. Outlook) organisations often are running very old versions of the tool, before full RSS support was implemented.

3) Often organisations choose monolithic all-in-one integrated "collaboration suites" rather than many pieces loosely joined using e.g. RSS and JSON.  Visibility into these integrated sites from outside apps or web tools is often limited.

4) Authenticated RSS is very weakly supported (including not being supported at all in Google Reader)

5) Mobile devices are often on external networks and come with only basic tools and support (email reader and web browser) rather than pre-installed RSS readers.

I don't actually know how to solve this problem.  One approach is to gateway everything to email, but that just turns everything into a mailing list, and I hate mailing lists.  Plus which most organisations don't want their internal content flowing out unauthenticated over email.

The short term solution would seem to be to educate organisations about RSS, ensure their internal services provide RSS, and make sure their desktop, mobile and remote (e.g. home VPN) users have RSS readers AND access to internal RSS.   (There are other notification infrastructures e.g. XMPP, but these are not as widespread as RSS, although they could always be supported by gatewaying/converting from RSS.)  You can also consider, rather than providing desktop & mobile clients, providing an internal RSS reader server website, that users can log into from inside and outside the network.

The long term solution I think is for organisations to both think clearly about how much of their content absolutely must sit hidden from public view in the intranet, and to provide widespread RSS authentication support so that users are able to access their work feeds from familiar web and mobile environments.

On a related note, similar issues apply to enterprise search, where you usually need a separate internal search portal in order to discover all internal content, and you have the same issues when e.g. you need to discover enterprise content when you're on your external-network iPhone.

I have focused on integration between systems as done by users reading and searching, but there is of course a similar issue about machine-to-machine integration both internal-internal and external-internal.  We have a bit more mature infrastructure there though, e.g. providing JSON and Web Services.  But it still requires an organisation that understands that its internal services should provide APIs.

Does your organisation have this issue?  How to do you manage alerting on internal content?  Do you provide internal RSS tools?  How are you handling internal search?

Jon Udell blogged about this issue in 2007, and the comments on his Authenticated RSS feeds post provide useful additional information.

plan your strategy first - select your library technologies later

A fantastic post from Meredith Farkas about why Web 2.0 initiatives succeed or fail at libraries - It’s not all about the tech - why 2.0 tech fails.

Whatever we’re doing should be tied to the library’s strategic goals and planning.

If it’s not tied to the library’s goals, then how will it be seen as a priority? Similarly, 2.0 technologies should be planned for in a strategic way, which I think has not happened at a lot of libraries. Some libraries jumped on the blogging bandwagon because they thought (or were told) that every library must have a blog. Other libraries started wikis because staff were really excited about the idea of having a wiki. Neither are good reasons to implement a technology. We first need to understand the needs of our population (be it patrons or staff) and then implement whatever technology and/or service will best meet those needs. We need to have clear goals in mind from the outset so that we can later assess if it’s successful or not. These technologies may be fun, but they’re simply tools.

The paragraphs above are as good a description of the need for a strategic planning function (which, being an enterprise architect, I naturally think of as an Enterprise Architecture function) at a library.  This is what I said on FriendFeed

It is absolutely not about the technology. In enterprise architecture terms, it's about determining the target (most commonly, a strategic plan) and then ensuring all activities are aligned with that plan. It can be hard for an org to understand that both working really hard, but on the wrong things AND indulging 'side' projects because the tech is trendy will get them nowhere. If it doesn't move you forward to your goals in a sustainable way, don't do it.

That's not to say you shouldn't allow "20% time" for experimentation, but draw a clear line between a blog experiment, and an official public blog presence. One can be tried and disappear, the other had better be sustainable.

Dorothea has quite rightly called me out for talking in my blog more about the technology than the people,  (The great thing about blogging is I get to test my ideas and their expression against an audience that thinks about these issues from many different angles.)  So I want to be very clear: the specific technology implementation does not matter, at a high level.  You must first decide, will creating this business function (e.g. "store organisational output") move me towards my strategic goals.  You must do this with a clear-eyed view of the ROI (that is, don't ascribe silver-bullet capabilities to a commodity technology - my joke at work is, to the extent that an IR is just a hard drive spinning in a corner, I could run an IR from home - I have 750GB of RAID 1 mirrored, networked storage).  The point being of course that an IR is not just a hard drive, it's a whole set of people and processes that make it valuable, sustainable, and worthwhile.

In summary - based on where you want to go (your strategic goals):
1. Chose the right set of aligned activities.
2. Ensure you have the right people and process to support those activities.
3. Only then concern yourself with the particular technology implementation details.

This is an echo of what I said in "librarians don't need to know SEO" - start at the high level first, don't just grab onto the latest buzzword.  I wish I could fit more about enterprise architecture into this blog, but it is a huge topic (I'd make a joke about "I have a wonderful solution, but sadly I've run out of space on the margin").  I have spent years trying to explain various aspects of it here, see the Enterprise Architecture category for my various attempts (including this one).

don't predict the future, create it

Give me a lever and the place to stand, and I shall move the earth.
- Archimedes

See Wikiquote - Archimedes for the Greek and the actual phrase.

In trying to plan for the future, to perform the strategic planning that is a core activity of Enterprise Architecture, you must stand in the right place.

Specifically, you must stand in the future, in the envisioned target state, and look back, to determine the plan.

If you stand in the mire of the current state, and you try to look forward, your boots will never get unstuck.

Hindsight, as they say, is 20/20.

If you stand in the present, and look forward, it's like looking into mirrored one-way glass.  Looking forward from the present, you see only the current state reflected back at you.  Start from your current set of running projects, and all the many many details of your organisation's resources, and tasks and operations, and you will never stride out of that minutiae to reach the future.

But when you stand instead in the future, and look back, you can see right through the one-way glass, you can see all the steps you needed to take to get there.  You have to, therefore, plan by first creating the future, envisioning it in target statements, mockups, use cases, whatever is a meaningful description to you of what your organisation does, in 2012, or 2015, or whenever.

Then look back... in 2015, we're providing semantic search for chemistry, so how did we get there, well we must have created an index of chemistry terms out of the literature, and we must have gotten some semantic search tools to extract and unify this terminology, and a nice user interface so chemists could enter queries... so that means we must have been holding full-text articles and had the rights to index them... and we must have developed a basic search capability... and on back in time until you reach 2008.

This can be done.

The alternative is to stand around in 2008, and say "we're doing a bunch of stuff, and then... um technology changes somehow and... then the entire competitive environment shifts and... then we do a bunch more stuff... and then err, a miracle occurs and then we have semantic search in 2015".  You can't get there from here.  You can't get to the future by speculating forward from the present.

Don't predict the future, create it.^*

* Alan Kay said "The best way to predict the future is to invent it."

implementing SOA at CISTI

My colleague Stephen Anthony has posted his author version of his peer-reviewed article for The Serials Librarian, "Implementing Service Oriented Architecture at the Canada Institute for Scientific and Technical Information".  At 14 pages, it's the most comprehensive and detailed look available at the challenges our organisation was facing, the Enterprise Architecture approach we took to help improve our technology planning, and the Service-Oriented Architecture that we are building as a result.

It's from 55 (1/2) which should show up eventually at

http://www.haworthpress.com/store/product.asp?sku=J123&detail=TOCList#TOCList

and, assuming I'm constructing the URL correctly, specifically at

http://www.haworthpress.com/store/Toc_views.asp?TOCName=J123v55n01%5FTOC&desc=Volume%3A%2055%20Issue%3A%201%2F2

Stephen reports the full citation as

Anthony, Stephen K. "Implementing Service Oriented Architecture at the Canada Institute for Scientific and Technical Information." The Serials Librarian, 55 (01-02), pp. 235 - 253.
DOI: 10.1080/03615260801970907 (doesn't yet resolve)

EA/SOA are about business process, not technology

I would claim that, by the time I left, the technical team within which I worked had a better overview of the business processes of the university than any other part of the organisation. We were able to successfully re-apply that knowledge to subsequent development, to great effect.

Paul Walk - SOA and reusable knowledge - February 26th, 2008

latest EA info from IT Architects in Academia

Without having ever attended, it nevertheless appears that this group is probably the closest in alignment to the sorts of activities and challenges faced by my architecture group at CISTI.

ITANA’s Constituent Group meeting [was October 25, 2007 as part of EDUCAUSE].  We had approximately 40 people attend. Many of the attendees were from newly formed architecture groups.

Jim Phelps started out the meeting with a short presentation on ITANA, how it is supported and how you can participate. He reviewed some of the findings from the CIO Survey on EA (Enterprise Architecture).

There was active discussion around the role of Architecture in Business Process Improvement. Most CIOs stated that Business Process Improvement was where EA had the least impact on the enterprise. The reasons that were put forth included:

• EA is often seen as a “technology group” with no real input or understanding of the business side
• EA is often on the technology side and the changes actually need to come from the business side and be made by the business side.
• The business owners often want to re-implement the processes that they are currently doing not adopt new processes. We often make the technology match the current business process rather than change the business process.

There was discussion about the role of EA in business process change. This mostly focused on the fact that EA cannot implement the change but they can highlight the issue, document the current process and the improvements and work to lead the discussion around the change.

We also discussed the way that EA engages in the enterprise. Some institutions use EA as a gate function. Projects don’t get the go-ahead without EA’s approval or they are allowed to continue with contingencies. Other institutions use EA to guide projects and illuminate issues. Where does EA plug-in to the organization and the project / funding processes?

itana.org - EDUCAUSE CG Meeting Notes - October 26, 2007

There is also an ITANA wiki.

Enterprise Services Architectures - Chris Mackie - DLF Fall Forum 2007

I was privileged to have been paired with Chris Mackie of the Andrew W. Mellon Foundation on the DLF programme, and fortunately I think our talks were quite complementary.  Chris talked about Enterprise Services Architectures (ESA), presenting a very clear-eyed view of the sustainability challenges of IT infrastructure for higher education, and how ESA and some specific Middleware ESA (MESA) projects are addressing those challenges.  His presentation (PDF) is now available from the DLF site.

future directions of library systems

It was great to talk with various enterprise/IT/technology architects at DLF Fall Forum 2007 about our various ideas an approaches to modeling and implementing the new library technology infrastructure.  But it's clear there are still more library technology architects out there (even if they may not have that in their job title), I ran across an interesting blog posting by Jonathan Rochkind:

Notes on future directions of Library Systems from Bibliographic Wilderness blog, September 28, 2007

Here are some notes on near/medium future directions of the library systems environment/architecture, and a sketch of requirements on where we want to go. These notes may or may not end up as part of an internal white paper here, as we analyze where we want to be headed (something good to do anyway, but we got a kick in the pants when the vendor ended development on our current ILS).

...

The library systems environment has grown from being composed of a single “Integrated Library System” at the origin of library systems in the 1980s, to being composed of an ‘ecology’ of systems in most libraries today. These different systems could be from various sources (proprietary and open source), fulfill various–both overlapping and distinct–functions, are used in various ways by different subsets of users (both library staff and our end-users), and interact with each other to varying degrees (some well, some poorly). 

I wonder if I should consider adding a "library technology architecture" category to SLP?

PS Jonathan was at DLF but we didn't formally meet, it's odd how sometimes virtual connection and discovery is easier than in the physical world.

my presentation on SOA for libraries at DLF Fall Forum 2007

Overall I think it went very well, the room was packed (to my considerable surprise) and I got good questions from the audience - for once actually too many questions for my allotted time, so I unfortunately stole 10 minutes from Chris Mackie, whose presentation followed mine.

The slides are up on Slideshare, they were done in Apple Keynote, in case you're wondering.

http://www.slideshare.net/scilib/serviceoriented-architecture-for-libraries/

All of the links and background info for the presentation are available at

http://www.connotea.org/user/scilib/tag/dlf2007akerman

The following is not the actual narrative, but it gives you enough to follow the (mostly conceptual) slides:

I started with a great quote via Katherine Kott of DLF Aquifer, from the previous night's session on collaboration, stating that we need "vision translated into operational success".

Service-Oriented Architecture addresses problems in technology planning,
design and implementation.
First, we must understand what some of these problems are.
Fundamentally, technology implements the desires of the organisation.

Too often we focus on our immediate desires ("What do you want") rather
than long-term planning.
This has meant we get what we asked for (e.g. incremental catalogue
interface changes) rather than what is best for the organisation
(transformed catalogue)

In order to better align our implementations with our goals, CISTI uses
an SOA derived from our Enterprise Architecture.  In our EA methodology,
we start with business needs (Who are you?), and only then proceed to
identifying the gaps to be filled.  But you must look beyond the current
state (Why are you here?  Where are you going?)

So what are the transformational challenges we face?

In a way, we are in a fantastic world for the traditional goal of a
library - ubiquitous, zero-cost, perfect copies of information.  No more
scribes devoting their lives to copying a few books.  But this has
tremendous implications in terms of preservation, versioning, citation
etc.

Also, there are major changes in moving from a physical world where we
engineer bridges, to a virtual world where we "engineer" software.
Software doesn't have most of the constraints that apply in the physical
world, and we have to be careful with analogies to physical objects.  It
would make very little sense (except perhaps to a US Senator) to build
half a bridge, or a bridge that goes nowhere.  But in the software
world, building components, even if they are only part of an overall
goal, can be extremely useful.  And software is not location constrained
- it can be accessed from any location - there should be no "software to
nowhere".  The network is everywhere, and we need to understand how that
transforms our systems design.

So how do we apply the notions of architecture that we understand from
the physical world, and that have for centuries and continue today to
enable us to build very complex structures with many components (heat,
lighting, electrictiy, plumbing, etc.)

Well, we can use the idea of modeling.  Just as we make models of
physical objects, we can make models of business processes.  But we must
ensure that we don't get stuck in our models or frameworks, and do move
to implementation at some point.

CISTI has found that its methodology, which is focused on getting to the
implementation framework, and beyond that to design and development, has
been invaluable for turning business ideas into implemented software.
We are fortunate in that we are a library with substantial in-house
technology resources, including a dedicated architecture team (the only
library EA team in the world??)

SIDEBAR: I met the Head of eArchitecture for the British Library at my presentation, so now I think I should say "one of the very few library EA teams in the world".  END SIDEBAR

This structure may be different for
academic organisations, with faculty groups that have departmental IT,
central IT, and academic consortia.

Within this larger mehtodological framework, SOA is an approach to
identifying "servicifiable" functions in the organisation, and building
larger service offerings from various combinations of services.

In the architecture analysis phase, CISTI models all of our
technology-related activities, at a high-level this model describes all
the technology-supported activities, with functions such as "provide
library service" that then decompose down into many many lower-level
models.

Recently we have been specifically modeling a Trusted Digital
Repository, using the OAIS model and converting and adapting it.

This EA modeling work then leads to the derivation of SOA services
[depending on audience profile I may talk a lot on this slide about the
details of SOA, or not].

CISTI has used and is using SOA successfully as the foundation of
various projects - this is not just theory.

SOA is also not an agility-killer bureaucracy - services actually
provide pieces that can be used for small experiments, while
simultaneously moving the organisation in its stated general direction.

Ideally, we should all be collaborating on SOA, rather than building SOA
silos.  I have blogged a lot about this, but with little response.  Are
the individual framework groups only talking amongst themselves?  How
can we use their efforts most effectively?

SOA is certainly being explored in many areas of the library space,
including various digital library initiatives and modeling efforts (some
of which will be presented in other forum sessions).

SOA also offers the possibility of improving the catalogue by breaking
it into component services, and enabling the sort of simple layers
(widgets, mashups, javascripty stuff) by providing sustainable,
well-architected underlying service capabilities.

For academic libraries in particular, SOA offers a compelling
opportunity to connect with existing and future scientific activities,
as many Cyberinfrastructure projects have SOA at their cores.  Can we
use modelling at even the highest level (the entire scientific
communication cycle) to guide our service implementations?

Ultimately to build sustainable, non-siloed, modern information systems
to support scholars and citizens worldwide, we need to build many
bridges (this one happens to be built of lego components, in case you're
worried that giant ducks have invaded the US).  Between libraries and
scientists, between technology and business, between funders and
software developers, between computer science and library science...

We need to move beyond frameworks, using solid governance to ensure that
SOA plans become running SOA code, while avoiding getting stuck in giant
projects that don't deliver.  As well, even if you don't have the scale
or the capacity to build your own SOA, you can still participate as a
service consumer.

I'm not crazy in this (at least I hope not), as evidence for which I
present some starting points about SOA in the academic and library
sectors.

Questions...

The End

SIDEBAR 2: In case you're wondering, there's a slight science fiction theme running through the presentation, inspired by the "federation" part of the organisation name.  The Four Questions are from Babylon 5, the goodish aliens, the Vorlons, ask "Who Are You?" and represent the forces of order, the evilish aliens, the Shadows, ask "What Do You Want?" and represent the forces of chaos.  "There Are Many Copies" is from the opening sequence in the new Battlestar Galactica, using the great Creative Commons images I found worked out as a nice way of avoiding using a screencapture (plus using a screencap might have been a bit distracting as it shows a scene from BG where the many copies of Sharon Valerii/Number Eight are not so much wearing any clothes).  I had a better closing quote but I couldn't remember it, so I used a tag line from Buckaroo Banzai Across the 8th Dimension, to close out the "where are you going" questions theme.  END SIDEBAR 2

SIDEBAR 3: I was actually amazed to be able to locate the "bridge to nowhere" Google Earth image.  I had seen the unfinished highway bridge out of the airplane window as the plane was coming in to land in Philadelphia and by the time I thought to GPS-mark it or take a photo it was already past.  Incredibly, doing an image search on

"bridge to nowhere" Pennsylvania

brought it up as the first hit - Google Search, Google Earth and the Internet continue to amaze me. END SIDEBAR 3

SIDEBAR 4: I bought my carbon offsets from MyClimate.org

flight from: Ottawa, ON [Macdonald-Cartier International Airport], Canada, YOW
flight to: Philadelphia, PA [Philadelphia International Airport], USA, PHL
flight via: Toronto, ON [Lester B. Pearson International Airport], Canada, YYZ
return, economy
flight distance: 1'845 km
flight passengers: 1

CO2 Emissions: 0,511 t

Total costs for compensation of your flight: SFr. 20.00

According to Google

20 Swiss francs = 16.3543582 Canadian dollars

CISTI jobs: Technology Architect

Technology Architect

Closing Date
06/26/2007 (CLOSED)
Applications will be accepted until 23:59 Eastern Standard Time

Ottawa - Ontario

CS-3
This is a 3 year term term position from the date of reporting.

Education
A Bachelor degree in computer science, computer or software engineering, or other related field.
Formal training in Enterprise Architecture is an asset.

Language Requirements
English

presenting library soa at IATUL 2007

I am pleased to announce that I will be presenting my paper "Library Service-Oriented Architecture to Enhance Access to Science" at IATUL 2007, my session is tentatively scheduled for

Tuesday June 12th 2007 at 10 am
The presentation will be about 20 minutes plus 5 minutes for questions.

UPDATE 2007-06-12: The presentation has now been completed and is available online.  ENDUPDATE

My paper is a brief summary of CISTI's EA and SOA success, based on the efforts of the rest of the architecture and application development teams.  I intend it both as a demonstration of what is possible, as well as a discussion starter, to help other libraries move beyond basic theoretical frameworks to start building a richer network of library machine-to-machine data services that we can all use for applications, websites, mashups, and whatever is coming next.

StatsCan conf on EA and SOA

John pointed out to me the Stats Canada Information Technology (IT) Conference 2007, April 4-5, 2007, Ottawa.

The conference is entirely about Enterprise Architecture and Service-Oriented Architecture,
including speakers from inside and outside of the Canadian government.

http://www.statcan.ca/english/conferences/it2007/

http://www.statcan.ca/english/conferences/it2007/all_abstracts_E.pdf

Among the new approaches, enterprise architecture has the objective of building a broad consensus within the IT community and among users. Having emerged in recent years as a central concept in the engineering of information systems, this architecture approach appears to result in a better organization of applications into discrete services.

Service Oriented Architecture (SOA) promotes an approach where automation logic is partitioned into individual units (services) that are highly interoperable, but generic enough to fulfill both immediate and future automation requirements. The major benefits associated with a successful adoption of SOA typically begin with improving an organization’s ability to share data across disparate systems and establishing an architectural model that is increasingly independent of the enabling technologies. Services are deliberately positioned as highly reusable IT tools that can be exploited over a long period of time. The end result is an increase in an organization’s overall business responsiveness (agility).

pondering SOA the UK and Aussie way

Service-Oriented Architecture (SOA) is not for the faint of heart, nor for those weak in technology expertise. 
SOA is fundamentally a methodology for systems design, informed by an overall Enterprise Architecture (EA).

If you don't have the scale of organization, or the organizational technical expertise, to sustain an EA and advanced development group, then SOA may not be for you.  You may want to focus more on consuming services and connecting services together.

Or in other words, we just have the latest in a line of IT best practices:

• If you're big enough to have an extensive software development shop, use SOA as your guiding methodology.  If you have a smaller shop, you may use or develop services defined by some other group, in their SOA.
• If you're big enough to have substantial IT infrastructure, use EA to guide the planning for that infrastructure.

Keeping that in mind, the library environment as a whole is clearly of the scale to benefit from an SOA, it's just a matter of how we put it all together.  Who defines the services?  Who will the service providers be?  What will the business model be for service consumers?

All this by way of introducing some recent reports and presentations on SOA, as well as signalling that I will be providing more of my library SOA thoughts in a few upcoming venues.

Firstly the National Library of Australia IT Architecture Project Report (PDF) has gotten some attention for its prominent mention of SOA.  (Thanks to Lorcan for pointing it out to me, DLTJ also mentioned it today.)

Second, via Brian Kelly I see that there were recent JISC conference presentations on EA and SOA.

e-Framework, SOA and EA: Using the e-Framework, Service Orientated and Enterprise architectures to develop more flexible and better aligned ICT infrastructures  Abstract

• Session commentary
• Presentation "The e-Framework, SOA & Enterprise Architecture" - JISC (PowerPoint)
• Presentation "Enterprise Architecture" - Open Group (PowerPoint)

Overall I think both SOA and EA are challenging topics to present simply.  That being said, I think both are typically presented far over to the extreme of complexity.  What was it that the rather inaccurately titled SOA for Dummies said you had to have in your SOA?  ESB, registry, broker, supervisor, BPM, yadda yadda... It's simply not true.  That's what you'll get if you let the kindly vendor hold you by the hand and take you into the twisty SOA maze.  All you actually need for SOA is to make sure you're using some basic software design principles:

autonomous, loosely-coupled and coarse-grained services with well-defined interfaces that support

the flexible reuse of application logic through composition

(The above an extraction from my 2005 presentation "Service-Oriented Architecture Methods to Develop Networked Library Services" which I am uploading to Slideshare... now.)

The same for EA.  Zachman is all well and good, but you need to be a mega-corp to fill in all the Zachman boxes, and even then the methodology doesn't help you much in translating that knowledge into action.  CISTI uses a much simpler and more practical methodology (based on InfoMajic) that focuses on translating business needs into an implementation framework.

There is lots of work and complexity in doing feasible EA and SOA without making it even more complex.

This may be why most of the published outputs available for library SOA are currently at such a very very high framework level.  I have to say, these frameworks don't do much for me.  As a technology person, all I see is some Dewey Decimal system for services, a top-level set of categories.  To actually start doing any service implementation work, you need to go much deeper, with models of how the detailed business functions inter-connect and inter-relate.

I wish I knew how to take the overall library community to that next level.  e-Framework (JISC - DEST - NZ MoE - SURF) is a piece, DLF Services Framework is a piece, Knowledge Exchange (JISC - DEFF - DFG - SURF) http://www.knowledge-exchange.info/ is as well... Maybe it's simply impossible given the scales of organizations involved to go beyond the highest-level conceptions.

All I know is we at CISTI can think of some relatively easy Web Service wins within an overall SOA - copyright / pricing services, query services - there are lots of possibilities.  Until we start getting more service providers and consumers up, we're all sort of stuck.

What do you think?  Is SOA too abstract or unachievable for your organization?  Are the service frameworks too high-level to impact on your development work?  Would you actually consume some library Web Services if more were available (keeping in mind the service interface might be simpler than full SOAP).  Are there services you're already providing that aren't being consumed?  Are the digital library people producing services that aren't being used by the rest of the library?  The Fedora repository already has Web Services, Amazon and Google already do - where do you see libraries fitting in?

Lots of questions... any answers?

I feel that I'm kind of reaching the limits of what I can do in terms of fairly abstract SOA evangelism - and I'm hoping there may be some concrete initiatives around which some services can coalesce...

standard IT architects

IT architect is one of the hottest jobs in tech right now, thanks in part to more companies trying to build service-oriented architectures. The problem: Not everyone agrees about what IT architects are supposed to do or know.

...

IT standards consortium Open Group is launching the industry's first professional association for IT architects. Called the Association of Open Group Enterprise Architects [AOGEA], it's looking to "elevate" the profession by promoting certification and establishing ethics standards and architecture principles.

...
A shortage of IT pros who can guide SOA development is one of the biggest hurdles for SOA adoption. Analyst firm ZapThink and staffing company Excel, seeing a market in that shortage, have launched the Architect Resource Center, a consulting service

from InformationWeek - IT Architects In High Demand - January 29, 2007

I'm all for architecture principles and standards.  I think certification is bullsh-t though (not just for IT architecture, for any kind of IT).  And if you can't figure out ethics on your own, no Association is going to help you.  AOGEA seems to have a needlessly complex structure based around certification levels.  Count me out for the time being.  Err, although the conference in Paris looks nice...

For the record, my title is Technology Architect - IT, but since that is rarely meaningful to people, I usually tell them I'm a technology planner (or if I'm feeling grand, an enterprise strategic technology planner).

reintermediation?

Dave Pollard has some interesting ideas in The Challenge of Reintermediation.

My comment was that the reintermediated IT professional role he describes sounds a bit like enterprise architect to me.

I do think that "information gathering" is going to be a big challenge to convince people to reintermediate.  Senior managers, who could benefit most from the time-saving, are too aloof to either surf the web or to request this kind of information service, so they operate in a perpetual information vacuum.

The rest of us are quite content to take a bit of time to surf the web on our own.

Remember the challenge of "switching cost".  People will only switch to a new service if you can demonstrate that it is DRAMATICALLY better than the previous one.  That's why we have settled on CDs/MP3s DVDs/Divx Windows 2000/XP Google.  It's good enough.  There are billions of dollars trying to move us to AAC and super audio CD and HD-DVD and Windows Vista, and you can see how successful that has been.  Now you're going to try, without billions of dollars, to move me from Google to a human intermediated search?  I don't think so.

my review of SOA for Dummies

Summary

I am reviewing this book primarily in the context of Service-Oriented Architecture (SOA) for Libraries, which is a rather dramatically different audience than it is intended for. 

I recommend the first three chapters of this book, along with the "loosely coupled" section of chapter 5 as a good introduction to this topic.  As well, Chapter 11, on SOA Governance, is excellent.  However, its main audience is large organizations (who are the main current adopters of SOA), so much of the content may not apply to libraries, particular those with limited technical resources.

Although the Dummies books proclaim "A Reference for the Rest of Us", this book would really more accurately be titled "SOA for Dummies... Who Work at Giant Fortune 500 Companies".  Much of the technology and processes they present are beyond the reach of all but the largest organizations.

To my surprise and dismay, there isn't a single mention of Enterprise Architecture (EA) in this book, whereas I feel that for the scale of organization this book targets, EA is absolutely essential.  Without EA, the scale of activities and technology described in this book is impossible.

There is also a very important discussion that needs to be had about agility versus structure.  I am not convinced you can maintain organizational agility as a goal, with the complexity of the management and technology structures this book describes.  The book does not address this issue at all.

Overall the flow of the book is a bit uneven, both in terms of writing style and in content, and it doesn't present a coherent "SOA story".  To some extent this is due to the nature of the Dummies books, where they want each chapter to stand alone.  I found it made for a bit of a "here's an important concept" ... "and here's another one, with some vague undefined relationship to the previous one".

This book will mostly give you a lot of "why" and "what" about SOA, not a lot of "how".  That is the nature of the beast, however - I don't think there's any other way to present it, with out it being a 3000 page book.

The companion website www.dummies.com/go/soafordummies is highly promoted on the front and back covers of the book but it does not seem to exist, which seems like a bit of rather bad planning.

Sidebar

UPDATE 2007-08-09:

If you're coming here from SOA Center, you may be expecting Beth Gold-Bernstein's review.
The SOA Center link is currently wrong.  She blogs at ebizq:

July 24, 2007  Beth Gold-Bernstein  SOA for Dummies
July 31, 2007  Beth Gold-Bernstein  Reactions to SOA for Dummies Book Review (in which she refers to my review as having some similar points to hers)

I did leave a comment on SOA Center correcting the link, but it has not been posted.

ENDUPDATE

Details

I don't come to this book with a blank slate, quite the opposite, I've been trying to find ways of clearly explaining SOA to both business people and technical people for the past few years.

Just to stake my prior claim, I think I did a pretty good job in my December 9, 2005 presentation "Service-Oriented Architecture Methods to develop Networked Library Services" (video, PowerPoint) which predates this book by about a year.

It is a difficult story to tell to be sure.  To my mind, SOA sits at an intersection of Enterprise Architecture, software engineering, Web Services, business and technology.  So in a way, the challenge is "let's take a bunch of things that people don't understand, and try to explain a layer on top of them, that depends on all of them".

Also, in the interests of transparency, the informal style used in this book is not one that I enjoy for learning.  My ideal informational book is just facts piled upon facts - I don't need a story, or to be put at ease. 

I tried in my analysis to ignore that stylistic aspect, as well as trying to come with as blank a slate as possible, keeping in mind I've been deep into SOA for the past couple years.

Here are my thoughts, chapter by chapter.

Chapter 1 - SOA What? (this makes more sense if you pronounce it so-a, as the authors do, rather than S-O-A)

All of Chapter 1 (PDF) is available free online from Wiley.

They talk about their audience, but I do think they make a mistake in not scoping the book.  The full SOA they describe in the book's 300+ pages is one that could only be used by a large organization.  Think, General Motors revising their information technology, not Bob's Corner Store.  They should have made it clear that you can adopt some SOA concepts, without going to the full, complex, vendor-rich (and vendor-enriching) extent that they document.

For example, they start out with some clear explanations about how SOA is an enabler for business to work better with the IT department.  You may at that point be thinking, "my problem is not working with the IT department, my problem is I don't have an IT department".

So in reading the book, be mindful of their audience.  And also, I suggest you think of how SOA could help the entire community of libraries, collectively, if the scale of your individual organization is not appropriate for SOA.

They do extract good key points, like the fact that reuse is at the core of the SOA goals and methodology.  At this level, they face a challenge that I don't think there is a good solution for, which is that you have to talk a lot at the beginning about SOA "why" and "what", without being able to easily walk through a lot of SOA "how".

I also think that their definition of SOA is a bit over-broad.  Some of what they describe as SOA is what I would consider Enterprise Architecture (EA).  Now, I certainly think SOA works best within an EA context, but there doesn't seem to be any mention of EA at all.  So I disagree when they say

1. The IT organization will partner with the line of business managers to create a high-level map of what the business will look like.

That surely needs to be done, but that's EA, not SOA.  To me, Service-Oriented Architecture is just the architecture of the services, not all the processes of business process analysis as well.

Chapter 2 - Noah's Architecture (oh I get it -- literally, just now - it's a play on words, see.  sigh.  I spent the entire chapter wondering, what does Noah have to do with it?)

So at this point, the conceptual assault really gets underway.  To get to the heights of SOA, you have to build concept upon concept.  Unfortunately, each one of those concepts could be a book in itself, and here they get a couple pages at best.  I'm not being critical - I don't see any other way you could do this, and it's the same approach I have used, but I can imagine it being overwhelming reading for someone coming from outside a software engineering background.

They give an ok explanation of software architecture, and hit the high points of the history of software engineering that got us here.  I liked their mentions of business logic and the key concept of separation of concerns.  Again, they are talking about big organizations, for example, on page 16, they state

In the Good Old Days, a business unit would ask the IT department to create an application to solve a specific problem.  To accomplish this goal, the IT department would write a set of customized programs.

This is absolutely, 100% true.  For a certain scale of organization.  If you're thinking instead, as I mentioned before, "I wish I had business units.  Or an IT department." then SOA may not be a good match for you.

They then take a bit of a side trip into components.  I'm not sure I would have used this terminology.  It's certainly true that we tend to run out of useful worlds to describe certain concepts, but to me components are about a specific subfield of software engineering, Component Based Design.  Software components have a very specific meaning, a meaning which is different from the one given in this book.

This is important, because if you read this book and say "component" to an IT person, they may not have the same understanding that you got from the book.  In a similar channel, the use of "plumbing layer", which is continued throughout the book, is most unfortunate.  "Infrastructure" would have been a much better choice.  Whatever dubious merit there is in the informal term of "plumbing" is going to be lost the first time a non-tech person uses the word to a technical person, and the technical person thinks "Plumbing? What the heck?  Is this guy an idiot?"

At this level, there does have to be a bit of "hand-waving" over the explanations, this is particularly egregious (although the authors themselves acknowledge this somewhat) in the "simple adapter" explanation.  Adapters are probably some of the most complex challenges an organization faces in transitioning to SOA.

I did enjoy the brief excursion into "application archaeology", particularly since my original SOA presentation on SOA had a slide or two on this concept, with a nice quote from Vernor Vinge's writings, but I had to take the whole segment out because people said "application archaeology, what the heck were you talking about there?" (I'll try to dig up the quote, it's excellent.)

I'm also not a super big fan of the "black box" terminology used, but it does capture the concept well enough I suppose.

Chapter 3 - Not So Simple SOA

A bit of a jump from components to Web Services, with a bit more handwaving.  I also didn't like the strong link between Web Services and SOA.  While their statement that SOA is not Web Services enhanced (or I guess I should say "is not Web Services 2.0") is certainly true, it's also true that you can do SOA without Web Services at all.  The two are, as we say, loosely coupled.

There's also a bit of a jumble about business processes, I'm not sure it fits well within this chapter.

Finally, while silos are mentioned, I think they really missed an opportunity to emphasize a key failure of siloed software - that it locks up useful services deep inside the silo, where no one can get at them.  This aspect is mentioned only in passing.

All those caveats aside, I did feel that Chapters 1-3 are about as good a clear explanation of this extremely layered concept that you can get (well, that is to say, almost as good as my explanation :)

You should really jump from Chapter 3 right to Chapter 5, I'm not sure why they placed Chapter 4 here.

Chapter 4 - SOA Sophistication

Woo boy.  Chapter 4.  If you are coming from a small organization, or a non-technology background, your service experience of chapter 4 will be akin to hitting a brick wall.  Chapter 4 layers on the SOA "essentials" at a breakneck pace, presenting as essential elements of SOA things that are, each of them alone, HUGE complex business activities.  We are definitely into the General Motors scale here.  The Galactic Motors scale, in fact.

Let me make it immediately clear: I do not personally believe you need all of the stuff they describe in order to use SOA, and to participate usefully in an SOA.  The basic SOA methodology can be used by almost any organization that has at least some idea of its basic business functions (and some business decisionmakers) and a small IT group to design and build services.

So, what are the bricks in the giant Chapter 4 wall?

• Enterprise Service Bus (ESB)
• SOA Registry
• Workflow Engine
• Service Broker
• SOA Supervisor (about which, I have more to say)
• Business Process Management (BPM)
• Service Level Agreements (SLAs)
• Automated Service Monitoring (related to SOA Supervisor)

In looking at that list, my first reaction is "hmm, are there any 30 million dollar cheques I haven't cashed yet?" and secondly, that almost anyone, even someone from a large organization, who reads this chapter is at great danger of rejecting SOA out of hand.  You're telling me that SOA is going to improve my organization by adding re-use and handling complexity, and you want me to add a gigantic laundry list of complex technologies (many of which don't yet fully exist) to my organization?  No thanks.  Call me in 10 years when you've got it all figured out.

Now I want to be clear: all of these are important aspects to consider, and many of them will need to be adopted, by very large organizations with huge numbers of services.  But I imagine that most libraries will have a handful of services - the basic functions that the ILS performed, plus a few others.  That doesn't require anywhere near the scale described in Chapter 4. 

I have certainly advocated that the most important piece I think will need to be layered on the base SOA is workflow, the orchestration server (you can listen to my presentation - yes, I will put the slides online eventually too).

I also want to address the SOA Supervisor, and the whole idea of automatically monitoring Service-Level Agreements.  I think this is very deep voodoo.  I am quite happy to be corrected, but I think this capability does not exist in any meaningful, easy-to-use, standardized way today.  To me, for SOA Supervisor, you might as well read SOA Magician, or SOA Magic Box.  It simply is not possible to perform most of the supervisory functions they describe, and even attempting to do so would be hugely complex and vastly expensive.

Chapter 5 - Loose Coupling and Federation

The loose coupling part of this chapter is good, but it really should have been separated out and put after chapter 3.  The Federation part only applies, as is often the case in this book, to very large organizations, that are so complex and diverse that they need to have multiple, federated, SOA domains, with a different set of rules applying in each one - think IBM HR SOA domain and IBM Supercomputing SOA domain.

(Note to CISTI architects: not the same as our domain concept.)

There is a bit of an overlong excursion into software-as-a-service (SAAS).  I think it could have been covered quite well by just saying "you may want to use external services, as well as your own internal ones".

I disagree quite strongly with the concluding section, which asserts that SOA is part of the industrialization of software development.  I certainly agree, and I have in fact asserted that SOA is simply our latest understanding of how best to do software development, but industrialization is way, way too strong a term.  Software is not constrained by the physical world, that's why you can't build software the way you build a bridge or a car.  I'm certain that Toyota knows, second-by-second and gram-by-gram, what is happening as a vehicle moves along its assembly line.  That is simply impossible for software, and it will probably never be possible.

Chapter 6 -  XML

To be quite honest, I don't really understand the emphasis placed on XML within SOA.  Thomas Erl certainly uses it as the keystone of the explanations he gives in his talks and books.  To me, you could get by with two sentences: "Software design works best when you can use standards.  XML provides the ability to create data standards."

This chapter goes into way way more than I think you need to know (for SOA purposes) about XML, SOAP, WSDL, UDDI and even (oddly thrown in) HTTP.  It also seems to express a view of XML as a sort of "magic glue" that ties different programming languages and systems together.  Um, we were able to connect systems together in the bad old pre-XML days.  We called it "using APIs".

Chapter 7 - Dealing with Adapters

This chapter starts out with a strong statement, unfortunately, one I disagree with

Adapters make SOA possible.  No adapters, no SOA -- it's as simple as that.

Um.  No.  I recognize that not all is green fields, but not all is layers and layers of legacy either.  This is coming from big corporation, banking thinking.  It is certainly true that when I enter a number on my bank's customer self-service website, it is going through layer upon layer of adapters, ending up at a mainframe Which Must Never Be Touched.

But for example, library catalogues?  You'd have to be mad to build adapters to catalogues.  Blow them up, replace them with a green field of services.

This chapter also creates a dependency between adapters and service registry and service brokers.  This is unnecessary.  It also continues the early "simple adapter" nonsense, on page 93

Technically, building an adapter isn't too difficult.

Um.  Yes it is.  What they're talking about is API Adapters.  My mainframe interface takes some ancient binary IBM format, and I wrap it with an XML adapter interface.  That isn't too difficult, yes.

But most of the problems facing libraries, at least, is that there is no interface.  There's nothing to easily adapt.  You have to write an entire interface layer.  That is hugely complex.  I have to imagine this also applies widely outside the library technology context.

I have to admit my final written note on this chapter reads, simply: this chapter is baloney.

Chapter 8 - The Registry and the Broker

We're now getting into more and more complex technology, and I'm getting more and more dubious about its actual existence and workability.  To be quite frank, unless I see lots and lots of demonstrations in the real world of this stuff working, I think it is a lot of wishful technothinking.  And I'm not convinced even if it was in place, it wouldn't create a huge agility barrier.

I'm in agreement with the people who say a UDDI service registry simply isn't going to be maintainable in the long term.  You might as well call it "the perpetually out-of-date service registry".

Chapter 9 - Enterprise Service Bus (ESB)

I'll willingly admit to some ignorance here.  We have certainly been discussion ESB at my organization, but we have yet to be convinced that it is necessary.

Again, there is a scale issue.  ESB is basically the reliable message queue idea.  You need a reliable message queue if every (service) message in your organization potentially has huge financial impact - that's the only thing that can justify the cost and complexity of an ESB.  For a big org, or one doing heavy financial stuff, this certainly applies.

To a library, loaning books for free?  If the occasional message drops, I think we can survive.

Chapter 10 - SOA Supervisor

This is currently indistinguishable from magic, which puts the lie to Arthur C. Clarke's famous maxim, because this is certainly far from being an advanced technology.

I'm not even convinced this is an existing technology.

Chapter 11 - SOA Governance

Excellent coverage of the transformational issues raised by moving to an SOA.  A must read for anyone doing a serious SOA initiative.

Chapter 12 - SOA Security

I'm not entirely convinced security is part of SOA, it's a separate service or services to be defined within the architecture.  But the kind of security they're talking about in this chapter is more "organization-wide complexity protection".  Again, the heavy-weight approaches they indicate are appropriate only to large-scale organizations. 

To get a taste, the essential pieces they recommend are:

• identity management
• software and data authentication
• audit trails

If you're not the scale or type of organization that has the above three concerns, you probably don't need to think too much about the type of security they are describing.

Chapter 13 - Where's the Data?

I have to admit a bias - I don't care about data.  I know however, that many EA people, such as Jane Carbone, consider data the essential starting point of analysis.  It is of course important that someone in the organization cares about the data, and this chapter has some useful guidance about the topic.

Chapter 14 - SOA Software Development

In this chapter, we get a little bit of "how".  But it's not really entirely about software development.

For example, I was again surprised to see yet another way to describe Enterprise Architecture, without saying "Enterprise Architecture".  This time they call it "building a business process map".

It then dives into iterative prototype development, which is all well and good, but not essential for SOA.  You certainly could develop services using the waterfall model.  Iterative prototype is certainly a good software development practice, but it doesn't have any strong connection to SOA in particular.

The last part of the chapter discusses Web Services and SOA testing, and this is an important topic which they cover well.

Chapter 15 - The Repository and the Registry

My concerns are as expressed about all these additional layers before:

• this is an important consideration for large organizations
• I'm not convinced anyone has technology that actually does this well in the real world
• I'm not convinced you can use it and still be agile

Chapter 16 - Do You Need a SOA?

Some relevant items for any organization, but some of their questions, e.g. #1 "Is Your Business Ecosystem Broad and Complex" apply again only to large organizations.  I think you can usefully have an SOA even if your business is simple and narrow.

Chapter 17 - Making Sure SOA Happens

Useful information, if a bit generically applicable to any type of major organizational change.

Chapter 18 - SOA Entry Points

Some reasonable guidance.

Chapters 19-26 are supposedly under the rubric "Real Life with SOA".  But I think more accurate would be "here are a series of chapters about major SOA vendors".  I was hoping for real organizational examples, but this section (which runs from page 225 to page 308) is mostly just "here is the SOA stuff from IBM / HP / BEA / Progress / Oracle / Microsoft / SAP / JBoss".  To me, it was a waste of almost 100 pages.  A single page pointing to the vendor SOA websites, with perhaps some of their key articles or white papers indicated, would have sufficed.

I think you've already seen from this review that SOA is a complex enough topic that 80 pages could have been usefully deployed for more explanation and for more discussion of the "how" of SOA.

We wrap up with Chapter 27 to 29, where we get some "Part of Tens" guidance and links, including even more vendor links.  My written notes conclude this part is "vaguely useful".

Overall, the book started very strong, and really could have been for the Rest of Us if they had continued in that vein, but instead it ends up being SOA for Really Big Business.

If that sounds like a good match for you, or if my review has piqued your interest, you can buy the book from my

SOA Book Store (powered by Amazon.com)
or
SOA Book Store (powered by Amazon.ca)

Wikipedia - Book Sources - ISBN 0470054352

EA Roundtable in Chicago, November 2006

UPDATE 2006-11-27: This conference was cancelled.  ENDUPDATE

CISTI's own Daping Tan will be presenting with the EA mentor/guru/consultant whose methodology we use, Jane Carbone, their session is

Integrating EA into the Organization - Roles, Processes, and Policies

The conference runs November 7-9, 2006 in Chicago.

Enterprise Architecture Roundtable 2006

If I have further info about this conference, I will be tagging it EAR2006.

Chunka Mui will be presenting on Emergent Knowledge.

You can view the slides (PDF) from a previous presentation Emergent Knowledge:The Spark for Information Advantage.

He's selling his essay on Amazon.com

but you can read an earlier version on his blog

Emergent Data

In a range of industries, including insurance, financial services, healthcare, and complex consumer and industrial products, emergent data is changing the strategic context in which companies manage their organizations, products, customers, and markets. In the right hands, new information about product and process conditions, customer preferences, and other environmental factors are sparking numerous innovations. But, because these data types are just appearing, their strategic significance is often not well understood, so it is usually ignored, and rarely leveraged.

SOA Service Ownership

We've been having some internal discussions about service ownership.  One of the things we found useful was separating the business-level service description, from the IT-level service implementation.

What we're currently thinking is that the business should own the business-level service ideas, but the IT-level implementation should have an IT owner.

This is important, because I think you can get a bit carried away in the "business drives technology" or "business-driven SOA" enthusiasm.  Yes, the business is in charge of generating and validating the business ideas, but they shouldn't be deep within your IT organization making technology decisions.

In addressing the question of service owners, Ali Arsanjani of IBM uses the concept of Service Domains:

Business/IT alignment. Great.

Each business line goes about their way to achieve it, but soon will find out that there is one conspicuous issue: who is the Service Owner? If I build this service and it is used in multiple places, who pays for the changes if the changes are those I do not specifically "care about" in my line of business?

Thus appears the notion of Service Domains, where a domain defines a set of related services that some "one" can own, maintain, support and (obtain) fund (ing).

To make a [change] to a service, contact the Service Domain Owner.

I guess it depends on the size of your organization and the number of services - I can certainly see large organizations with many services needing to take a Service Domain approach.

Ronan Bradley examines this in SOA Architect required: Only superheroes need apply

The key artefacts within a SOA are of course the service definitions (and their related data models). The benefits of reduced cost and increased flexibility rely on these being maintained, evolved and most importantly used as often as possible. So what is the role of the owner and what does he/she own? To my mind ‘owning’ in a business-aligned SOA environment should be in the sense of being responsible for making sure the services are used and used in the way to give most benefit to their business. This person can’t be our super-SOA architect who has all those other jobs!

The solution to me seems quite obvious – if a SOA is business aligned, the business must be more involved in SOA. this suggests that the owners of the service need to be directly aligned to the business owners of the associated business services. How this is done in practise will depend on the organization and its structures and goals. What will be true in every case is that the focus of this role should shift towards business skills and away from technical skills. The service owner can always go ask the architect for help and the architect can then focus again on ensuring that principles and guidelines are followed.

I think this is useful - the owners of the implementation should work with the business owners of the business services.  I am not convinced though that this means the implementation owner skillset must inevitable move towards a business focus - I think the implementation owner must retain a deep technical understanding, in order to be able to best advise the business of the impacts of changes.  If the implementation service owner is going to come to the architect all the time, you might as well just make the architect the implementation owner - the whole point of having a separate service owner is that the implementation owner has the power and ability to make decisions about the service (in consultation with stakeholders).  So to me, implementation owner is a primarily technical and coordination role.

I wonder if we can agree on some terminology.
business service seems fairly clear
but below that?  IT component owner?  implementation owner?  implemented service owner?  IT service owner?  What are you using in your organization?

Bradley points to a useful Gartner article that exposes some of the transformational impacts a fully-implemented SOA may have on your organization - The Organizational Impact of SOA

Then the idea of quality of service comes into play. Because an organization is going to a service-oriented architecture, one of the biggest transformations is going to be in delivering services that do what they are supposed to do, when they are supposed to do it. In other words, quality of service is more important now than the services that are actually delivered because quality of service is what determines whether someone can actually use what has been delivered.

The organization has to be ready and able to stand up and say, "Ah, we can track, manage, and deliver or guarantee delivery of a certain quality of service." In some organizations that may mean, actually bringing in or using part of the staff to be service providers, to become a service provision organization, as opposed to an application provision organization.

Two other issues come to mind – ownership and distributed transactions. Who owns what? Who owns the data, services, and processes, and how does one bill back for those processes? Who is responsible for the system that those processes run on? In terms of distributed transactions, an enterprise has processes that span organizations, business applications, computing systems, and even companies. And you have to understand when you are doing this transaction across a lot of different boundaries, how are you going to handle that?

Another issue comes as you build capabilities out of BPEL orchestrations of services.  Do the orchestrations then have owners?  How do you avoid having layer upon layer of owners? 

enterprise architecture library... thing

What do you do if you're in a technical library, but you are not, technically, a librarian?
You embrace the joys of cataloguing.  Repeatedly.
Our little architecture library has its own page of manually-entered titles on our internal wiki, and now
Steve has manually input them all into LibraryThing.

http://www.librarything.com/catalog.php?view=cistiarchitecture

I had come up with a clever way to get them into Universal Import, but LibraryThing Import is either paralyzingly slow, or broken, at the moment.

Of course, you can't actually borrow the books in our library (unless you're at NRC CISTI), so it's unclear how much public good we're providing.  I do plan to write some reviews - perhaps we're a node in an EA, SOA etc. book club?

Danish e-Government Architecture

On June 13 2003 the Danish Ministry of Science, Technology and Innovation published a white paper* on governmentwide enterprise architecture. This whitepaper was the starting point for the succeeding initiatives in the Danish architecture for eGovernment programme which covers all tiers of government. The paper Architecture for e-Government* from sums up the major initiatives up until March 2004.

A National OIO Enterprise Architecture Committee has been established. A National OIO Enterprise Architecture Forum covering the same subjects is open for all interested parties within government as well as within the private sector.

Ressources in english are marked with *.

from http://www.oio.dk/arkitektur/eng

The Interoperability Framework is the Danish e-Government Interoperability Framework.

The framework includes recommendations and status assessments for 609 selected standards, specifications and technologies used in e-government solutions.

This is version 1.2.14 of the framework. If you know the Interoperability Framework already, see What's new?. Newcomers might want to check the survival guide. All users should check the advanced search services, and of course also read the (updated) guidelines for use. Above all, everyone should dig into the 609 standards in the categories below.

from http://standarder.oio.dk/English/

There is a blog in English and Danish - Interoperability Framework Blog.  Here's the latest English posting, from February 13, 2006

Another standard has been added to the Framework. This time it is Role-Based Access Control (RBAC).

RBAC is a conceptual model, which uses employee assignments, functions, qualifications, responsibilities and authorizations as its starting point. RBAC is based on users, roles, role hierarchies, relations and limitations (e.g. separation of functions) and is a foundation for controlling who is able to do what, when, where, in what order etc.

As a part of the Danish government wide enterprise architecture a project focusing on a common approach to user and access management has been established. This project is among other things concerned with RBAC.

Previously:
April 30, 2006  Government of Ontario EA and SOA
April 30, 2006  Government of Canada EA and SOA
November 24, 2005  EA and SOA info overload, courtesy of John Gøtze

a new Enterprise Architecture blog

My colleague Stephen Anthony (contrary to popular belief, we are not the same person) has started an EA blog:

Enterprise Abstraction

He's already made the Business-Driven Architect blogroll - from newbie to 1337 in record time.

the Enterprise Architect as time traveller

To be an Enterprise Architect is to live in a bit of a temporal disconnect from the rest of the world.

This disconnect is two-fold:
1) Architecture, to my mind, must be looking at the continuously receeding 3-year time horizon.
That creates the critical bridge between the day-to-day and month-to-month activities of the organization, and the organization's stated 5-year strategic plan.
2) The people in your organization may not even be living in the present.  They may have a technological viewpoint that lags several years behind.

So you're in the meeting room in 2006, and you're trying to have a conversation based on where you think things will be in 2009, meanwhile, the organization is still back in 2001.

That is a huge challenge.
In fact, I would say one of the biggest single challenges is trying to find ways to move people's thinking forward.  And I don't have any solution.  So far my recommendation is:

inform and be ready

Tell the organization that based on their goals and the current technology environment, they will need X, do the necessary models to support X and then... wait.  Generally in about 9 months, after having completed their journey of discovery, the organization will come to you and say "we've just discovered, we need X".

Is this business-driven?
To me, as long as you are working from the strategic plan created by the business, you are business-driven.  A key role of the architect is to be the Keeper of the Plan, and the Great Reminder.

It's not that you can't change the plan, it's just that that should be a conscious decision, not a function of drift.  If you said your strategic goal for 2010 was to build a soaring stone cathedral, and you land in 2010 having built only a bunch of wooden houses and two sheds... to me that's the biggest single metric indicating success or failure of the EA.

Anyone else have ideas on key metrics to qualitatively or quantitatively assess the success of the EA?

Any thoughts on bridging the time warp between 3 and 5 year strategic visions and the day-to-day of project and operations execution?  Better ideas for moving the organization's technology thinking forward, rather than just "inform and be ready"?