20th century organisations with 21st century challenges

In general, people want their organisations to do wonderful things. Yet even when organisations are staffed with individually excellent employees, often the result is less than the sum of its parts. Or in other words, "Why can't my organisation do x?"

You can think of this as the lifecycle of an idea.

idea -> implementation -> operations

The above of course only works if you have infinite resources, to take the vast number of ideas, implement them all, and maintain them forever.  Since this is impossible, instead we try to talk about this using an analogy to living things, e.g. from seed to growing tree to mature tree to chopped down tree.

There are many different angles on this, in technology commonly we talk about the Software Development Life Cycle.  Unfortunately in most organisations, this is actually the Software Development Lie Cycle, for many reasons.

First is that the above line from idea to operations is woefully oversimplified.  It's actually a loop with current state turning into future state (which becomes the new current state)

(entire current organisation and current technology and idea environment) -> new ideas and improvements on current systems -> gateway -> prototypes and upgrade projects -> gateway -> new and existing operations (future state) -> death gateway -> decommisioned systems

There are at least three problems that organisations encounter as they go around this loop

1. The processes are bad at each step

2. The staff and management are unable to properly execute the processes, even if the processes themselves are good

3. The entire world changes while you're trying to move from your fixed current state to a wishfully-fixed future state

Specifically when it comes to information technology, big organisations are bad at IT.  Also small organisations.  Also software engineers.  Also, well, basically everyone.

The good news is we have a handle on some of the main problems.  In fact we've been trying to find solutions to these problems for decades.  The bad news is that solving these problems is very hard, and so quick fixes rather than real fixes are very tempting.

One of the underlying issues is the overall factory model we have both for producing and managing.  This is very deeply embedded both in every organisation as well as in the assumptions of every employee and manager.  It is hard to break out of a century-long mindset; in fact it's hard to realise you even have that mindset.

Software engineers have tried to attack this problem from all fronts: better languages (e.g. Ada from the US Defence Department), better methodologies (e.g. agile instead of waterfall), better overall context (e.g. Service-Oriented Architecture), better component sharing (e.g. github).

All of these are challenging because of the underlying factory organisation model.  For large organisations in particular, certain approaches pull the entire organisation as if by magnetism.  In particular, organisations will tend to do

* big projects (in people resources, dollars, and time elapsed)

* complex projects (e.g. many different technologies to be integrated)

* ambitious projects (e.g. trying to fix many problems at once)

* business-incremental projects (i.e. to add new features to a current process or product offering)

* perceived low-risk projects (e.g. with elaborate cost analysis and justifications)

* poorly-communicated projects (management doesn't communicate their intentions "down", IT doesn't communicate complexities "up" and comms doesn't talk to EA which doesn't talk to the coders who don't talk to... etc.)

Also business tends to measure certain costs while ignoring others (typically capital expenditures and coding time are carefully tracked, but having endless meetings is often considered to have zero cost).

This force pulls on all initiatives, pulling them towards having more stakeholders, more committees, more senior-level buy-in, more analysis, more consultations.  And once substantial effort is invested, the "sunk cost" psychology means that it is almost impossible to stop a running initiative, or to turn off a running system.

This force pulls towards bigness and complexity in general - towards big bang Enterprise Architecture, elaborate Service-Oriented Architecture, big-iron computers and networking hardware, "enterprise-class" heavyweight software backed by giant consulting firms... organisations somehow believe if they can just do big enough things with big enough technology, their problems will be solved.

There are many, many reports over decades that have documented this.  A recent one is System Error: fixing the flaws in government IT from the Institute for Government in the UK.

Pretty much everyone wants to fix this.   (Of course consultants may in some cases have perverse incentives to perpetuate dysfunctional processes that generate a lot of money for them, and managers may want to retain processes that give them power, but in general people want to do good work.)

One key problem is that usually people don't take a systems view.  They identify one problem area in the cycle and want to attack just that, thinking that will solve the entire problem.  For example:

* if we could just generate more / better ideas, everything else would work

* if we had a better planning process, everything else would work

* if we did more prototypes, everything else would work

* if we had more hackers, everything else would work

* if we changed development methodologies (e.g. to agile development), everything else would work

The reality of course is that unless you attack ALL steps in this cycle, the end results will always be, as they say, hashtag #fail

And even if you do each individual step well, you may still be choosing to do the wrong things.

Let's take a typical example.  Your organisation has an existing telephone system (PBX with voicemail) and knows to the penny most of the costs (in reality it probably knows most of the clear expenditures but has a more hazy idea of systems maintenance costs and opportunity costs).  Someone, probably at a senior level, has read about Voice over IP (VoIP).  Investigation initiated.  RFP written.  Consultants consulted.  ROI laid out.  Network upgrades needed.  Hardware to purchase.  Desktop software to integrate.  Options to evaluation.  Choices are made, purchasing, coding, implementing... a nice project that takes a year+ to replace a telephone on an employee's desk with... a telephone on an employee's desk.  Everyone happy.

Except in all this process no one stepped back to say: do we need the telephone at all?  Does anyone like using the telephone?  Are there other cheaper solutions?  What if we got rid of telephones altogether?  What do telephones do to knowledge work?

In fact, you can find whole threads of productivity literature that will tell you that telephone interruptions disrupt knowledge work, and that other methods of communication may be more effective.  In parallel the trends are clearly towards a global decline in voice traffic and a huge increase in data traffic.  People are not talking on their cellphones, they're texting and Facebooking.  Cartoons are appearing about the "phone gap", where texting users expect to receive a text asking whether a call can be made, rather than the phone just ringing out of the blue.

Over a decade ago in The Time Trap, Telephone Interruptions get an entire chapter

There is something wholly irresistible about a ringing telephone. ... Yet tune it out you must, for telephone interruptions can shatter your concentration and fracture your productivity as nothing else can.

(In fact the solution to this is quite simple, it just requires throwing out 20th Century convention and turning the ringer on your phone off, which is what I do.  My work phone and Blackberry never ring.  Although mostly these days they never ring because no one uses the phone any more.)

All this to say, even if somehow every stage of the giant VoIP project was perfectly executed, they still would have done a good job building the wrong thing.  But in general of course, every single stage is badly executed.

So you need a plan of attack for every stage.  At every stage you need BOTH better process and better-trained capable people.  Like a bike lane system which is barely used at 98% complete but starts to hum when every pathway is connected, you only get the big improvements when every stage of your ideas cycle works well.  In addition, we need the ability to continuously step back to the big picture and decide whether an entire initative still makes sense in light of changing conditions.

So you must

* know your current state well (most organisations have no idea what their actual processes are and are often shocked to see ridiculous legacy processes once they're diagrammed on a whiteboard)

* have a good process for gathering and generating ideas (often perceived as a major barrier, this is actually quite straightforward - the issue is rarely a shortage of ideas)

* have a very good process for gatewaying ideas to the investigation stage (making sure to let through both some high-risk prototype ideas and some improve-the-business ideas)

* be very very fast and agile at cranking out high-risk prototypes

* have a very good and very strict process for gatewaying prototypes into full implementation (sunk cost and idea attachment tend to make it hard to kill a running system no matter how many times it was emphasized that it was temporary)

* have a short process to take very focused features to production (e.g. a few months for a few very specific features)

* be adaptable as everything changes along the way

* have a process to kill things from production once they're no longer relevant (this is like the prototype killing problem, except multiplied by 100 as there may be decades of sunk cost)

* have a process to continuously review running initiatives and kill them (or modify them dramatically) if the world has changed while they were running

* Start tracking real employee costs not just specific activities and expenditures.  If you have to have a committee and burn $5000 of people time to decide to spend $500 on an Internet tool license, you have a costing misallocation problem.

At every stage just the process alone won't do it, you also need people who are creative, well-trained, adaptable, communicative, sharing their knowledge and documenting what they're doing as they go along.

So for example you might:

* immediately document your current systems (enterprise architecture)

* introduce an environment scanning process (and/or partner with existing horizon-scanning initiatives)

* devise a ruthless gating process that senior management can implement with input from all staff

* reward success and failure - reward successful products but also celebrate the lessons from prototypes that don't pass the gates

* create a prototyping group (innovation team, tiger team, skunkworks etc.) but ensure it has a close working relationship (or is simply a role within) the main implementation/operations team

* Ensure everyone is getting continuous training.  Including senior management.  Create time for reverse mentoring and innovation.

* Commit to FULLY FUNDING THE OPERATIONS of prototypes which pass the gate into full implementation

* Commit to annually review the relevance of running systems and to plan to kill the ones that need to retire entirely or be replaced with new approches

* If you are doing technology projects, ensure you aren't a technology island.  That means many different things including:

** the teams need full, unblocked Internet access, because that's where the information and the tools are

** the teams need to connect out to modern approaches and other developers - this means participating in local hackfests, sharing code on github, etc.

** the teams need to be able to share, publically, their challenges, the work they're doing - through blogs, presentations etc.

This is all a long, long way from our current factory model.  In the factory model, interchangeable employees toil on the factory floor, silently, while high above the managers gaze down benevolently from their glass-encased offices, and every project is initiated from senior management, for senior management.  It is a long way from a hierarchical 20th century factory organisation to a 21st century agile adhocracy that is both well-connected internally and well-connected out to the entire world of agile innovation.

Plus which, even if you have all of this top of mind, you can still fail.  Microsoft and Google hire the top computer scientists (and PhDs in many other areas) every year, AND have deliberate initiatives to attack every stage of this process, to try to ensure they don't fall into the big organisation trap... and yet both Microsoft and Google continue to fall into this trap in various ways.  The point being, even organisations that set out to build agility into their DNA like Google can fail, which is why you get endless "why Google can't build (small startup x)" or "why Yahoo Flickr didn't invent (new photo sharing approach y)".  In fact to prove my point here's both Why Google Can't Build Instagram AND Flickr Should Have Built Instagram. But They Didn’t. Here’s Why.

The fact of the matter is, whether your organisation is 300, 3000, or 300,000 it is never going to be the same as 8 people in a room with a coffeepot.  The best you can do is be mindful of that, and have a mix of internal improvements and standalone innovation, spinning in and out of your organisation continuously.

And that is my answer to @dbast's question

@sdkibb @jesgood @s_dav @mrpwalt @jeromefb how can I convince you to organize a full-day #goc "hackothon", besides that it would be awesome?

Our organisations are not incapable of coming together to create awesome prototypes.  The data.gc.ca prototype was done with a mix of dedicated resources and corner-of-desk, in 7 weeks.  It would be another year before it was made publically available.  GCPEDIA was done in months and has run for over two years, but it has yet to be operationally integrated or sustainably funded.  Hacking for hacking's sake will produce more orphaned prototypes that no one wants to sustain but no one is willing to shut down.

Hacking for hacking's sake also won't help if at least at a minimum the organisation itself has not recognized the problems of big projects and backed up that recognition with both public statements and effective internal initiatives.

I'm not saying hacking to demonstrate the art of the possible isn't great - it is.  Hackathons and apps contests have in many cases opened the eyes of CIOs to what can be done with openness combined with outreach and enthusiasm.  But unless you're willing to turn everything off the day after the demo, you still need to have an organisation-wide transformation in order to have a repeatable process to get from innovation through a gateway to full implementation and sustainable operations.

Basically the situation sucks.  This is not a factory, Taylorism doesn't apply to solve the entire problem.  We can't chop this up into smaller and smaller problems and make each piece more and more efficient.  If you have The Innovation Group and The Gateway Group and The Operations Group you just end up with efficient silos and an inefficient organisation.  This problem needs both bottom-up attacks to demonstrate local efficiencies and possibilities as well as systems thinking, a systematic organisation-wide attack on the problem as a whole.  Taylorism is easy and high-achievers get to be individual Linchpin superheros.  Systems change is hard.  But systems change is what we need.

Comments

[Gabrielcossette]

Very interesting post Richard! It's sure is a lot to think about ;)

When you were talking about prototypes, I thought of another great post from a scientist I know at the National Research Council of Canada :

"The merits of chasing many rabbits at the same time (part 1)"
http://alaindesilets.org/MyPublicSite/tiki-view_blog_post.php?postId=13

Also, here's an example of a "Hack Week" that Novell is sponsoring from time to time :
http://features.opensuse.org/hackweek

Any users submit new ideas and even join in the hacking with the employees! All for the good will of a Linux distribution that anyone can use (and that they also use in their business).