Friday, August 26, 2011

Too big to fail


Have you been to Denver airport? It is one of the more modern airports in the US.  It handles vast numbers of travelers, with surges of visitors arriving for skiing weekends.  Many bring oversized ski gear, mountain bikes, various outdoors equipment.  The airport uses a manual system to handle the luggage. 

At the time Denver International Airport (DIA) was being built, there was a project to build a state-of-the-art automated luggage-handling system to improve speed and efficiency of the operations.  The luggage-handling system was being built by BAE Systems, an independent vendor who was originally hired for a smaller project.  When airport authorities sought bids for building the all-airport system, none of the submitted proposals promised to deliver this highly complex project within the offered timeframe.  So BAE was approached directly, and soon agreed to a fixed scope, schedule of under 2 years and fixed budget deal to deliver the all-airport baggage handling system.

As the project went on, it became apparent that the deal with BAE has been made in haste, many stakeholders in the system did not get a chance to provide their input.  Thus the project was pressured into accepting more changes later in the cycle.   Having started working on the system, BAE Systems ran into many technical problems both with system design and system constraints, while at the same time changes of directions were still being introduced.   In addition, death of Chief Engineer for the Denver International Airport and subsequent change of leadership complicated the project even further.

Throughout this ordeal, neither BAE Systems nor DIA team raised the question about terminating the project.  For nearly 3.5 years seasoned managers and engineers continued to press on (and spend taxpayers money) on a project that was late from the start, was handled by an inexperienced vendor, and was continuously pressured to agree to yet more last-minute changes, with no risk management or mitigation in place.

BAE Systems did not have the expertise in building systems of that magnitude, and was looking to gain experience on the Denver airport project so that it could pursue more big projects in the future.  While some people at BAE realized the risks and complexities, the management chose to press on because the stakes were so high.    DIA was shooting for a most modern, state-of-the-art airport facility, and deemed the project too big to fail. Both groups chose to ignore reality and expert advice in the pursuit of building the most complex system in the world.

DIA baggage-handling system of the 1990x is one of the more spectacular failures among IT projects to date.  Bad technical problems, poor planning, scheduling pressure, unexpected last-minute changes and complete lack of risk management all contributed to the situation.   The scary part is, all of these are present in most software projects.  High (or perceived high) stakes cause people to press on to the breaking point, demanding managers up the pressure, inexperienced technical people struggle with problems and plan poorly, over-confident customers insist on last-minute changes.   The successes still happen – often by a miracle of dumb luck, an amazing technical genius, blatant disregard to work-life balance, -- but not nearly as often as they could, and should.