Software development and IT: Achieving success as an industry

This post is in response to the article "The failed record of the software industry – why software people don’t get it" by Jeffrey Palermo on the Headspring blog. 

Software is not like any other product it is often compared to.   It is fundamentally different from a car or a house.  Every project is different, even if only the team is different, so there is less predictability.   The industry is small,fast-moving, and there is no complete well-known set of templates and methodologies that guarantee timely delivery of a wide variety of tasks – simply because many of the best methodologies, practices, and platforms and tools are a work in progress.  The important result of these differences is that planning software projects is harder and achieves worse results, than planning in other industries.  

Industry surveys that try to measure success of software development and IT projects usually define success as being on time and within budget.   This is the usual metric for manufacturing and other traditional industries.  However, it is often a misleading measure for software development.

Consider wildly successful software projects, like Google search or Facebook social network.  Who cares whether these projects were on-time and on-budget.  They redefined how people live their lives, and delivered stunning ROI for investors, and the entire world.   

The industry-watching outfits should reconsider what they define as ‘success’ and ‘failure’ in software development and IT industry.  When a project does not deliver according to a time and budget projections, the reason is often not poor team performance, but incompetent estimates.  ROI and delivered value is a more meaningful measure, than adherence to an arbitrary timeline.

It is true that software industry does fail a lot, and it could, and probably will, do better as it matures.  However, on the B2B side of the industry, the clients need to broaden their understanding of how software development works.  Client engagement is very important for project success, and is very often the piece that causes either the project failure, or limiting projects’ potential for success.  Software engineers (or any person responsible for delivering software) know it, and the best ones have been pushing for it for decades now.    

The IT and software development industry can and should do more to educate clients about what it means to have a successful project. It is up to the software engineers to learn and to teach the society at large what is needed for the entire industry to be more successful, and to deliver value to clients and users.

Challenges of "challenged" projects

It is no secret that IT projects often fail.  That is expected, and it is overall a good thing – it takes more than one attempt to achieve success.   Other IT projects succeed, which is great all around.  And then there are projects that are considered “challenged”. 

“Challenged” projects are not the same as challenging.  Challenging projects gather enthusiasm of the team, and, hopefully, support from the rest of the organization.  In contrast, “challenged” projects are in enough trouble to not be considered a success, but they lack attention and support that would lead to reorganization - or cancellation.    These projects are project-management debt on the organization’s balance sheet, similar to technical debt a team can incur on a software project.

Here is what a “challenged” project looks like.  It is behind schedule.  The team morale is low.  Stress levels are high.  Technical debt is piling up, while the technical quality is deteriorating.  Project documentation is also deteriorating. Rising pressure to deliver, growing disconnect between the team and the rest of the organization.   Lots of time spent in long meetings, where blame is assigned and passed around.

The main difference between “challenged” and failed project is the finality.  Project that is declared a failure generally ceases to exist as a time, effort and money sink.  The hard decisions are made, and the team can move on.  In fact, when one of the Ambysoftsurveys asked about it, over 40% of respondents said that they consider canceling a troubled project a success.  However, as much as 40% of projects continue to exist in the “challenged” state in traditional and ad-hoc environments.  That means at any given time nearly half the organization is unproductive, upset, and is busy blaming their teammates.  

The Agile and iterative environments do much better – only 20% of projects are “challenged”, or, more likely, troubled projects spend less time in “challenged” state.  Iterative processes, including Agile, operate with short time-boxes, and require regular decision-making about the state of the project.   Regular integration and demos, required for Scrum and other iterative methodologies, exercise the pain points of IT developments, and emerging architecture, a hallmark of Agile, allows deal with technical challenges as they are encountered.   An involved Project Owner helps the rest of the team determine whether the product addresses the underlying business problem, which helps build the right product – or realize that an IT project is not doing well. Most importantly, Agile minimizes the stigma of failing – and lightens the burden of cancelling a project that is not worth saving.  

Canceling a failed project is always hard, but it opens new opportunities for the organization - opportunities to start over, to benefit from lessons learned, to boost morale and do better next time.  Agile helps achieve a fresh start.