Finding the value

What is the most valuable goal of  a project?  A software project is usually about getting the users features that they will enjoy and that will make them more productive, getting it out on time, and making it work well.  But what is more important:

• making users happy
• getting more users to use the application (but not necessarily elated)
• hitting the deadline
• never making a mistake
• being able to trace and correct every single mistake on the back end
• ... lots of other choices?

"All of the above" is never a good answer, because time and expertise are limited resources.

Sometimes an organization will have a clear preference, based on its overall culture and nature of the project.  For example, some systems are deadline-driven, such as software that supports Mother's day or Christmas sales season, systems used in elections, tax-preparation applications.  In others, being able to trace and correct an error is essential, for example, in software dealing with money.

However, in a lot of cases the project owner faces a choice, and being aware and deliberate in making that choice can make or break a project.

Agile steps

The teacher looked a bit glamorous and very assured of himself.   “The most important thing is steps.  We do a step forward, a step sideways, a step backward, and another sideways.”   The class tried, lots of stepping on other people’s feet ensued.  A few students missed the first bit, and found themselves off by a step, confusing their neighbors.   

A brave soul raised his hand and asked out loud: “What if people next to me are doing something else?” The teacher responded with authority: “They will know to do these steps.”

A lot of Agile learning takes their students through similar ordeal.  Steps are shown, expectations are announced, and students are told that the rest of the world will just fall in line.  With that people are sent out to their regular workplaces and teams to create beautiful new work successes with Agile. 

Class discussion is often framed differently than in the real world, and there is a lot of pressure to agree with the teacher and with the other students.  Make no mistake, a group of professionals taking an Agile training does not represent a regular workplace crowd, a typical organization, or a normal project team. Students participating in an Agile class are different in that they all agreed to learn Agile, are curious and open-minded enough to consider a different work philosophy and methodology, and are willing to challenge themselves with training.   

When the class is over, and everybody goes back to work, the students will need something other than just Agile steps or procedures to guide their every day work. If the organization is serious about switching to Agile and getting the benefits of the new approach, everybody needs to get the underlying philosophy, the ways to get into the rhythm and on the right step.  The team needs to be able to stay in sync,  while realizing that someone will inevitably drop the ball and require help.  When going through training in Agile, practitioners must learn to realize when they’re off beat, when other people are off, and how help the team to all follow the rhythm.    

Steps are necessary, but they are simple and easy to do. Agile training should be about being and doing Agile in the real world, where someone will always be off-beat, and not know the steps.    

---

Registration for AgileDotNet 2013 in Dallas, TX is open now - the most advanced discussion on Agile and technology in the real world.

Success of the Narwhal

The Atlantic has a great article about the technology behind the Obama Campaign - a large system called Narwhal.

Building great software for a political cause is inherently difficult.  In case of the presidential elections, the scale of the operation at its pick is grandiose.  Yet, there is no reason to build a high-quality product, since the goal is not to create good software, but to win just this once.  There is great pressure for one all-important day.   The deadline is set by the law of the United States, and is unlikely to move for any reason.

There are at least two large-scale examples on the books where technology projects for presidential campaigns crumpled early on the Election day, folding under the building pressure.  One is ORCA, Romney’s campaign effort to monitor voting results throughout the Election Day and to publish this information to its campaign workers.   The system crushed repeatedly, and in large parts was not available during most of the Election Day.

Another example of the political campaign technology that did not deliver is Houdini, a project developed by Obama 2008 Campaign, to track votes on the Election Day in 2008 via a hot-line.  The hot-line crashed early in the morning, not able to handle the deluge of calls.

So, it is very possible to build technology for political campaigns – and rather easy to have it fail at the critical moment.   But what was different about the successful system, Narwhal?

Narwhal achieved amazing scale: 4Gb/s, 10k requests per second, 2,000 nodes, 3 data centers, 180TB and 8.5 billion requests., as twitted by Scott VanDenPlas of the Obama for America technology team.  This is far beyond the capabilities of most existing software systems.

Narwhal was built by top-flight engineers – industry veterans from time-tested Google, Twitter, and other brand-name companies that have the know-how in scaling up software engines.  Yet, it was built by a team internal to the campaign, with unobstructed access to their direct customers.

The system relied on the modern technology built by others – open source software, cloud computing, many popular tools.   It was also build as a set of largely independent applications that had a way to exchange data, allowing to build and scale up each piece separately – and to limit the scope of inevitable bugs.  At the same time, Narwhal was built conservatively – staying away from shiny new technologies that haven’t been thoroughly tested by other projects, relying on experienced engineers, allowing technologists to work in their preferred languages and platforms.  The project had been tested and stressed-tested, and prepared for various contingencies.

Finally, Narwhal was released to the campaign staff in iterations.   First version was pushed out just a few short weeks after the technology team first got together, and it had been very limited.   It was scary, not exactly welcomed by other campaign staffers, who expected a working product, or nothing, and caused tensions.  Later versions followed, with more functionality and better integration, as the project was coming together. In the end, it helped to evolve the tools in the right direction, build trust in the technical team, and supersede user’s expectations.  Just as important, early and frequent delivery allowed users to receive first-hand experience with the software, grow and adapt with each consecutive version.

With the election done and gone, it is safe to say that the approach worked.  The software had less than 30 minutes of downtime on the Election Day, and proved immensely useful in a close race. And it paved the way to the future – for future where big campaigns, although not necessarily political campaigns, can rely on large-scale, user-friendly, success-delivering technology.