Success of the Narwhal

The Atlantic has a great article about the technology behind the Obama Campaign - a large system called Narwhal.

Building great software for a political cause is inherently difficult.  In case of the presidential elections, the scale of the operation at its pick is grandiose.  Yet, there is no reason to build a high-quality product, since the goal is not to create good software, but to win just this once.  There is great pressure for one all-important day.   The deadline is set by the law of the United States, and is unlikely to move for any reason.

There are at least two large-scale examples on the books where technology projects for presidential campaigns crumpled early on the Election day, folding under the building pressure.  One is ORCA, Romney’s campaign effort to monitor voting results throughout the Election Day and to publish this information to its campaign workers.   The system crushed repeatedly, and in large parts was not available during most of the Election Day.

Another example of the political campaign technology that did not deliver is Houdini, a project developed by Obama 2008 Campaign, to track votes on the Election Day in 2008 via a hot-line.  The hot-line crashed early in the morning, not able to handle the deluge of calls.

So, it is very possible to build technology for political campaigns – and rather easy to have it fail at the critical moment.   But what was different about the successful system, Narwhal?

Narwhal achieved amazing scale: 4Gb/s, 10k requests per second, 2,000 nodes, 3 data centers, 180TB and 8.5 billion requests., as twitted by Scott VanDenPlas of the Obama for America technology team.  This is far beyond the capabilities of most existing software systems.

Narwhal was built by top-flight engineers – industry veterans from time-tested Google, Twitter, and other brand-name companies that have the know-how in scaling up software engines.  Yet, it was built by a team internal to the campaign, with unobstructed access to their direct customers.

The system relied on the modern technology built by others – open source software, cloud computing, many popular tools.   It was also build as a set of largely independent applications that had a way to exchange data, allowing to build and scale up each piece separately – and to limit the scope of inevitable bugs.  At the same time, Narwhal was built conservatively – staying away from shiny new technologies that haven’t been thoroughly tested by other projects, relying on experienced engineers, allowing technologists to work in their preferred languages and platforms.  The project had been tested and stressed-tested, and prepared for various contingencies.

Finally, Narwhal was released to the campaign staff in iterations.   First version was pushed out just a few short weeks after the technology team first got together, and it had been very limited.   It was scary, not exactly welcomed by other campaign staffers, who expected a working product, or nothing, and caused tensions.  Later versions followed, with more functionality and better integration, as the project was coming together. In the end, it helped to evolve the tools in the right direction, build trust in the technical team, and supersede user’s expectations.  Just as important, early and frequent delivery allowed users to receive first-hand experience with the software, grow and adapt with each consecutive version.

With the election done and gone, it is safe to say that the approach worked.  The software had less than 30 minutes of downtime on the Election Day, and proved immensely useful in a close race. And it paved the way to the future – for future where big campaigns, although not necessarily political campaigns, can rely on large-scale, user-friendly, success-delivering technology.

Smart technology in customer service

Modern technology is no small wonder.  Even more amazing are the simple, yet brilliant, solutions that small businesses put together, using simple, widely available hardware and software, to deliver better customer experience.

A popular restaurant on a Las Vegas Strip has the longest lines on Saturday mornings - and an excellent system for queue management.  A lady in front takes names and numbers for each party wanting to get a table, and types that info directly  into her iPad. Immediately after customer information is recorded, there is an SMS sent to customer phone informing her of the expected wait time, and that she will be notified via SMS when her table is ready. During the wait time, there are a few more updates sent to the customer about the remaining wait, followed by the SMS advising the customer to come up to the hostess stand to be seated. The stream of SMS is just enough to foster the warm-and-fuzzy feeling that the restaurant is working hard to serve the waiting customer soon. Using the phone for notifications also allows people to wonder around, and still not miss their table when it becomes available. 

The restaurant also offers customers to opt-in into their SMS-subscription list, to regularly receive information about food and drink specials.  This is a great way to reach customers directly, without depending on a 3d-party platform, like Facebook.  

Another interesting detail about this restaurant: it has lots of Yelp! reviews, and a huge number of pictures. Once customers get out their smartphones to communicate with the restaurant, they go on to snap pictures of food and drinks, and share them with the world. 

Another wonderful example of a small business creating magic with modern gadgets is one shoe vendor I met recently.  The lady uses her iPad for credit card processing, as many shop owners do, but then she goes a step beyond.  Every customer is emailed a customized receipt for each purchase, adorned by a picture of the customer in her brand-new shoes.

This system accomplishes several good results. The customers get to discuss the photo of the new shoes right away, an many immediately show it off to friends via social media. It completes the sale, and simplifies record keeping for both the seller and the client. The customer knows exactly why she spent [a significant amount of] money - beautiful shoes look lovely in the picture. In addition, the customer has an easy way to contact the vendor, if needed, simply by replying to the email.  And the vendor has built and saved a very informative customer profile: name and email, price paid, and information about product preferences.          

Follow rules of the road

A tall scrawny guy looked up at the street light, showing red for pedestrians, at the row of taxis waiting to make their turn, and started walking across the wide stretch of Las Vegas Blvd.   I heard him say “Oh, what’s the worst than could happen…” before his voice drowned in the noise of dozens of cars honking and slamming hard on the breaks.    

The signs were all there – the street light, clearly visible, shining bright red and showing a stick figure of a walker standing still.  The crowd – the typical Vegas crowd, rushing about their usual touristy business between the many temptations of the Sin City, was patiently standing on the sidewalk, waiting for the light to turn.  But this man somehow decided that he was different, that his getting on his way was more important, that the rules of the road did not apply to his situation.

Many teams that go through Agile adoption act similarly.  Scrum, a popular Agile process, comes with very detailed directions and roles.   While Scrum has a lot of ceremony, it is an easy process to follow, and it is all but guaranteed to deliver results when followed faithfully, in every detail, for at least a few months.   Except hardly anybody follows Scrum exactly as prescribed.    People tend to read through the directions and go: “This is all fine, but…”

What comes after is rarely original.   Very often people say that they do not have time to spend on daily meetings, iteration planning, or grooming the backlog.  That the project owner is not available once a month(!) to see and comment on the demo.  That their project is big and complicated, and therefore their sprints should be longer.  That there is not enough resources to use proper development practices. 

So the team goes forward with the Agile adoption, and takes up some new process, that cannot truthfully be called Scrum, or Agile, in all conscience. That process is still heavy on ceremony, uses similar terminology as Scrum, but largely allows the work to proceed as before – and to be stuck as before. 

It is as if a team was walking across the wide Las Vegas Strip against a clearly visible red light, away from a pedestrian crossing, but holding hands, as a team should. The cars are screeching to a halt all around, with drivers getting red in the face screaming at the silly tourists.   The team members respond by getting in a tight circle and holding their ground right there, in the middle of the road, while the traffic jam piles up all around them.

Granted, Scrum is an old and heavy process.  But it has proven itself over and over again, and has a great record of success – when followed exactly.   The same way it pays to obey the rules of the road, it makes sense to implement Agile processes to the letter.    The honking will stop, and the destination will be so much easier to reach.