Success of the Narwhal

The Atlantic has a great article about the technology behind the Obama Campaign - a large system called Narwhal.

Building great software for a political cause is inherently difficult.  In case of the presidential elections, the scale of the operation at its pick is grandiose.  Yet, there is no reason to build a high-quality product, since the goal is not to create good software, but to win just this once.  There is great pressure for one all-important day.   The deadline is set by the law of the United States, and is unlikely to move for any reason.

There are at least two large-scale examples on the books where technology projects for presidential campaigns crumpled early on the Election day, folding under the building pressure.  One is ORCA, Romney’s campaign effort to monitor voting results throughout the Election Day and to publish this information to its campaign workers.   The system crushed repeatedly, and in large parts was not available during most of the Election Day.

Another example of the political campaign technology that did not deliver is Houdini, a project developed by Obama 2008 Campaign, to track votes on the Election Day in 2008 via a hot-line.  The hot-line crashed early in the morning, not able to handle the deluge of calls.

So, it is very possible to build technology for political campaigns – and rather easy to have it fail at the critical moment.   But what was different about the successful system, Narwhal?

Narwhal achieved amazing scale: 4Gb/s, 10k requests per second, 2,000 nodes, 3 data centers, 180TB and 8.5 billion requests., as twitted by Scott VanDenPlas of the Obama for America technology team.  This is far beyond the capabilities of most existing software systems.

Narwhal was built by top-flight engineers – industry veterans from time-tested Google, Twitter, and other brand-name companies that have the know-how in scaling up software engines.  Yet, it was built by a team internal to the campaign, with unobstructed access to their direct customers.

The system relied on the modern technology built by others – open source software, cloud computing, many popular tools.   It was also build as a set of largely independent applications that had a way to exchange data, allowing to build and scale up each piece separately – and to limit the scope of inevitable bugs.  At the same time, Narwhal was built conservatively – staying away from shiny new technologies that haven’t been thoroughly tested by other projects, relying on experienced engineers, allowing technologists to work in their preferred languages and platforms.  The project had been tested and stressed-tested, and prepared for various contingencies.

Finally, Narwhal was released to the campaign staff in iterations.   First version was pushed out just a few short weeks after the technology team first got together, and it had been very limited.   It was scary, not exactly welcomed by other campaign staffers, who expected a working product, or nothing, and caused tensions.  Later versions followed, with more functionality and better integration, as the project was coming together. In the end, it helped to evolve the tools in the right direction, build trust in the technical team, and supersede user’s expectations.  Just as important, early and frequent delivery allowed users to receive first-hand experience with the software, grow and adapt with each consecutive version.

With the election done and gone, it is safe to say that the approach worked.  The software had less than 30 minutes of downtime on the Election Day, and proved immensely useful in a close race. And it paved the way to the future – for future where big campaigns, although not necessarily political campaigns, can rely on large-scale, user-friendly, success-delivering technology.