Success of the Narwhal

The Atlantic has a great article about the technology behind the Obama Campaign - a large system called Narwhal.

Building great software for a political cause is inherently difficult.  In case of the presidential elections, the scale of the operation at its pick is grandiose.  Yet, there is no reason to build a high-quality product, since the goal is not to create good software, but to win just this once.  There is great pressure for one all-important day.   The deadline is set by the law of the United States, and is unlikely to move for any reason.

There are at least two large-scale examples on the books where technology projects for presidential campaigns crumpled early on the Election day, folding under the building pressure.  One is ORCA, Romney’s campaign effort to monitor voting results throughout the Election Day and to publish this information to its campaign workers.   The system crushed repeatedly, and in large parts was not available during most of the Election Day.

Another example of the political campaign technology that did not deliver is Houdini, a project developed by Obama 2008 Campaign, to track votes on the Election Day in 2008 via a hot-line.  The hot-line crashed early in the morning, not able to handle the deluge of calls.

So, it is very possible to build technology for political campaigns – and rather easy to have it fail at the critical moment.   But what was different about the successful system, Narwhal?

Narwhal achieved amazing scale: 4Gb/s, 10k requests per second, 2,000 nodes, 3 data centers, 180TB and 8.5 billion requests., as twitted by Scott VanDenPlas of the Obama for America technology team.  This is far beyond the capabilities of most existing software systems.

Narwhal was built by top-flight engineers – industry veterans from time-tested Google, Twitter, and other brand-name companies that have the know-how in scaling up software engines.  Yet, it was built by a team internal to the campaign, with unobstructed access to their direct customers.

The system relied on the modern technology built by others – open source software, cloud computing, many popular tools.   It was also build as a set of largely independent applications that had a way to exchange data, allowing to build and scale up each piece separately – and to limit the scope of inevitable bugs.  At the same time, Narwhal was built conservatively – staying away from shiny new technologies that haven’t been thoroughly tested by other projects, relying on experienced engineers, allowing technologists to work in their preferred languages and platforms.  The project had been tested and stressed-tested, and prepared for various contingencies.

Finally, Narwhal was released to the campaign staff in iterations.   First version was pushed out just a few short weeks after the technology team first got together, and it had been very limited.   It was scary, not exactly welcomed by other campaign staffers, who expected a working product, or nothing, and caused tensions.  Later versions followed, with more functionality and better integration, as the project was coming together. In the end, it helped to evolve the tools in the right direction, build trust in the technical team, and supersede user’s expectations.  Just as important, early and frequent delivery allowed users to receive first-hand experience with the software, grow and adapt with each consecutive version.

With the election done and gone, it is safe to say that the approach worked.  The software had less than 30 minutes of downtime on the Election Day, and proved immensely useful in a close race. And it paved the way to the future – for future where big campaigns, although not necessarily political campaigns, can rely on large-scale, user-friendly, success-delivering technology.

Smart technology in customer service

Modern technology is no small wonder.  Even more amazing are the simple, yet brilliant, solutions that small businesses put together, using simple, widely available hardware and software, to deliver better customer experience.

A popular restaurant on a Las Vegas Strip has the longest lines on Saturday mornings - and an excellent system for queue management.  A lady in front takes names and numbers for each party wanting to get a table, and types that info directly  into her iPad. Immediately after customer information is recorded, there is an SMS sent to customer phone informing her of the expected wait time, and that she will be notified via SMS when her table is ready. During the wait time, there are a few more updates sent to the customer about the remaining wait, followed by the SMS advising the customer to come up to the hostess stand to be seated. The stream of SMS is just enough to foster the warm-and-fuzzy feeling that the restaurant is working hard to serve the waiting customer soon. Using the phone for notifications also allows people to wonder around, and still not miss their table when it becomes available. 

The restaurant also offers customers to opt-in into their SMS-subscription list, to regularly receive information about food and drink specials.  This is a great way to reach customers directly, without depending on a 3d-party platform, like Facebook.  

Another interesting detail about this restaurant: it has lots of Yelp! reviews, and a huge number of pictures. Once customers get out their smartphones to communicate with the restaurant, they go on to snap pictures of food and drinks, and share them with the world. 

Another wonderful example of a small business creating magic with modern gadgets is one shoe vendor I met recently.  The lady uses her iPad for credit card processing, as many shop owners do, but then she goes a step beyond.  Every customer is emailed a customized receipt for each purchase, adorned by a picture of the customer in her brand-new shoes.

This system accomplishes several good results. The customers get to discuss the photo of the new shoes right away, an many immediately show it off to friends via social media. It completes the sale, and simplifies record keeping for both the seller and the client. The customer knows exactly why she spent [a significant amount of] money - beautiful shoes look lovely in the picture. In addition, the customer has an easy way to contact the vendor, if needed, simply by replying to the email.  And the vendor has built and saved a very informative customer profile: name and email, price paid, and information about product preferences.          

Follow rules of the road

A tall scrawny guy looked up at the street light, showing red for pedestrians, at the row of taxis waiting to make their turn, and started walking across the wide stretch of Las Vegas Blvd.   I heard him say “Oh, what’s the worst than could happen…” before his voice drowned in the noise of dozens of cars honking and slamming hard on the breaks.    

The signs were all there – the street light, clearly visible, shining bright red and showing a stick figure of a walker standing still.  The crowd – the typical Vegas crowd, rushing about their usual touristy business between the many temptations of the Sin City, was patiently standing on the sidewalk, waiting for the light to turn.  But this man somehow decided that he was different, that his getting on his way was more important, that the rules of the road did not apply to his situation.

Many teams that go through Agile adoption act similarly.  Scrum, a popular Agile process, comes with very detailed directions and roles.   While Scrum has a lot of ceremony, it is an easy process to follow, and it is all but guaranteed to deliver results when followed faithfully, in every detail, for at least a few months.   Except hardly anybody follows Scrum exactly as prescribed.    People tend to read through the directions and go: “This is all fine, but…”

What comes after is rarely original.   Very often people say that they do not have time to spend on daily meetings, iteration planning, or grooming the backlog.  That the project owner is not available once a month(!) to see and comment on the demo.  That their project is big and complicated, and therefore their sprints should be longer.  That there is not enough resources to use proper development practices. 

So the team goes forward with the Agile adoption, and takes up some new process, that cannot truthfully be called Scrum, or Agile, in all conscience. That process is still heavy on ceremony, uses similar terminology as Scrum, but largely allows the work to proceed as before – and to be stuck as before. 

It is as if a team was walking across the wide Las Vegas Strip against a clearly visible red light, away from a pedestrian crossing, but holding hands, as a team should. The cars are screeching to a halt all around, with drivers getting red in the face screaming at the silly tourists.   The team members respond by getting in a tight circle and holding their ground right there, in the middle of the road, while the traffic jam piles up all around them.

Granted, Scrum is an old and heavy process.  But it has proven itself over and over again, and has a great record of success – when followed exactly.   The same way it pays to obey the rules of the road, it makes sense to implement Agile processes to the letter.    The honking will stop, and the destination will be so much easier to reach. 

Software development and IT: Achieving success as an industry

This post is in response to the article "The failed record of the software industry – why software people don’t get it" by Jeffrey Palermo on the Headspring blog. 

Software is not like any other product it is often compared to.   It is fundamentally different from a car or a house.  Every project is different, even if only the team is different, so there is less predictability.   The industry is small,fast-moving, and there is no complete well-known set of templates and methodologies that guarantee timely delivery of a wide variety of tasks – simply because many of the best methodologies, practices, and platforms and tools are a work in progress.  The important result of these differences is that planning software projects is harder and achieves worse results, than planning in other industries.  

Industry surveys that try to measure success of software development and IT projects usually define success as being on time and within budget.   This is the usual metric for manufacturing and other traditional industries.  However, it is often a misleading measure for software development.

Consider wildly successful software projects, like Google search or Facebook social network.  Who cares whether these projects were on-time and on-budget.  They redefined how people live their lives, and delivered stunning ROI for investors, and the entire world.   

The industry-watching outfits should reconsider what they define as ‘success’ and ‘failure’ in software development and IT industry.  When a project does not deliver according to a time and budget projections, the reason is often not poor team performance, but incompetent estimates.  ROI and delivered value is a more meaningful measure, than adherence to an arbitrary timeline.

It is true that software industry does fail a lot, and it could, and probably will, do better as it matures.  However, on the B2B side of the industry, the clients need to broaden their understanding of how software development works.  Client engagement is very important for project success, and is very often the piece that causes either the project failure, or limiting projects’ potential for success.  Software engineers (or any person responsible for delivering software) know it, and the best ones have been pushing for it for decades now.    

The IT and software development industry can and should do more to educate clients about what it means to have a successful project. It is up to the software engineers to learn and to teach the society at large what is needed for the entire industry to be more successful, and to deliver value to clients and users.

Challenges of "challenged" projects

It is no secret that IT projects often fail.  That is expected, and it is overall a good thing – it takes more than one attempt to achieve success.   Other IT projects succeed, which is great all around.  And then there are projects that are considered “challenged”. 

“Challenged” projects are not the same as challenging.  Challenging projects gather enthusiasm of the team, and, hopefully, support from the rest of the organization.  In contrast, “challenged” projects are in enough trouble to not be considered a success, but they lack attention and support that would lead to reorganization - or cancellation.    These projects are project-management debt on the organization’s balance sheet, similar to technical debt a team can incur on a software project.

Here is what a “challenged” project looks like.  It is behind schedule.  The team morale is low.  Stress levels are high.  Technical debt is piling up, while the technical quality is deteriorating.  Project documentation is also deteriorating. Rising pressure to deliver, growing disconnect between the team and the rest of the organization.   Lots of time spent in long meetings, where blame is assigned and passed around.

The main difference between “challenged” and failed project is the finality.  Project that is declared a failure generally ceases to exist as a time, effort and money sink.  The hard decisions are made, and the team can move on.  In fact, when one of the Ambysoftsurveys asked about it, over 40% of respondents said that they consider canceling a troubled project a success.  However, as much as 40% of projects continue to exist in the “challenged” state in traditional and ad-hoc environments.  That means at any given time nearly half the organization is unproductive, upset, and is busy blaming their teammates.  

The Agile and iterative environments do much better – only 20% of projects are “challenged”, or, more likely, troubled projects spend less time in “challenged” state.  Iterative processes, including Agile, operate with short time-boxes, and require regular decision-making about the state of the project.   Regular integration and demos, required for Scrum and other iterative methodologies, exercise the pain points of IT developments, and emerging architecture, a hallmark of Agile, allows deal with technical challenges as they are encountered.   An involved Project Owner helps the rest of the team determine whether the product addresses the underlying business problem, which helps build the right product – or realize that an IT project is not doing well. Most importantly, Agile minimizes the stigma of failing – and lightens the burden of cancelling a project that is not worth saving.  

Canceling a failed project is always hard, but it opens new opportunities for the organization - opportunities to start over, to benefit from lessons learned, to boost morale and do better next time.  Agile helps achieve a fresh start.

What is Agile to you?

- Why would they need to talk?

A big man in the fourth row just could not see the point of having a daily conversation with coworkers. The rest of the audience beamed with helpfulness and were offering examples how helpful it can be to communicate with others. The man remained unconvinced.

Yet, there he was - in the middle of the Agile track at Houston TechFest, an open conference for developers and others involved in IT, on a bright summer Saturday. However skeptical this person was about talking to others at work, he chose to spend a weekend day talking to strangers about work. 

The popular stereotype, and old movies, show programmers as deeply introverted, hiding from the light of day, lonely guys with poor hygiene and terrible haircuts. In reality, the industry is populated by all kinds of people – men and women, homebodies and party animals, early risers and night owls; and proper grooming and professional appearance are a matter of course. Conferences like the TechFest tend to attract the best people in the field – those seeking to learn new things and to share their knowledge, to exchange ideas and discuss challenges.

Agile discussions have become a topic de rigueur when two programmers, or two thousand, get together. While simple, defined by just a few clear values, Agile philosophy goes against many a business practice and management theory. Agile methodologies are somewhat more cumbersome, but are far from the most complicated ones ever used in software development and IT. Still, the approach is a subject of very emotional debates, quasi-religious beliefs, and an occasional angry outburst.

More simply, Agile is about being human.  Throughout history, people learned to value one another, interact peacefully with one another, and roll with the changes. So should programmers.

Responding to life at work

"Can you do this real quick?"
"You should spend some time every day talking to people around you"
"Do not let your emails sit unanswered for too long"
"It is a good practice to tweet several times a day, and update blogs a few times a week"

When are we supposed to get any work done?

I am going to start from afar and consider a historical example. When computers were first deployed in the 50s and 60s, they did batch processing. A job was started, the machine chewed on it until done, and only then was ready to accept another job.

Technology has improved, and today's successful computers no longer employ batch processing.  All of surviving systems are interactive - while processing a task (virtually always) they are ready to respond to input in the timely manner.   A shrinking, yet non-trivial, portion of resources is allocated to the task of always being ready to respond.

Batch processing is largely dead because people who buy computers (which is all of us) prefer systems that respond quickly to systems that respond later and, sometimes, very much later.  People need complicated graphs and tables to understand how well a particular system performs on a complex task, but buyers make a quick and emotional judgement whether a machine is a "good one" based on quick response - or lack thereof.

The same kind of quick and emotional judgement applies to people: good people respond quickly and with predictable timing.   They smile back, answer questions, deliver results in a timely manner.  Excellent people keep up with the news, blog, tweet, and answer strangers' questions via email and various forums online and off.  Great people also take the time to signal "I'm here for you, should you need me" - by sharing advice with the larger world, working with user groups, mentoring, and following up with those they've connected with before.   The best people also find a moment to connect personally: pay attention to people around them, their needs and their motivation. 

So where are we supposed to get any work done? The grunt, technical, head-to-the-ground work that requires sharp technical skills and deep concentration, and constitutes the basis of a technical career.

The only answer is to learn to manage time according to priorities.  Responding to  people quickly and helpfully is very important, so is working with the larger world, and so is getting the grunt work done.  Multitasking does not work, so just apportion the time available to the tasks that need doing.  It helps to accept reality and basic rules of arithmetics when planning a workday. If there are two hour-long meetings, and you'd like to post work-related tweets twice a day, and keep your work day to 8 hours, realize that there is no way to spend 6 hours coding.   That is OK, and you will accomplish sufficient progress in the available time.  Just realize how much time you will dedicate to working on the code, and also leave time to spend reading and writing office email (organizational and technical), helping out coworkers, and getting water and latest office news from the water-cooler.  Blogging is best left for after-hours.

It is important to reserve time for urgent projects, keeping up with the news, maintaining relationships with colleagues past and present, while also getting regular work done at a sustainable pace.   Thinking clearly and deliberately about the daily schedule is important for managing time and productivity.

Men may be from Mars, but is it a different planet?

The difference in conversational styles continues to stun me.

I recently attended a reception for a group of highly educated and very successful people.  Successful women, including the honoree of the party, who the rest of the attendees were trying to woo, said things like:
- Oh, you probably know more than I do about this
- I can give a lecture, but it's hard to talk about me
- I am not a specialist in this

While men, even noticeably less successful men, said:
- Oh, but you don't understand
- You are wrong, this is not how it works
and overall did significantly more bragging, even for less momentous achievements.

Is the difference caused by nature or nurture?  If it is nurture, then who is doing the nurturing, and, more importantly, why?

More practical question: which way is better?
And what is "better" in this context - leads to greater success, allows for more peace of mind, something else?

How long is your game?

These days, the latest spin is king.  So much information is published so often and publicized so forcefully, it’s hard to remember what came before.  It is true in politics where GOP and Democrats fight it out for electoral votes, it is just as true in the office politics, where the winner gets a budget and a chance to build the project she is passionate about.  To win at the short game, often it is enough to be loud and clear, and to offer an  attractive soundbite.

Then there is the long game:  what happens after all the noise dies down, the news cycle moves on, and it is time to execute and produce results.  Often, it feels invisible and unimportant, since nobody is watching or cheering. Yet it matters a great deal. Although yesterday’s announcements are no longer in the news, the information remains available forever. 

So, do you play to win at a short or a long game? Do you work hard toward your announced goals after they are no longer exciting news?  Do you continue to uphold your values after everybody else has cooled down and moved on?  Your reputation is built on how your results relate to your promises. 

Having an audience, making promises and setting ambitious goals is exciting, saying what the client wants to hear is a great experience, and it is wonderful for relationships and for business.  On the other hand, executing on the promise often less glamorous, and involves tedious work.   New and exciting things are happening every day, and that makes it easy (and sometimes tempting) to forget what was said and promised earlier.

However, most serious projects take time and effort to build.  We have to be able to tune out the excitement of  the breaking news of the day long enough to concentrate on the technical side of the job and to get into the flow of work.   To step aside from the limelight and do the grunt work, which is not nearly as exciting as setting goals and producing soundbites.  

It's that non-glamorous, down-to-the-ground work that makes winners at the long game. It delivers results, innovations, and builds reputations.