Good Agile teams break the rules

Agile frameworks are sets of rules. To become Agile, an organization or a team learns these rules, tries to understand what the rules mean, and executes according to the rules. Not necessarily in that particular order.

Good teams learn rules quickly and follow the rules well. Or do they?

Agile philosophy is all about learning, gathering information and feedback, making small changes, and gathering feedback again. As teams mature and understand the philosophy of being Agile, a rule may get re-assessed, adjusted, and sometimes even completely disregarded. The team will consider the outcome, and either go back to the previous version of the rule, stick with the updated rule, or make another change based on new learnings.

Consider this example, discussed by Mike Cohn: standard Agile rule recommends that the team works on backlog items in the order set by Product Owner. But depending on the technical details of the project, the team may be able to maximize delivered value if it is allowed to do minor adjustments to the order of stories.  

Another standard Agile rule that is often broken is to use collocated teams only. This rule is often not followed due to business constraints, and in those cases breaking this particular rule can and often does lead to serious problems. However, more mature Agile teams that choose to let some members work remotely part of the time, can stay productive and perform on par with completely collocated teams. Being able to occasionally work from home helps people concentrate, allows to save time on the commute, and overall improves enjoyment and morale of the team, which promotes better productivity.

Rules of the Agile frameworks are designed to help teams create a situation that encourages learning, taking responsibility, and enjoy their work. Teams that have developed a deep understanding of the reasons behind the Agile rules, should be allowed and encouraged to modify the rules to fit the details of their physical environment, particular complexities of the project, and organizational specifics. A mature self-managed Agile team can be trusted to develop their own rules for best performance.

Myths and half-truths about estimates

Estimating is in our DNA. It is about the future, the knowledge, the power – all the things that make us feel in control.  Estimates are required, and are the most straightforward (or even the only) way to gain trust, and budget, to go forward.

Yet… it is an open secret that estimates are flawed. Things do not go as planned. Resources become unavailable, assumptions turn out to be incorrect, changes trickle down in a never-ending stream, and communication becomes strained.

While having flawed estimates may be preferable in some situations to having none, commitment to wrong numbers can wreck havoc on any project. If you are going to estimate, steer clear of these mistakes people frequently make, even if they have plenty of experience with estimating.

• Myth: If your estimates are a range of dates, you are doing a good job managing expectations.
• Only the earlier, lower, more magical numbers will be remembered. And those will be accepted as firm commitments.
• The lower bound is usually set at "the earliest date the project can possibly be completed". In other words, there is absolutely no way the work can be completed any earlier, even by a day. What are the chances of hitting that exact date? Practice shows - close to nil. 

• Half-truth: You can control the quality of your estimate by putting more work into producing this estimate.
• By spending some time learning about the project, researching resources available, considering major and minor risks, one can produce a somewhat better estimate.
• The above activities are only going to take the quality of the estimate so far.  Past a certain point, no matter how much effort goes into estimating a project, the quality of the estimate is not going to improve. Then the best bet is to simply start working on the project.

• Myth: People can learn to estimate better, as they gain experience.
• It is possible to get better at estimating – if one keeps estimating the same task, which becomes known and familiar with experience. This is hardly ever the case in software development. Every project is different, most teams are brand new, and technology is moving along fast enough.
• Do not expect to produce a better estimate for your next project than you did for your last one.
•  By the same token, do not expect a worse estimate. The quality of the estimate is going to be low, and it is going to be random.

• Half-truth: it is possible to control the schedule by giving up quality.
• Only for short-term, throw-away temporary projects.
• For most projects, aiming for lower quality has a negative effect on the schedule.

The human side of enterprise

Once upon a conference, I asked a small technology business owner about the interactions his 12-person team had, and he was sincerely surprised by the question. “Why would they need to talk? I tell them what to do, and then they code, each in their cubicle.” – was his response.

We like to think about work strictly as job function-dictated activities. Gathering requirements, evaluating options, making product decisions. Meeting to discuss who does what, how hand-offs happen, what are the acceptance criteria. Org chart-based relations that dictate who is in charge, who reports to whom, and who will get the credit or blame. The result of these interactions is work product, the how-to of getting to a solution.

Yet, there is more to work than “just work”. It is also the true, human relationships we develop with people we spend a lot of time with most days of the week. A quick exchange by the water cooler about a concert, a site of a geeky new mouse pad on someone’s desk, kids’ pictures on the screen. A small group is discussing the latest smartphone while getting coffee; when waiting for a meeting to start someone mentions that she is looking for a rock-climbing gym; and somebody else is asking for advice on the user groups during lunch.  The result of these conversations are not necessarily a work product (although it could be directly useful for work, too), but the building of a team of friends.

 

A team comes together through learning small, personal, often seemingly insignificant details about each other, discussing experiences, finding shared interests. Working with friends matters. It is a lot easier to be open-minded and welcoming to a friend’s ideas, than to someone’s who is a stranger. Friends deal with disagreements by working together, while strangers get upset and push each other away. Knowing and liking people one works with makes everyone happier and more productive.

When the opportunities for social interactions within a team are limited, work suffers. Teams of strangers are less interested in working together cooperatively, sharing information, establishing and striving toward a common goal. There is more shirking, less innovation and leadership. And less success.

Simple solutions to complex problems

Complexity and simplicity are often, and intuitively,
regarded as two extremes of the same continuum,
or spectrum. Yet, this may be a simplistic view, indeed.

Complexity of a solution reflects many things: complexity of the problem, of the problem domain, the level of understanding of the above by the people creating the solution, and the complexity of the organization and communication of the people in charge of building the solution.  Finally, complexity of the solution will reflect the incentives people were facing: was it in their interest to create a complex solution? Or to make the solution as simple as possible?

Integer calculator is inherently less complex than, say, mapping software, but implementation of the algebra functions can be ridiculously complicated. Or fairly simple, depending on the attitudes of individual architects and implementers, and their organization.  Business rules expressed as decision trees are relatively simple, yet there are plenty of solutions in that space that are mind-bogglingly complicated.

Agile approach results, among many other things, in reduced complexity in the delivered product. Cross-functional self-managed teams, while not easy to build, are a lot less complex than matrix organizations of the traditional firms.  Communication patterns are simpler and more straightforward. Preferring a team of generalists over narrowly specialized experts levels out the playing field, and encourages more level understanding of the problem domain within the team. The emphasis on speedy delivery of small slices of the solution leads to the team concentrating on the “lowest hanging fruit” at any point in project timeframe, ensuring that the team works on the best-understood pieces of the problem, while it expands its understanding.

Engineering practices that are often coupled with Agile also carry a significant focus on simplicity in design and implementation. Emerging design, developed with continued and immediate feedback from the implementation and user assessment, is simpler than the up-front design created early in the project, prior to having project work feedback loop established. Unit-level automated tests are simple to create and maintain, and having adequate test coverage and infrastructure reduces both probability and complexity of inevitable bugs.

Frequent collaboration with users and customers, required by most Agile frameworks, forces all parties involved in a project to develop a joint understanding that eventually becomes the basis of the solution. Joint understanding of a diverse group is always less complex and less nuanced than that of a single mind (participating in the group), or a similarly-minded group, thus leading to a simpler solution.

As the business of software development moves forward, it is taking on more complicated problems, and attempts to deliver increasingly complex solutions. Keep the complexity in check by simplifying communication paths, promoting learning and feedback, and striving for joint understanding among diverse groups. 

Never talk to strangers

People as species have evolved in social groups, cooperating and competing. Simply sharing a space helps to build trust, create a body of shared knowledge, establish social customs. Patriotism, love and pride for one’s land, interest in tails of the forefathers are all feelings that knit together a community of people that live and interact in close proximity.

But when it comes to interacting with people from far away, the situation is different. Historically, people living afar have been perceived as either villains or gods. People who are not participating in every day events in person, if they are not completely disregarded, appear weird, scary, and are ascribed bad thoughts and behaviors. Tribes everywhere are known to have fought long-lasting wars with neighbors living just far enough to not be part of the community.

The entire human history has prepared us to like and trust those close to us, and expect bad things from people who we cannot see, hear, touch and smell. We may not be consciously aware of this bias, but it is impossible not to notice its consequences. Long-distance relationships have a dramatically lower rate of survival, compared with relationships where partners see each other regularly. Most business activity still happens locally, within a single geographic area. Despite the televised debates and online advertising, politicians still fight and win elections by talking to voters on the campaign trail, and building organizations of supporters to do that on candidates’ behalf.

When an organization is working with a remote team, the people on that team are outsiders, not part of the tribe. Remote team members, unable to participate in daily face to face interactions, are subject to all the suspicions and biases outsiders have suffered throughout the human history. To create trust and build good working relationships all sides must work against their natural instincts of distrust toward outsiders.

There are many things that could help make the remote team a part of the inner circle. It could be as simple as giving a voice on the phone a benefit of the doubt – and a chance to explain in greater details how they came up with the ideas being presented. Or as complicated as organizing regular trips to a common location, to let people spend time together, working face to face. Whichever way is chosen, it is still hard work, requiring intelligence, an open mind, and willingness to learn.  

<

The "family" objection

We recently held a panel titled “Software development is a team sport: Women In Technology” at Houston TechFest in Houston and Austin CodeCamp in Austin, Texas.  The goal was to have a conversation about men and women working together in IT and software development industry. Research shows that we are more productive because of our differences, not in spite of our differences, when we manage to work together. We had great participation: men and women shared stories, concerns from their experience, concerns they have for their daughters, difficult situations and techniques that worked in different environments.

Each time, there surfaced an old-time objection to hiring women:

-          Why hire a woman, who may get pregnant and go on leave for weeks, and then perhaps be somewhat distracted for a few more months, when one can perfectly well hire a man, who is not in any danger of ever bearing a child?

This entire line of reasoning strikes me as odd.

Fertility rate in the US is 1.9 children per woman, i.e. over a lifetime an average woman is expected to have 1.9 kids. Women with more education tend to have fewer children, so for women with a college degree the average is just 1.1 children. 

A college graduate typically has a 45+ year career, male or female, first entering the workforce at age 22 and working at least until his or her late 60s.  Today’s typical tenure in IT at a given position, job or company is 2-5 years.

So, a woman interviewing for a job today, has an approximately 2-in-45 chance to have a child while working at this job.  If she already has a child, that chance falls to 2-in-450. 

In order to get this interview, this woman has shown herself to be a stronger candidate than, typically, 2 to 100+ other people, who also applied for this position.  She still has to beat 2+ other candidates, also invited to interview for this position. Overall, her chances of becoming the top candidate are between 1-in-4 and 1-in-100, which makes her (or anyone reaching that point) quite awesome!

Now, consider another angle. Productivity among mediocre, good and great developers varies by orders of magnitude.  Hiring the best person for the team makes a significant difference in bottom-line productivity, quality, time-to-market and maintainability of the project.  A woman who is x10 as productive as the next candidate, even if she takes 2-4 months (over her 2–5 year tenure in a particular job) to take care of her family, is still a net win for the team with a realized productivity gain of 90%+.  And this is the worst-case scenario!  In a typical case, your best candidate will deliver an even better value, regardless of her gender.

The argument about women's propensity to take time off to have children makes no sense in the context of hiring the best person for a software development job. Any organization's best bet is to identify and hire the best person they can get, and then make sure that person stays with them as long as possible. Regardless of their gender.

Discussion about good code at DallasTechFest 2014

Many programmers are dedicated, or even obsessed, with writing “good” code.

Often we follow certain patterns and practices because we believe they are the best solution for the problem, or because the experts tell us so. There are many quasi-religious wars about proper ways to write code; those wars last years, inflame the most peaceful communities, and make many heads spin.

While it is easy to argue whether a particular piece of code is good, it is hard to come up with an overall idea of what makes code good. We had an interesting discussion about good code at Dallas TechFest, with a great crowd of fellow developers.

@jprusakova speaking on the aspects of good code. Totally full room! @dallastechfest #dtf2014 pic.twitter.com/6o0AyQ8YbA
— Ty Crockett (@TTCrockeTT) October 10, 2014

Stretch your mind

“The more you know,
the more you realize you know nothing.”
– Socrates

There is a big push on young engineers (and colleges that educate them) to build up a wide variety of popular, currently in-demand skills. More popular technologies, common platforms, more courses and projects give students a better chance to hit every bullet point in job recs. It is favorably considered by recruiters and looks extremely attractive to hiring managers who think they are getting a well-rounded engineer at a recent-graduate salary. In addition, it sounds impressive in casual conversations with peers and professionals alike. 

Unfortunately, breadth often comes at the expense of depth. Mastering depth of understanding fundamentals, theories and concepts that are the foundation of modern technologies takes time, effort and dedication to learning. It also takes a lot of effort to learn to work in technologies that give developer more direct access to underlying systems, and data structures. It takes passion and drive to gain an understanding of how to use any technology well, and to set high standards for one's work.

Those with deeper understanding, and the curiosity to gain that understanding, go on to become better software engineers and architects. While not immediately required by a job rec, knowledge of the underlying concepts is regularly needed to solve tough technical challenges in real life. Familiarity with more difficult concepts and technologies makes it easier to learn the popular and simpler ideas that change many times throughout one's career. It is well-established that ability to learn, to think critically and creatively is trained by pushing the mind out of its comfort zone, studying hard, understanding rather than memorizing.

Popular technologies and skills that are “hot” in the jobs market are designed to be easy to acquire, and to solve a narrow set of common business problems, with the trade off that these technologies may not be well-suited to a wider range of applications. These skills are easy to learn and apply. The difference comes in the speed of learning, the quality of outcomes, and the critical thinking that better engineers bring to the job, together with the technical skills. In a competitive marketplace, those who went through the trouble of gaining the deeper knowledge of computer science fundamentals, software construction, and code craftsmanship will fare better.

Two are better than one

“If you can't explain it to a six year old, 

you don't understand it yourself.”

― Albert Einstein

Pair programming is uncomfortable, exhausting, and un-ergonomic. With pairing it takes two or more people to do a job of one. It is time-consuming to organize, tiring to participate in, and painstakingly slow to observe. The benefits are non-obvious.

Code reviews are nerve-wrecking, aggravating, and painful. They take enormous amounts of time, bring out the worst attitudes, and spur religious wars on code minutiae. Results are often inconsistent, suggestions are petty, and resulting improvement is invisible.

Yet, there is plenty of evidence that teams that manage through the pain and find the time to pair, to review, to swarm, to mob-program, and to work together using any other technique, do significantly better. Pairing and code reviews contribute greatly to the value being delivered in higher product quality, shorter time to release, and in lower maintenance costs.

It comes down to a simple truth, noted a long time ago:

“Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.” - Brian W. Kernighan and P. J. Plauger

 Left to our own devices, we want to be clever. But if we have to explain every turn of logic, every leap of faith, and every tiny assumption to another human, the desire for cleverness retires to the back of our minds, and leaves the center stage for the desire to be understood. This difference alone causes a big improvement in the work that we produce. Readable code has a much better chance of being good code, than code that’s hard to read.

It also means that the person reviewing the code, or being the non-writer in the pair does not have to be a developer, she could be anyone familiar with the product.  Even if that person does not contribute directly to the code, their questions will influence the programmer to write better software.

In the case of two or more developers working together, in addition to producing better code, all involved will be intensely learning from each other. And that’s the best thing that could possibly happen to developers.

[It will also make them tired.  I suggest getting a lot of colorful plastic balls, and investing in the fanciest espresso machine your office can afford.]

More Agile smells

"Agile" means many things - there are Agile processes, Agile values, Agile ways of working together as a team.  Becoming truly Agile requires a lot of education, willingness to keep an open mind, and lots of trust within the organization. 

When things are less than perfect, as they usually are, many teams get to experience "an Agile smell". Agile smell, similarly to code smell, is not necessarily a problem by itself, but rather a likely indicator of a serious problem with the team’s implementation of Agile.  

Learn to identify Agile smells, and the problems that may lead to them. Then get busy fixing the problems.

Smells "Our KPIs measure the wrong thing"

• Team talks about velocity points, rather than discuss stories and value created
• Members worry about being busy and putting in hours, rather than about producing valuable results
• The team strives to meet arbitrary deadlines, rather than focus on delivering quality working software

It is tempting to measure the team's success by velocity points archived, hours worked, or hitting deadlines. But velocity points are imaginary and unequal units, that differ from team to team, and even for the same team over time. They are too easy to game. Same with deadlines - deadlines are derived from imperfect estimates, and hitting those deadlines causes loss of quality and team spirit. Hours worked is straight-forward to measure, but has little correlation with value delivered. 

Smells "We are not using our tools well"

• The information in the tool is treated as the source of higher truth, rather than a starting point for a conversation
• Fill out the fields by the letter of the rules, rather than store useful information in an easy-to-understand format
• Use the tool insofar as the process requires it, but are uncurious about better ways a tool can help us improve

Tools are there to serve the team. They help to compile information over time, store details, and offer a choice of presentation layouts. Tools are not there to drive the team's value, do the work of coordinating and facilitating communication, resolve issues. It is OK to tweak the rules imposed over a tool usage, if it helps the team to work smarter. It is not OK to be a servant to the tool, rather than its master. 

Smells "We are not being awesome"

• Compile excuses why we are not doing a good job, rather than look for a better way to work
• Look for ways to make Agile process entertaining, rather than focus on the excitement of delivering great software
• Build Chinese walls between specialties, Agile team members and their responsibilities, rather than work together 

IT and software development is complex and complicated work. Every team that works hard and strives for success is going to make mistakes sometimes - and that's OK. In fact, it is good, because the only way to not make mistakes is to play it safe, aspire to little, and settle for less.

Problems do happen, and it is important to admit and examine what doesn't go well, take responsibility. So that the team can improve, do better in the future, and have fun while doing it.  

This is a second post in the series on Agile smells.  Read the first part here. 

Empowering Agile teams: culture clash

"The test of a first-rate intelligence is the ability 

to hold two opposing ideas in mind at the same

 time and still retain the ability to function." 

- F. Scott Fitzgerald

Agile frameworks empathize empowered teams, flat structures, and meritocracy. Team members and those in supporting roles are expected to earn trust and respect, rather than get authority based on org chart.

Organizations in general do not work this way. A modern organization is all about hierarchy, where authority comes from one’s position and place on the chart. Responsibility and decisions each travel in exactly one direction in the hierarchy: decisions come down, responsibility goes up.  This kind of structure is easy to manage from the top. But it does not lead to best decisions.

An Agile team is encouraged to take control of, and responsibility, their own work, but it comes in direct conflict with the not-so-Agile organizational structure. Entire corporate cultures are built around deferring to someone with a higher title. Those cultures do not disappear when an authority figure up the hierarchy decides that some projects should try Agile.

People on an Agile team are taught to care about producing value, make their own decisions, and take responsibility – but for employees with years of experience in traditional organizations it is not easy to accept this kind of message at face value.  It is even harder if Agile has a narrow mandate, for example, when transformation is happening only in a small part of organization.

This cultural shift is the hardest and the most painful part of a large-company Agile transformation.  It is also the most valuable change a business can make to improve its culture.

Everybody wants a pony

In a recent conversation, a colleague mentioned that a project his team worked on was considered a failure. That surprised me. I heard that this team delivered a high-quality product; the software quickly went into production, and has been delighting users for some time now. 

  

Turns out, when the team first got started on a project, it came with a specification document and a deadline. As the work proceeded, the team was learning more about the project. A number of features were added based on conversations with stakeholders and organizational knowledge, other features described in the specification document were cut. Technical complexity was uncovered, and eventually conquered.  The team iteratively delivered the system for stakeholder review, each time presenting more functionality, and at some point it was decided that the software is ready to go into production. 

Well, as it happened, for this particular project the production release date was after the original deadline set on the project, back when the project consisted only of a specification document and that deadline. Therefore, the deadline was declared missed, and the project was declared a failure. 

I find this mind-boggling.  Before anything was known about a project except that there is software to be built, there was this random date.  Now, that the world moved forward, the project came into existence as a working, helpful system that delights users and produces real, tangible value for the organization. Yet, somehow, that original date is still the deciding factor whether the project succeeded.  Conforming to this picked-before-we-knew-anything-important date is more important than whether the project was delivered at all, whether it useful, or valuable, or makes users happy and productive.

It's understandable that people and businesses want a pony – projects that are predictable enough to be able to forecast completion date, scope, and user satisfaction ahead of time.  In practice, most of software development work happens on complex projects, with a lot of unknowns, where information is uncovered gradually as more work is done.  There is a very little chance that any given project is a pony. More likely it is a zebra, with a variety of stripes, and lots of unpredictability.  

Estimates in the real world

Estimates sound like a simple, beautiful, thing no project should ever be without. As always, if something is too good to be true, it is not true. Having a detailed estimate up-front for something highly uncertain is desirable, and impossible.

Estimates are hard to use properly, and very hard to get value from. There are people who understand probabilities, have a good grasp of variability and resulting deviations. These people are able to work with, and to benefit from, early estimates while considering all the uncertainty of the situation. These people also do not exist in IT project-planning roles. 

Estimates look like a wonderful thing to have, but they are hardly necessary. Not having a good estimate may be less than ideal, but never disastrous. Estimates do not change the flow of work to the better, although they could encourage cutting corners.  Many projects have been successfully completed without any estimates, or with bad ones. 

Overall, the most desirable estimates are impossible to get right, all estimates are easy to misuse, and even the best estimates provide limited value even if everything goes well. Estimates are normally discussed in terms of a best-case scenario, which is very different from what usually happens.

• Best case scenario
• On-point estimate. All necessary resources available when needed. No changes in requirements.
• On-time and on-budget delivery.
• What usually happens
• Best-case scenario estimate, further shortened because of business need.  All necessary resources are not available until much later, except a few, that are not available at all. Massive and conflicting changes in requirements. 
• Complete chaos when estimated delivery date arrives.

Estimates are a nice thing to have in a perfect world.  Having good estimates allows to predict far ahead of time when and what resources will be needed, when and what milestones should be completed, and when to expect the final delivery. However, in a perfect world, resources are available quickly, milestones are hit reliably, and delivery has perfect timing - whether there were estimates or not. 

Real world is far from perfect. In the real world, more often than not, estimates are way off, inputs are late, milestones keep changing, and final delivery turns out to be a first draft.  The only piece that remains set in stone is the original estimate, while the rest of the situation changes dramatically. Rough guesses become deadlines, do-or-die commitments, and the cornerstone of blame-passing culture. 

For those projects that attempt to succeed in the real,  rather than perfect, world, consider doing away with the estimates-come-deadlines.

For hard projects, choose Agile

What’s the cheapest way to build software?  

Agile approach is expensive.  There are meetings that take valuable engineering time. There are non-engineering roles, i.e. people that need to be paid, that are not writing code and, therefore, whose contribution to the product comes into question.  There is a significant body of test code, and which is obviously not helpful to the end user. There is constant refactoring work as the team discovers better architecture for the emerging product.  

Waterfall is cheaper. Only hire an architect to for the initial phase of designing the system.  Pay the technical team just to write the code for user functionality, and to do it quickly.  The engineers can be cheaper, too, since all the thinking has already been done in the earlier design phase.  Let the engineers go once all the code is written. Next QA comes in, runs tests, and violá, the product is done.  No meetings ever needed, since all the people involved are working off the earlier phase’ documentation.

Why pay more?

Agile process has no single point of failure. Fault-tolerance of the process is extremely high, error detection and recovery is “baked in”. Waterfall, in contrast, requires perfect execution at every point in the process, i.e. every point is a single point of failure. Errors are hard to detect until the very end, and even harder to fix. As projects get more complex and require more investment, it makes sense to place a higher value on the probability of success, rather than on minimizing initial cost. 

"Slowly and carefully"

I once asked a coworker with an official title of Senior Software Architect, how he makes decisions and gives recommendations on the architecture of our product. “Slowly and carefully”, he replied. He was an honest and hard-working person, a very smart guy, and probably a decent architect. Yet our product’s architecture was terrible, and it was not improving as time went by.

Two deadly sins of software architecture were ubiquitous in our code base

• Everything was tightly coupled with everything else
• Same information was defined in multiple places

In day-to-day work these problems caused lots of hard-to-trace bugs, showing up in seemingly unrelated portions of the code. Changes required touching lots of code, leading to more bugs. And it was very hard to write tests – so tests were not written.

So, what was my colleague doing wrong? The architectural changes he and the team inflicted on the code base were fine - mostly because very little changes were made. There was no opportunity for a serious refactoring effort to improve on the underlying structure. The bad stuff that accumulated over time continued to accumulate. The new features represented good ideas, but were poorly implemented – or, rather, were implemented in much the same way as old ones, creating more bad code.

Looking back, there was a culture problem with the approach of “slowly and carefully, with no objective criteria, teamwork, or testing”. The architect did not write code. He provided suggestions that were implemented by other people. Programmers wrote code per instructions, and did not create tests to fully exercise the added or modified code. The architect did not have the feedback on how well his ideas fit into the code base, and programmers had very vague feedback on how their changes affected the system.

Most of the time, the architect's solutions were fine.  He was familiar with the code, has been on the project for a long time, and knew the team well. But since his suggestions were hardly ever challenged, and never tested against competitive ideas, it is impossible to tell whether what he offered was really good - or barely good enough.

Every so often, a developer would complain that the solution suggested by the architect did not work, and it became a point of pride for others to write some “clever” code to fit the square peg (architect’s suggestion) into a round hole (existing code). It is well established that the “clever” code is more bug-prone and harder to maintain, compared to straight-forward and easily readable code.

Sometimes, no amount of cleverness could fit a new elegant idea into the clunky existing code, and then people improvised on their own. Most of the time, those were good, solid solutions, but not always. There also was an illicit feel to the situation, and some developers tried to avoid code reviews in these cases. There were a few scary monstrosities quietly sneaked into the code base as a result.

Splitting the implementation and design work prevents everybody from learning from their successes and mistakes. Having an architect dictate unchallenged solutions creates sub-par performance, tension and funny feelings even on the best-intentioned team.

Magic-8 ball, and other approaches to problem-solving

There is a complicated way to do complicated things. Think long and hard. Spend hours staring into the blank space, screen or a sheet of paper or a list of specifications, or what-have-you. Then spend more hours furiously typing or writing or arguing out loud, while pacing nervously. Grab heavy, plain-jacket books with no pictures and very small font, throw them around. Doodle, draw up UML charts, and write down dozens of numbers… Do all of the above in freshly dry-cleaned business attire, or PJs, or worn-out jean shorts, or nothing at all.  Repeat until half hour before the deadline. Feel very excited about the results. 

This is pure magic. It is shrouded in mystery and emotion, takes magical amounts of time, and delivers unpredictable results with unknowable certainty. When it succeeds, it triumphs heroically, and when it fails, it does so with a big loud bang.  

There is also an easier way. Make up something, anything, however wrong it may be. Devise a test, and perform it. Go back to the drawing board, the text editor, the spreadsheet, etc. Notice the problems in the previous idea and implementation. Then try again, taking the new information into account. Update the test as well. Continue until happy with the results. Heavy books, typing furiously, and worn-out jeans may still come in handy, but this way those are means to an end, not a pure expression of experiencing a difficult and unfair world.  

When a complicated problem is solved by taking a small step at a time and adjusting, there emerges a combination of two possible solution paths. One path is as expected: iterating over small steps leads to a locally optimal solution. The achieved solution is local to wherever the initial, the least educated and therefore the most random, guess happened to be. In a lot of situations, this is good enough: achieved solution is proven to fit the problem by a series of tests, it is achieved in a reliable and predictable way, and no magic is required.

The other path is less expected, but nevertheless fairly common. Making small steps in the possible solution space allows to build an intuition about the underlying patterns and relationships. In addition to a set of concrete, tried-and-tested solution points, there emerges a deeper understanding of the problem. This leads to an intuition that allows to escape the local maximum, and helps reason about the entire solution space. Furthermore, a test-driven approach gives additional freedom to try out and gain confidence about further-fetched ideas in a systematic way.   

The iterative approach is simple, rational, and pragmatic. It succeeds routinely and quietly, and fails rarely and cheaply. Magic may still happen, but it will be more grounded and less dramatic.  More importantly, magic is not a requirement for success. 

How magical do you want your problem-solving to be? 

Monkey see, monkey do

"Wikipedia is not considered a credible source." 

-  from Wikipedia

A programmer, computer programmer, developer, or software engineer is a person who writes computer software. [Wikipedia, "Programmer"]

Computer programming (or programming) is a process that leads from an original formulation of a computing problem to executable programs. … The purpose of programming is to find a sequence of instructions that will automate performing a specific task or solve a problem. [Wikipedia, "Computer Programming"]

Software engineering is the application of a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of software.  It involves the establishment and use of sound engineering principles in order to economically obtain software that is reliable and works efficiently on real machines.[Wikipedia, "Software Engineering"]

Engineering is the application of scientific, economic, social, and practical knowledge in order to design, build, maintain, and improve structures, machines, devices, systems, materials and processes.  Origin of term: from Latin ingenium, meaning "cleverness" and ingeniare, meaning "to contrive, devise". [Wikipedia, "Engineering"]

Code monkey means an unskilled computer programmer who can only perform programming tasks by copying examples, usually where they do not truly understand how the code works. ‘Code monkey’ is also a job description for a job that treats even experienced programmers in a way that trivializes their problem-solving abilities.[Wikipedia, "Code Monkey"]

Agile smells

“In computer programming, code smell is any symptom in the source code of a program that possibly indicates a deeper problem.” – Wikipedia

Agile way of working provides for a lot of flexibility.  Agile values and frameworks come with significant freedom for all participants, letting teams and stake-holders design their own implementations of Agile. And with the freedoms come the risks, and the smells.

Agile smell, similarly to code smell, is not necessarily a problem by itself, but rather a likely indicator of a serious problem with the team’s way of work.  Smells can be caused by following the letter, but not the spirit, of the Agile framework, gaming or cheating the system, or simply lack of understanding the reasons behind the framework rules.

Does your team’s Agile implementation smell?  

Smells “We do not really work together” 

• Team members are routinely late to or even skip daily Scrum. Those who come to Scrum deliver short statuses: “Everything is good. My work is on track.”
• Immediately after the daily Scrum meeting is over, team members retreat to their private spaces to work on their tasks. Or put on headphones. Or display in some other way that talking is over, and "real" work has began.
• Scrum board is out of date. Online tracking tool is hardly used for anything, but reporting.

Daily Scrum, visual board and knowing how the Spring work for the entire team is progressing are important to facilitate team members working together to get work done. When a Scrum meeting or a Scrum board do not bring forward interesting and relevant information about the minutiae of team’s work, this is an Agile smell of the team not working together.  

 Smells “Agile is something we do when we are not too busy with real work”

• Agile framework meetings are regularly rescheduled or cancelled to make time and space for technical and product meetings.
• There are permanent roles on the team, that are not Agile roles.
• During the retrospective the team reflects on the iteration behind, but hardly anything happens based on that discussion.

Agile framework may look like a game, but every role, ceremony and artifact are there for a reason.  Doing just bits and pieces that fit into the schedule does not make a team Agile.  It just makes people busy.  

Smells “We are not as self-organizing and empowered as Agile tells us to be”

• The conversation in meetings revolves around the manager or Scrum Master, with each team member talking to that person, rather than the entire team.
• The work is prepared, researched and even assigned by the manager or one senior team member.  That person coordinates all interactions with other teams. 
• All or most communication between team members is facilitated by the manager.

Agile approach is built around a strong, motivated, powerful team with a servant leader working to serve the team.  Many groups struggle with becoming that team, especially when coming from the traditional model where the manager is above and in charge of the team. These smells are an indicator that the team is operating in the command-and-control structure, rather than following Agile principles.  

Rules of software optimization

“Rules of Optimization:

Rule 1: Don't do it.

Rule 2 (for experts only): Don't do it yet. ” 

- Michael A. Jackson

The presenter talked about processing TBs of data in less and less time, even as models get more complicated. Slides covered different communication options, kinds of multithreading, benchmarks measuring time and memory foot-print for massively parallel processing. Chip capabilities and layout, high-speed interconnects, highly-specialized libraries and compiler options to take the best advantage of the available hardware.

Dealing with complex models on large data sets available all at the same time is a serious and very interesting problem.  Modern computers are immensely powerful, with a typical laptop packing multiple cores, popular languages and compilers doing a great job of optimization, and many high-performance tools widely available for learning and using. Yet most of existing and in-progress software only uses a tiny fraction of available horsepower, and the vast majority of developers are not aware of the vast amount of work their systems are capable of handling, with the right setup.

This is a fine state of things.

Optimizing performance comes at a cost. This cost can be time – it takes time to assess performance, find bottlenecks, and do something about them. This cost can be money – it takes a better-educated developer to measure and reason about performance, and what it should be. Most often, this cost also comes in correctness – optimizing code is tricky, and bugs can and will be introduced while chasing speed. The hard-core performance optimization, to take the best advantage of the available hardware and software, can make code longer and more complex, less readable and maintainable.

However, there is an even bigger reason why getting the best performance is only relevant in a narrow set of applications. A typical business application today is interactive, works with live user input, and produces output for human consumption in real time. Once output is produced fast-enough, there is no value in producing it any faster.  If showing a document on the screen takes 200msec, doing it any faster has no value.  

It is best to leave the performance optimization for the applications and scenarios that will deliver better value because of the speed. For the rest of modern software development, it is better to concentrate on correctness, maintainability, and making code as simple as possible.

What does it mean to do a good job?

I am lucky to know many hard-working, dedicated, well-meaning people who like their jobs, greatly enjoy what they do, and care a great deal to do good work.  Yet, the idea of different people of what it means to do good work can vary tremendously.   

There is a career-driven definition of doing good work. Good work is work that gets the person to the next step of their desired career path. It usually translates into doing what the boss wants to get done, exactly the way the boss wants it done. Occasionally it means doing the things that get noticed by the decision makers, and little or nothing else.

Particular definition of ‘doing good work’ in this case can vary slowly over time, as bosses, their opinions, and the person’s understanding of bosses’ opinions change. 

Another approach to defining what it means to do a good job is a craftsman’s definition. Technical quality of individual’s work is measured against expectations set by an independently defined standard, unconnected to the organization. 

In this scenario, what it means to do good work can vary quickly or slowly, depending on the chosen base standard and its rate of change. Unfortunately, the changes in the standard can, and is likely to, inspire changes that are unconnected to the practical needs.

Finally, there is a value-driven definition to doing a good job. Is the work being done gets the organization the most benefit toward its vision and goals? Then it’s excellent work. The trouble with attempting to do good work by this definition is that organization's true vision and goals are often not known to many workers. Honest and comprehensive communication of the organization's vision and goals are very hard, and rare in large organizations.

Good job under value-driven definition can and does change, as goals change, or perhaps goal perception changes. It can be very frustrating, though, when goals, or available information about goals, change frequently and in contradicting ways.

What is your idea of what it means to be 'doing good work'?  Is it different from your bosses’, or your organizations? Have you changed your understanding of what it means to do a good job in your career? 

No fighting in the war room

“I will stand up against development to fight for software quality!” – said a perspective QA manager. She was smiling, but in way that left no doubt – she meant every word, and she was good at fighting and winning against bad things and bad people.

As a developer, I liked the candidate’s strong stand on quality, but was not sure why we should be fighting. The QA was a part of Dev organization, and a new QA manager was going to report to Dev manager. After the interview, we discussed how our Dev team works with our QA organization, cooperating and helping each other. We all had a common goal of producing high-quality product, and fighting was nobody’s idea of fun, nor was it helpful in achieving this goal.  

As it turned out, our team was both lucky and smart, creating and maintaining a helping and cooperative spirit in the office. As we learnt through talking to this and other QA manager candidates, there were plenty of companies where testers fought against developers. QA groups won by finding enough quality issues with the product to prevent delivery.  Developers won by getting the product delivered, despite known problems.  A strong development group put a QA team at a disadvantage, while a thorough testing created problems for the developers. Which makes no sense, because having great people and organizations in both Dev and QA should make everyone's lives better, not worse.

As satisfying as winning is, it is important to keep in mind the overall goals of the organization. In IT and software development, it is usually creating and delivering a quality product for the customer.  Nobody wins if the product is not tested properly, is of poor quality, nor if the product is not released to the user. Everybody wins if QA and Development (and the rest of the organization) work together.

Introverts: finding comfort in Agile

Agile transformations seem to require everyone to be an extrovert: isolated cubicle dwellings are quickly converted into open spaces, there are so many meetings to attend, with many opportunities and plenty of encouragement to speak up. The introverts, a large share of professionals in IT and software development, become increasingly uncomfortable as we learn how much working together, sharing space, and group discussion appears to be required by the new system.

Introverts need not worry. Agile way of doing business brings many changes, but it is not about making everyone suddenly become extremely outgoing. The Agile transformation itself, a relatively short time period of intensive education, organizational restructuring, and, perhaps, moving furniture, can be stressful and tiring for all. It can be especially tough on introverts, who prefer stable environment and few interruptions. However, once the dust settles, and the organization is firmly on a path to agility, both introverts and extraverts are able to find a comfortable spot within new frameworks.

Agile frameworks are built around working in small teams toward a common goal, building trust over time, and developing skills to help each other. The team members are encouraged to get to know each other, and teams are expected to stay together for a long time. An agile team is a perfect environment for introverts – a small group of colleagues, with common interests and motivation, in a stable environment.

A big part of Agile philosophy is about having a lot of communication between all people involved in a project. Communication in the corporate environment is often understood as large, long-running and highly ceremonial meetings, delivering official reports, addressing large groups of people – all the activities that make introverts want to run and hide. But most communication required by an Agile framework happens in small groups (as small as 2-3 people), and is built of casual conversations. It is also concentrated within the small team working together for a significant amount of time, where people had had an ample chance to build relationships, get to know each other well, and build trust. This is the kind of communication that introverts handle well, and often enjoy.

The toughest part of Agile setup for introverts is the office environment. Open space destroys any chance of privacy and time alone, while encouraging social chatter and frequent interruptions. However, one of the more important tenets of Agile philosophy is the concept of self-organizing team, that is, a team that it is in charge of its work and environment. Teams are allowed and encouraged to take control of their office furniture, in addition to everything else. Mature Agile teams with introverts get to re-arrange the office environment the way it works best for all the people on the team, and create opportunities for working alone or in pairs. Quiet spaces, no-interruptions-allowed zones, those traditional introvert heavens, reduce stress and boost productivity for both introverts and extraverts, as long as there is also a space for collaboration and communication.

Rolls Royce or Ford Model T?

This question applies to every successful idea.  Some ideas only make sense for a limited audience of Olympic-level talent, and others scale beautifully for a large population.  For example, the upscale resort at pristine Musha Cay, a tiny tropical island near Bahamas, accommodates no more than 24 guests, while Disney World hosts an average of over 52,000,000 visitors per year.

The philosophy of Agile Software Development started as a bright idea of a few very talented, very engaged developers who wanted to deliver highest-quality software while providing better value to their customers.  And so they did, with very powerful results.  They also shared their idea of Agile with others.

But is Agile a Musha Cay kind of idea, or is it more like a Disney World?  Will Agile Software Development benefit just the brightest, the most talented, hard-working and engaged, or will it help every average Mort and Elvis and their clients to enjoy better success?

In its early days, Agile was invented and embraced primarily by a small group of developers, the best and the brightest, and the most engaged. Through their painstaking work, Agile slowly found (or not) its way through the traditional organizations and processes, one conversation at a time. 

Now Agile arrives from the top, carefully tailored for the management approval, and is swiftly installed in IT departments, whether the entire cadre of employees want it or not, in one big organizational shake-up.  Is it the same Agile that worked so beautifully for the most talented few who embraced it, fought for it, and followed it with their whole hearts? Probably not.  But is it better than the alternative, that is, no Agile? 

Agile Manifesto: working software over other things

Manifesto for Agile Software Development second statement is about valuing working software over documentation and over things.

This is so simple and logical, it may even seem obvious: software development should be about software that works.   However, this is a new idea offered by Agile movement, and not the default choice in the contract-driven Waterfall environment.

The business of software development involves many different tasks, many of which are not as complex and unpredictable than building working software. Creating documentation is a common example of such a task.  It is tempting to deliver on those tasks and to declare victory based on easier-to-achieve results. It takes a lot more discipline to execute on the more complicated, core portion of the work required to create software - and deliver no less than the working software.

Delivering working software is raison d'être for the industry.  It is fundamental that working software is created and delivered, and it should be the focus of the industry - not documentation, nor other more or less significant artifacts.

Long-term teams

It takes time and effort for a team to learn to work together. Agile teams improve dramatically over the first 6-9 months of working together. Not only teams become more productive, they also get better at estimating, and learn deliver at a steadier pace. Team members learn to communicate well, acquire the cross-functional skills needed to reduce bottlenecks, become good at brainstorming and conflict resolution, form mentoring and supportive relationships that help maintain a great team atmosphere. Within a year of working together, many teams are rolling through the project smoothly, at a steady pace, and enjoying their success.

Yet, it is somewhat rare for the team to stay together for longer than half a year to a year.  There is churn of projects, people come and go, successful teams get broken up to “seed” or improve other teams, that may not be doing particularly well. A great value of a well-gelled team seems to be unaccounted for, and regularly lost, without the organization even noticing.

Agile frameworks do not promote any meaningful way to value teams.  It is hard to talk about the value of a well-gelled, long-term team, and compare it to the value of adding a person or two to some other teams. But it is a conversation worth having.

Early bird misses the bug

For teams doing Agile, there is often a push to get ready for implementing a story early – long before the story is scheduled for implementation. The prep work is often way beyond getting the basic understanding of what the story is about, and sharing the vision for the result. It involves delving deep into the technical details and complexities, code minutiae, architectural and environmental setup.  By gathering that intelligence early, the team tries to minimize the risk of uncovering additional complexity of the story and missing estimates when this story is finally scheduled into the iteration.

The intentions are good – having more details early seems more prudent, being knowledgeable about the story beforehand gives the team more confidence in providing estimates, and PO feels that she is doing a good job communicating with the team about future stories.

However, there are problems with pre-preparing stories ahead of time.   

For one, the team is pushed to work on stories that are not in the current iteration.  This work has no definition of done, no acceptance criteria, and pulls the team away from working toward current iteration goal.

A bigger problem is that once all this work is done, the story is about as likely to go into an iteration soon, as not. The intelligence gathered may get written up or not, but even if it is, this knowledge is still very perishable. If the pre-worked story sits on the backlog even for a couple of iterations, much of that intelligence deteriorates beyond any usefulness.

The biggest problem with pre-work, though, is the pressure it creates. Pressure to create exact estimates, since all this work has been done on the story beforehand.  Pressure to not ask questions later on, when the story is actually being worked, since the onus is on the team to ask questions early.  Pressure to burn the midnight oil, if the story turned out to be more complicated than expected, despite all that time spent researching it before.    

One of the rules in eXtreme Programming is “Keep the system uncluttered with extra stuff you guess will be used later”.  This is also a good rule to plan work by.  Do not pre-work the stories, let the team ask the questions and use the answers immediately, as they are working the stories in the iteration. It is OK to take the risk that sometimes some stories will turn out to be more complex than others, and will need to be continued in the next iteration.  Overall productivity will be higher, because the team will spend less time on work that is not moving the team closer to the iteration goal.  Overall quality will be higher, because the team will have the latest information available, obtained at the last responsible moment, and will not have re-research outdated intelligence.

Distributed teams: making the effort

Distributed teams have to make additional effort, compared to collocated teams, in communication and organization.   That much is clear – distributed teams can perform as good or better than collocated teams, but only if the people rise to the challenge of creating and maintaining continuous and clear communication within the team.

Many teams are distributed only part-time, or consist of several collocated groups, or even almost entirely collocated with an exception of a lone remote member.   When one team member decides to work from home one morning a week – that makes the team a distributed team.  When a person who joins a team is located on another floor, or in a different state – that makes the team a distributed team.

And in a distributed team, in order for the team to gel, everyone must put in the effort to keep the team in sync and communicating – not just the person who is temporarily out of the team room.  This is a tall order, especially since very often the decision to make the team distributed is made inadvertently, by the management rather than the team, and without full consideration of what it will take to run a successful distributed team.

When a team becomes “slightly distributed” for the first time, the pressure to “make it work” is usually on the person who is remote.   But in order to succeed, all of the team members need to pull together and figure out new ways to communicate, to work together, to build bridges and have water cooler-type conversations for the entire team, including the team members not physically present.