Why Agile

Agile, and it’s most popular implementation Scrum, is becoming standard in software development. Yet, each organization’s journey to Agile adoption is different, and is not always smooth. Scrum adoption is often a significant change in how the organization functions, a change that requires people to assume new roles, acquire new competencies, develop new relationships.

So, why Agile? Changing the organization is hard work, risky, and painful.  What does Scrum deliver to the organization?

There are a lot of great ways how Agile and Scrum can make a software development organization more productive and successful.  Lets look at some of them.  

• Scrum can help restructure and reduce delivery risk and increase predictability: potentially shippable software in small chunks, delivered frequently. The most important structural element of Scrum methodology requires delivering working software every 2-3-week iteration. The product must be ready to ship or release. Continuous focus on upcoming delivery forces the organization to focus on finishing up work quickly, and allows to deliver often and on-time.
  
• Scrum helps deliver a better product, by encouraging earlier incremental delivery, and regular and frequent communication with customers and stakeholders during development. Frequent incremental deliveries allow to collect feedback on the incrementally delivered, yet already useful, software. Short iteration cycle allows the team to accommodate desired changes as requests come in during the development cycle.
  
• Agile and Scrum recognize and encourage better software engineering practices, to improve and maintain higher technical quality of the product. Higher software quality allows for fewer bugs, shorter time and less effort adding new functionality and fixing defects, and makes it easier for new developers to become productive working on the product. Commonly suggested practices are automated builds, frequent or continuous integration, automated developer and acceptance testing, requiring and maintaining code quality using automated code analysis tools and code reviews.
  
• Scrum helps organizations achieve better ROI by delivering better product earlier, and working incrementally. The customers get a chance to start working with the incrementally delivered product earlier. The organization enjoys an even-paced delivery schedule that builds trust with stakeholders, improves morale, and limits pressure and stress.
  
• Agile approach to software development is built around the understanding that creating software is an inherently complex and complicated business, with tremendous upside, expensive downside, and high uncertainty. Scrum and other Agile frameworks do a great job to allow for experimentation by limiting the impact of mistakes. In order to succeed in a creative enterprise, errors should be allowed and encouraged, and it helps if the mistakes are cheap and easy to identify and fix. Agile encourages communication and experimentation, makes mistakes cheap and visible, and thus creates perfect condition for hitting the ultimate project jack-pot.

Have you considered adopting Agile? What are your reasons for the change?

Dealing with the technical debt

Been chewing on the idea suggested by @nycplayer last night that messy code is not technical debt...it's just the interest.

— Sarah Mei (@sarahmei) November 19, 2015

That's an interesting way to think about technical debt. The amount of interest on a debt is controlled by the interest rate, and also compounding frequency.  Higher rate leads to more interest accumulating over time, and so does higher compounding frequency. Compounding frequency for the technical debt interest rate is determined by amount of work that still needs to be done on the code.

For any level of technical debt, i.e. code mess, the cost of technical debt could potentially be nothing - if that code never gets touched again.

However, if the codebase is for a useful, living product, chances are it is continuously going through the transformation. Features are being added and removed, bugs are being worked out or at least mitigated, and structure gets updated. That raises the cost of the technical debt in that codebase tremendously, since the interest is being compounded all the time.

When starting to work on reducing the technical debt in a large codebase, the starting point makes a difference. Starting with the code that changes frequently is a high risk/high return approach: improving technical quality will make a noticeable difference in the technical debt burden. However, it is risky to introduce changes to the code that is exercised a lot and in ways that change frequently.

Alternatively, one can choose to start with less-frequently touched portions of the codebase, to build experience and understanding.  The technical debt reduction benefits will be less significant, but the code changes will also be less risky.

Judging Scrum

Scrum is supposed to be simple. Scrum.org by Scrum creators Jeff Sutherland and Ken Ken Schwaber attempts to define Scrum less than 60 words:

“Scrum is a management and control process that cuts through complexity to focus on building software that meets business needs. Management and teams are able to get their hands around the requirements and technologies, never let go, and deliver working software, incrementally and empirically.
Scrum itself is a simple framework for effective team collaboration on complex software projects. ”

How simple is Scrum, really?

Scrum, as a management framework, is a practical approach to solving the Agency problem. Agency problems are pervasive in modern economies due to the extensive division of labour and specialization.[1]

The problem of agency was first laid out in 1970s[2], noticing the essential imperfection of agency relationships: people (agents) hired to tend to interests of others (principals) actually behave in their own best interests. Economic and institutional approaches to principal-agency problem look at incentive and punishment schemes and various management issues that arise when trying to bring agents’ behavior closer to what the principals would prefer.

Organizations that engage in software development and IT projects almost always involve a group of people concerned with the final product (principals), and a different group of people (agents) tasked with creating that product. The problem of agency is fairly easy to see in this setting: the interests of the business are not identical to interests of the creators of the product. There are different ideas about what the principals group wants and expects the creators to do, but all agree that interests and behavior of the agents group are never perfectly aligned with principals’ group desires.

Scrum is one of many systems designed to address the agency problem in the commercial software development and IT space. History and current state of management science and business practices show that the problem of agency continues to lack a perfect solution. Among many systems designed to align behavior of principals and agents in a given business situation, most only achieve a partial solution, and some are unreasonably costly. Like other systems, Scrum may be helpful in achieving a better alignment between principals and agents. Scrum can also turn out to be costly, depending on implementation.

Scrum targets to improve communication between the two groups, with the idea that better communication will allow for greater transparency and trust. Scrum provides fairly detailed rules for communication structure for the two groups, and instructions for both the principals and the agents how to set and sustain this structure. This generally works, although trust is not a guarantee. However, communication is costly, and is subject to diminishing returns, i.e. more is not always better.

In addition to the prescribed communication structure, Scrum dictates limited autonomy and accountability for the agents group, and institutes measures to maintain its own framework. These changes are also costly, and run contrary to common organizational structures, thus making them complex to implement. Even worse, the benefits, although sometimes great, may not be easy to quantify, and are very hard to measure.

While the rules of Scrum appear to be simplistic, the overall system is quite complex, with ambitious goals and significant overhead. Scrum affects operation of both principals and agents, steering emotions on both sides. It effectively introduces a brand-new principal-agent contract, with differently defined roles and responsibilities, and new assumptions about each groups’ motivation.

Overall, Scrum is an imperfect approach to a very hard problem – one of many imperfect approaches. There is no known solution as of yet, and it is quite possible that such a solution does not exist at all. When considering Scrum against other less-than-perfect attempts to align behavior between principals and agents, it appears that Scrum does a pretty good job delivering on its goals. Yet, it is far from simple.

Scrum master woes

The scrum master was genuinely upset:

-  Scrum is so hard. Engineers do not like it. They feel exposed.
-  Well, yes – Scrum calls for transparency, and that is a good thing. Engineers will learn to share.
-  But we have to do estimates. And then we have to deliver to those estimates. Then things happen. And engineers hate to do it, and then explain how a task that was estimated at 2 hours took three and a half.

It is easy to see why engineers do not like this process. Rapid churn of iterations calls for very frequent estimates – and regular inspections of why these estimates turned out to be not quite right. It is painful to be continuously wrong, and even more frustrating to have to defend tiny details of the complex minutiae that is software development.

What can the team, and the Scrum Master, do, to lessen the pain?

The scope of estimates is negotiated between the project manager or product owner, and the product team. Many a project manager would like correct, non-changing estimates provided on the first day of the project. Many a team would be perfectly happy to not estimate at all. Usually, neither gets exactly what she wants, but instead negotiates a compromise with the other side.

The scale of possible compromise solutions

• The team does not provide any estimates. Project manager, with the help of the Scrum Master, makes educated guesses, based on their experience with the team and the project, when the backlog items will be completed. In this case the team may not share how it is breaking down the work, the progress it is making during the iteration, nor the problems it encounters. Priority of backlog items is established based on their value, and only project managers’ perception of the size and complexity of each item, since the team may not share its evaluation.


• The team provides detailed estimates to project manager at each iteration, and works hard to deliver per estimated completion times. Project manager gets a full report of each working hour of the product team, and gets to control how the work is broken down and accomplished by the team. In this case the team may not be fully forthcoming about how long each tiny item will take, and pads the estimated schedule so that there is enough slack to deliver even if things go wrong. Even if everything goes fine, the team is more likely to deliver on a padded schedule and use the slack time to relax, rather than forge ahead. Even with seriously padded timeline, estimates are often too optimistic, in which case the team will be blamed for incorrect estimates, encouraging even more padding going forward. Detailed estimates take a lot of time and effort, and so does hiding the slack/padded time that the team puts into estimates. The team will be working harder, without a corresponding rise in productivity.


• The team provides coarse-grained relative estimates, by sharing with the project manager how hard each item of work is expected to be. The expected delivery timeline can be deduced, but is not promised, so that the project manager shares in the uncertainty of software development. Still, the project manager can use that information to adjust priorities of various backlog items. The product team does not have to be defensive about not delivering per estimates, because no particular timeline is being promised. The mutual expectation is that most of the estimates will turn out to be wrong, but when the estimates are averaged over multiple backlog items and several iterations, they will provide valuable, actionable information to the team, the project manager, and the rest of the organization.

What are coarse-grained estimates?

It is helpful to classify pieces of work on a scale of Trivial, Small, Large, etc.  Fewer options means coarser estimates.  Using numbers (prime, Fibonacci, etc.) may encourage defining too fine-grain a division.  Using hours or days creates a direct timeline, suggesting that a particular delivery date has been promised. Another option is to use anonymous units of complexity, and those are great if the team can in fact treat them as abstract units (and not convert units into timeline, as it often happens). 

Having less-than-perfect estimates, and no timeline, requires and helps build trust between the team and the project manager. The team is held responsible for the average productivity, rather than each particular estimate. The project manager is asked to give up the illusion of full control over the project timeline and trust the team to deliver as best they can.

Scrum Master is tasked with negotiating the compromise, and holding both sides responsible for their respective contributions.

Lack of women in technology industry starts early

As a part of the wonderful HoustonTechFest, an annual community event for the technology industry, we got to have a very important conversation on diversity in our communities. A very small portion of software engineers are women. It is rather rare for women to enter, thrive, and build successful long-term careers in the technology industry.

During our panel discussion at the HoustonTechFest “BuildingSoftware Is A Team Sport: Women In Technology” we discussed the benefits of having both men and women participate and be successful in technology field. Participants talked about the problems for different ages and career stages, and what we as a community can do about it. We also looked at the societal biases, rooted far outside the technology industry, that keep men and women apart and less than successful professionally, and how each one of us can help our friends and colleagues lessen the effects of these biases.

As a technology professional, I observe these biases nearly every day. I work with many smart, successful, and open to diversity and true meritocracy men and women. I also interact with people who believe that women cannot be capable engineers.

Recently, I’ve come across a scary source of these biases.  My toddler attends a wonderful daycare center – a loving, fun place where kids play, learn, and build their social skills through interacting with other kids and many excellent teachers. Yet, one day my child came home from daycare with this scary idea: “Girls are nice. Boys create messes. I do not like boys.” She went on to name girls who she likes, because they are girls, and boys who she does not like, since they are boys.

We have had several conversations as a family how boys and girls are both kids who play and learn together, how boys and girls can both be nice and fun. How it is OK to not like kids who are mean or angry, but it is not fair to not like or not play with a kid simply because he is a boy (or a girl, for that matter).  

However, this is a tough message to deliver to a little kid. Just as tough as it is to convince grown up, professional men and women with extensive work and life experience, that both men and women can be smart, capable, innovative, and contribute greatly to the success of the industry, if given a chance.

The struggle continues.  Please join us for our next discussion about successful diverse teams in technology.

Make your code reviews Agile

When discussing dealing with technical debt, fragile and buggy code base, and improving quality of the technical community, code reviews are a popular suggestion. Code reviews sound wonderful in theory: a junior or simply not-so-good programmer presents his work for review to an experienced grander-than-thou colleague, receives excellent suggestions within one hour, and another hour later all suggestions are implemented and the code transforms from nasty to near-perfect.

Enter reality. Thorough review of non-trivial code takes a long time. It is becoming increasingly harder to do a quality review as the time needed for the review increases. The pressure to provide a quality review suggests that more time is needed, and more time spent in review leads to poorer reviews, creating a downward spiral leading to deadlock.

Team dynamics are also affected. Giving a small number of reviewers the authority to dictate changes in work they are barely familiar with, without much feedback on the outcome of those changes, reinforces the bad habits of the reviewers and further entrenches their biases into the codebase. People required to accept suggestions without a possibility of challenge or feedback soon start to despise the reviews, the reviewers, and the suggestions. Neither reviewers nor authors feel responsible for the quality of the end results: reviewers because they only spent a little time reviewing, and authors because they were forced to make changes against their judgement.

So, are code reviews doomed to fail?

For code reviews to be a successful code-quality tool, consider an Agile approach. Make code reviews small and nimble, and have the entire team in charge of making, accepting and implementing generated suggestions.

That means everybody reviews everybody else’s code, in tiny little chunks. No formal meeting should be required. Everybody is invited and encouraged to generate improvement suggestions on all other team member’s work – and expected to defend their ideas. There is no separation of roles into those who critique, and those who are being critiqued. All team members are required to ask for and work with peer feedback.

Successful remote teams

In this day and age of total technological connectedness working from home seems like a no-brainer. There is email that we check every passing second, the smartphone that can reach us anywhere anytime, and increasingly sophisticated tools that allow sharing any kind of elegantly structured and totally chaotic data across every imaginable divide.

Yet, telecommuting remains rare, and remote projects remain hard. Outsourcing, in its more typical implementations, fails spectacularly and often. Organizations’ worst performers are often found among those “working from home”. (Best performers work from home, too, but that’s a different blog post.)

Overall, distributed groups face higher challenges and more risks on the way to becoming strong, coherent teams. Remote contributors must strive harder to successfully work together with others, compared to the regular folks who show up to the office every day.

For a team to be successful, members must share a vision, a flow, sense of connectedness that allows to think and make decisions together while maintaining unique perspectives.People on successful, productive teams interact many times a day to maintain working relationships. These interactions can be small or profound, like a quick ‘hello’ in a hallway or a heated discussion over lunch. More importantly, such exchanges are typically informal, unscheduled, and “just happen” – and they are absolutely essential to keeping the team in great shape.

Successful distributed teams have the technological means to have these interactions, too – but the opportunities do not ‘present themselves’. When other team members are not within eyesight, it takes conscious, deliberate effort to think pick up the phone, start up an IM conversation, or send an email. It is even harder to regularly take the time to share a joke, a story, something that’s not immediately essential for the work at hand, but serves a more long-term purpose of maintaining team spirit.

People who work remotely regularly and with great success, have made these kinds of intentional “extra” interactions part of their regular routine. The same way mature teams, organizations and their members produce high-quality results, mature distributed teams, organizations and their members make sure to stay connected and engage in lots of team interactions.

The maturity and effort required for a productive, successful remote working relationship may or may not be visible to a typical regular office worker Joe, who is having grand visions of working from home in his or hers pajamas. 

Scrum in TV shows and in real life

In an old episode of HBO “Silicon Valley” show, a soft-spoken MBA-bearing non-technical dude introduces Scrum to the hard-charging engineering team by presenting them with a Scrum board. The purpose of the system is declared to be “visibility into who is working on what”. As it becomes clear in the next 30 seconds, the actual value the team gets out of using the Scrum board (and, by extension, Scrum) appears to be the competition between the engineers who appear to work harder and faster on their separate stories to one-up each other.

So what is Scrum, and what does it do?

Scrum.org offers this definition: “Scrum is a management and control process that cuts through complexity to focus on building software that meets business needs.” If that’s too complicated, here’s another attempt at an explanation: “Scrum itself is a simple framework for effective team collaboration on complex software projects.”

While this does not explain what Scrum is (one should take multi-day training courses to fully groke that), it starts to emerge that Scrum is about:

1. teams
2. complex projects
3. visibility into the work of and collaboration between individual contributors

First two items describe common reality, while the last one is often a hard-to-reach state.

Visibility and collaboration are a worthy goal for many organizations. Teams produce better value if all participants were fully aware of the overall vision and what other people are working on.

Collaboration and friendly competition are known to improve productivity and quality of the work. Installing and maintaining a Scrum board is a lot less work than adopting proper Scrum. Yet, it is a tiny investment with outsized return.

A story of one code review

As told by a developer: 

A. and D. told me that the changes that were done to method “E” on class “C” are not needed, and also those two test cases for these changes were not needed. So A. and I removed those. I promoted these changes to my stream.

I don’t really understand why. 

Also A. said that the test “T” should make sure we are asserting that those values are really still there. I thought it did do that, but I guess not.

Also D. suggested an improvement to the code that I don’t fully understand: In class “U” method “D” we can pass the “c” into the “T” method on line 35 and then we don’t need to pass it in on line 39. I didn’t catch exactly what he was saying to do.
I may be missing some other things too.

---

I hope the code was improved as a result of the changes made based on this code review.  But even so, this is not the best way to run code reviews.

Code review is there to get all the team members on the same page in regards to writing good code, catch hard-to-detect issues, and learn from each other.  The discussion that happens in code review is an invaluable tool in building a strong technical community, and ensuring the technical quality immediately and over the medium- and longer-term. It is important that all participants gain an understanding of what changes would make better code and why, in addition to simply following typing instructions.

Here's a slide deck "Effective Code Review" for a discussion about code review practices , that focus on learning, creating a community with a common vision, and building social capital.

Are we having fun yet?

Legoland theme-park has very few Lego creatures. A staff member explained:

- It is a lot of work building them out of small bricks. 

That just sounds backwards. Building with Legos is fun, not work. How can that be?

For the folks working at the Legoland building from Legos is hard work. The resort operates by strict, unbending rules. Cheerful staff, while smiling and genuinely trying to be helpful, is going by a well-memorized script. Thinking on the job is discouraged, and absolutely no initiative or having fun is permitted. These guys look and act like well-behaved robots, which they are payed to be during the work hours. Robots do not have fun, no matter the activity. 

Most of us are not paid to build Legos, but a lot of our jobs can be no less fun and creative than putting little colorful bricks together. It is important to not let rules, scripts, and check-your-humanity-at-the-door culture ruin that fun.

Empowering Agile teams: culture clash. Part II

This is a follow up to the earlier article on this blog "Empowering Agile teams: culture clash".

Agile frameworks empathize empowered teams, flat structures, and meritocracy. Agile transformation is about re-defining who holds authority and responsibility.

Agile approach both allows and requires all participants to care. This is the other side of the struggle. While it is hard for command-and-control management to step away and let the team self-manage, it is even harder for the team members to suddenly start to care.

Power concentrated on the top of the org chart leads to classical principal-agent problem. In many a traditional organization team members have no reason to care about technical quality, steady pace of delivery, or building the product that customers actually want. Management makes decisions about what gets done and how results are accessed, and carries responsibility to the customer. Team members are expected to show up and do what they are told.

Self-managed teams, the corner stone of the Agile organization, have the information, the skills and the power to deliver what the customer wants. To become self-managed and claim that power, the teams also needs a will, and perhaps some incentives, to care about the organizations’ success. Agile transformation is about reminding people of their power, and waking up that will to care and be in control.