Metrics for Agile

This content is syndicated from Insights You Can Use by Esther Derby. To view the original post in full, click here.

“How can we tell how far along we are with our agile adoption?”

I heard this question again the other day.

Usually, the person who asks the question starts to answer it:

Number of teams using agile

Number of people trained in agile

Number of projects using agile

Number of certified coaches.

Metrics like these won’t tell you what you need to know. More likely, they will lead you astray. How? Let me tell you a story.

Years ago, I worked for a company that was “installing” a Big Methodology from a Big Company.  (The fact that they thought they were “installing” a methodology was probably the first warning sign.)

Every one in the department attended Big Methodology training. (This practice is sometimes called “Sheep Dip” training).

The VP mandated that all projects would use the Big Methodology.

The Installation Team audited to ensure that project managers and teams were complying and producing the required “work products” in accordance with the Required Work Products grid in the back of the very large Big Methodology binder.

Of course, there was some grumbling (from the people the Installation Team referred to as “Change Resisters.”)  Eventually, people did comply. Every one went to training. Projects managers filled out the required templates, and checked the appropriate boxes.  The metrics looked grand!

The VP declared, “Big Methodology is now business as usual!”

At the time, I scoffed at that statement. It was clear to me that people were not using Big Methodology, and that the promised benefits were nowhere in sight. The only things that had really changed were some check boxes and some names (documents became “work products” or “job aids,”).

But, now, I realize that the VP’s statement was TRUE!

We had Big Methodology, and things went on as they had–business as usual! Well, maybe a little worse because people were spending time producing the many documents specified on the Required Work Products grid.

The metrics the VP tracked were easy to count. But they only revealed surface compliance. They didn’t say anything about whether the organization was achieving the improvements promised by Big Methodology and hoped for by the VP.

So when you think about assessing how far along you are in your agile transformation, consider what you are trying to achieve.

I often suggest that managers track three metrics to understand how well their organization is functioning, and whether they are trending in the right direction.

The ratio of fixing work to feature work. How much time are people spending developing valuable new features vs. fixing stuff that wasn’t done right the first time? If you can figure out the sources of fixing work and make some progress there, you have a boost to productivity. Agile methods can address some of the sources of fixing work…but not all of them.

 Cycle time. How long does it take to go from an idea to a valuable product in the hands of a customer? Again agile methods can help with delivery. But if it’s the upstream process–planning for products and releases–is broken, you may not see improvement until you address those issues, as well as the development process.

Number of defects escaping to production. This is a category of fixing-work that is a direct indicator that the quality of the development process is improving.

For each of these metrics, it is the trend that is important, not an absolute number.  The trend will tell you if your attempts at improvement are having an effect. Remember, most changes take time to take hold. If the trend doesn’t move in a month, it may not mean you have taken the wrong action and need to change direction. If the trend isn’t moving over time, then, examine what is happening in the development area. But also look at other aspects of the system. There are few one-to-one cause and effect relationships in complex systems and the trend you see may or may not be directly related to your change. One company I worked with was alarmed to see that defects released to production went up after they started using agile methods. It turned out that prior to the effort to measure defects released to production, no one paid much attention unless the defect brought down a customer site. The increase in the defects trend was related to reporting, not a failure to improve quality.

I find that the three metrics above are generally useful for understanding how a software development organization is functioning as a system. But your reasons for adopting agile methods may be different.  Consider the goals you are trying to achieve.  What signals would tell you that you are moving in the right direction?  How might you measure those?  When you think about measures, be wary of target numbers. Measuring against targets almost always causes distortion. That means that people will behave so as to reach the target, perhaps in ways that are counter to the actual goal behind the target. Distortion will keep you from seeing the real picture, and may also cause real harm to your organization.

Useful metrics give you a window into how the system is functioning, and whether your change is having an effect. The numbers themselves are neither good nor bad. They are information that signals you to go and find out, investigate and reason about the system.


© derby for Insights You Can Use, 2011. | Permalink | 17 comments | Add to del.icio.us
Post tags: , , ,

Feed enhanced by Better Feed from Ozh

Leave a Reply

What is 8 + 3 ?
Please leave these two fields as-is:
Please do this simple sum so I know you are human:)

There are 101 ways to approach anything.
To find the best way, sometimes you need expert help

What People Say

“Kelly revolutionised the way our digital department operated. A true advocate of agile principles, he quickly improved internal communication within our teams and our internal clients by aligning our business and creating a much enhanced sense of transparency in the decisions the business was making. Kelly also introduced a higher sense of empowerment to the development teams...”

PETER SILVA-JANKOWSKI
IPC MEDIA

“Kelly’s a leading program director with the ability to take charge from day one and keep strong momentum at both a program and project level driving prioritisation, resourcing and budgeting agendas. Kelly operates with an easy-going style and possesses a strong facilitation skill set. From my 5 months experience working with Kelly, I would recommend Kelly to program manage large scale, complex, cross company change programs both from a business and IT perspective.”

LUKE SHARKEY /STRATEGY & IMPLEMENTATION LEADER
SUNCORP

“Kelly is an extremely talented and visionary leader. As such he manages to inspire all around him to achieve their best. He is passionate about agile and has a wealth of experience to bring to bear in this area. If you're 'lucky' he might even tell you all about his agile blog. Above all this, Kelly is great fun to work with. He is always relaxed and never gets stressed - and trust me, he had plenty of opportunity here! If you get the chance to work with Kelly, don't pass it up.”

GILES BENTLEY, DEVELOPMENT & OPERATIONS DIRECTOR
TIME INC

“Kelly is an Agile heavy-weight. He came in to assess my multi-million $ Agile development program which wasn’t delivering the right throughput. He interviewed most of the team and made some key recommendations that, when implemented, showed immediate results. I couldn’t ask for more than that except he’s a really nice guy as well.”

DAN PULHAM, DIGITAL DIRECTOR
TELSTRA

“Kelly and I worked together on a very large project trying to secure a new Insurer client. Kelly had fantastic commercial awareness as well as his technical expertise. Without him I would never had secured this client so I owe a lot to him. He is also a really great guy!”

GINA MILLARD
GLASS'S INFORMATION SERVICES

“Kelly came to the department and has really made a huge impact on how the department communicates, collaborates and generally gets things done. We were already developing in an agile way, but Kelly has brought us even more into alignment with agile and scrum best practices, being eager to share information and willing to work with us to change our processes rather than dictate how things must be done. He is highly knowledgable about agile development (as his active blog proves) but his blog won't show what a friendly and knowledgeable guy he is. I highly recommend Kelly to anyone looking for a CTO or a seminar on agile/scrum practices - you won't be disappointed!”

ANDY JEFFRIES/TECHNICAL LEAD
IPC MEDIA

“Kelly was a great colleague to work with - highly competent, trustworthy and generally a nice bloke.”

HANNAH JOYCE
GLASS'S INFORMATION SERVICES

“Kelly was engaged as a Program Director on a complex business and technology transformation program for Suncorp Commercial Insurance. Kelly drew on his key capabilities and depth of experience to bring together disparate parties in a harmonised way, ensuring the initiate and concept phases of the program were understood and well formulated. Excellent outcome in a very short time frame. ”

BRUCE WEIR/EGM
SUNCORP

“I worked with Kelly on many projects at IPC and I was always impressed with his approach to all of them, always ensuring the most commercially viable route was taken. He is great at managing relationships and it was always a pleasure working with him.”

BEATRIZ MONTOYA/CONSUMER MARKETING DIRECTOR
IPC MEDIA

“I worked with Kelly whilst at Thoughtworks and found him to be a most inspiring individual, his common-sense approach coupled with a deep understanding of Agile and business makes him an invaluable asset to any organisation. I can't recommend Kelly enough.”

PETER THATCHER, SENIOR ACCOUNT DIRECTOR
ThoughtWorks

“Kelly was a brilliant CTO and a great support to me in the time we worked together. I owe Kelly a great deal in terms of direction and how to get things done under sometimes difficult circumstances. Thanks Kelly.”

JULIE PEEL
GLASS'S INFORMATION SERVICES