Measuring Agile Investments – Quality

This content is syndicated from Agile Development Blog: Scaling Software Agility by Isaac Montgomery. To view the original post in full, click here.

Quality – there may be no more multi-faceted and powerful attribute in successful software development. Quality is central to everything we do and seek.
  • Higher quality leads to greater productivity, throughput and velocity
  • Higher quality leads to increased responsiveness, reduced cycle-times, shorter lead-times
  • Higher quality leads to improved customer satisfaction, employee satisfaction
  • Higher quality leads to better predictability, reduced risk, improved decision making
Or at least that’s my hypothesis… And that hypothesis is widely shared amongst the Agile and product development communities.  We’ve developed numerous principles, practices and techniques intended to improve quality:  Test Driven Development; Continuous Integration; Automated Build and Deploy; Pair Programming; Customer Demos; Behavior Driven Development; Acceptance Test Driven Development; and Set-based Design techniques are all at least partially focused on yielding quality improvements. But quality can’t simply be viewed as a set of tools and techniques – independent variables/levers which we hypothesize will lead to improved business outcomes. Quality is also a business outcome unto itself. This series emphasizes the need to focus on business outcomes (success) first – methods and practices second.  So, putting aside the methods and good practice assumptions of Agile, and focusing solely on the business outcome of improved quality: Quality = Fewer Defects in Production We apply Agile quality practices and techniques, because we believe that doing so will yield improved business outcomes – quality (fewer defects in production) being one of those outcomes – along with productivity, predictability, responsiveness, customer and employee satisfaction. Large, manual, end-of-cycle execution of formal testing by an independent QA organization is also a method aimed at improving these business outcomes. I hypothesize that it is less effective than alternative Agile techniques. But I don’t take that on faith, and neither should you. We must test our hypothesis. How Do We Measure Quality? There are innumerable quality metrics that have been devised over the years – each with its own strengths and weaknesses.  In my experience, it’s important to keep metrics simple, and to not let great become the enemy of good enough. In other words, if you have a metric that does a good job of providing insight into the quality of your product/solution, and is simple to collect and interpret; that is likely better than chasing after a metric that will do a great job, but would be more complicated. For my part, I’ve had success over the years using a couple relatively simple metrics:
  • DEFECT DENSITY – # Defects / KLOC
  • DEFECT ARRIVAL – # Defects Identified / Month
What Do We Consider a Defect? In both cases, I include only defects in the production system. Measuring defects found and eliminated during the development cycle may be useful for managing your development and quality processes.  But from a business outcomes perspective, our focus is reducing the number of defects that make it to production – not making assumptions about how or when to achieve that. Not All Defects Are Created Equal Any good metric should drive in more questions than answers.  I find it useful to tag defects with information about type and severity, so that we can consider some of those questions more deeply.
  • Our defect density is high; but our severity 1 & 2 density is low. What is the impact on other outcomes (productivity, customer satisfaction, etc.) if we were to invest in reducing our low severity defects?
  • Our defect arrival is very high immediately following a major release. But the defects are mostly Type = Usability. Why are our customers having such a tough time using our new features; and how can we ease them through the learning curve?
You may have some hypotheses based on these questions. Perhaps those hypotheses involve application or improved use of Agile tools and techniques. What experiments would you run to prove or disprove your hypothesis?  What new questions will those results yield? This is the third post in our blog series, Measuring the Impact of Your Agile Investments. The series focuses on measuring the impact that Agile practices have on business outcomes.

Leave a Reply

What is 5 + 6 ?
Please leave these two fields as-is:
Please do this simple sum so I know you are human:)

There are 101 ways to approach anything.
To find the best way, sometimes you need expert help

What People Say

“Kelly revolutionised the way our digital department operated. A true advocate of agile principles, he quickly improved internal communication within our teams and our internal clients by aligning our business and creating a much enhanced sense of transparency in the decisions the business was making. Kelly also introduced a higher sense of empowerment to the development teams...”


“Kelly’s a leading program director with the ability to take charge from day one and keep strong momentum at both a program and project level driving prioritisation, resourcing and budgeting agendas. Kelly operates with an easy-going style and possesses a strong facilitation skill set. From my 5 months experience working with Kelly, I would recommend Kelly to program manage large scale, complex, cross company change programs both from a business and IT perspective.”


“Kelly is an extremely talented and visionary leader. As such he manages to inspire all around him to achieve their best. He is passionate about agile and has a wealth of experience to bring to bear in this area. If you're 'lucky' he might even tell you all about his agile blog. Above all this, Kelly is great fun to work with. He is always relaxed and never gets stressed - and trust me, he had plenty of opportunity here! If you get the chance to work with Kelly, don't pass it up.”


“Kelly is an Agile heavy-weight. He came in to assess my multi-million $ Agile development program which wasn’t delivering the right throughput. He interviewed most of the team and made some key recommendations that, when implemented, showed immediate results. I couldn’t ask for more than that except he’s a really nice guy as well.”


“Kelly and I worked together on a very large project trying to secure a new Insurer client. Kelly had fantastic commercial awareness as well as his technical expertise. Without him I would never had secured this client so I owe a lot to him. He is also a really great guy!”


“Kelly came to the department and has really made a huge impact on how the department communicates, collaborates and generally gets things done. We were already developing in an agile way, but Kelly has brought us even more into alignment with agile and scrum best practices, being eager to share information and willing to work with us to change our processes rather than dictate how things must be done. He is highly knowledgable about agile development (as his active blog proves) but his blog won't show what a friendly and knowledgeable guy he is. I highly recommend Kelly to anyone looking for a CTO or a seminar on agile/scrum practices - you won't be disappointed!”


“Kelly was a great colleague to work with - highly competent, trustworthy and generally a nice bloke.”


“Kelly was engaged as a Program Director on a complex business and technology transformation program for Suncorp Commercial Insurance. Kelly drew on his key capabilities and depth of experience to bring together disparate parties in a harmonised way, ensuring the initiate and concept phases of the program were understood and well formulated. Excellent outcome in a very short time frame. ”


“I worked with Kelly on many projects at IPC and I was always impressed with his approach to all of them, always ensuring the most commercially viable route was taken. He is great at managing relationships and it was always a pleasure working with him.”


“I worked with Kelly whilst at Thoughtworks and found him to be a most inspiring individual, his common-sense approach coupled with a deep understanding of Agile and business makes him an invaluable asset to any organisation. I can't recommend Kelly enough.”


“Kelly was a brilliant CTO and a great support to me in the time we worked together. I owe Kelly a great deal in terms of direction and how to get things done under sometimes difficult circumstances. Thanks Kelly.”