Measuring Agile Investments – Productivity

This content is syndicated from Agile Development Blog: Scaling Software Agility by Isaac Montgomery. To view the original post in full, click here.

This is the second post in our blog series, Measuring the Impact of your Agile Investments. The series focuses on measuring the impact that Agile practices have on business outcomes. For many leaders, increasing the productivity of the development organization (delivering features the business needs and generating improved ROI) is their primary and over-riding goal. ‘Doing more with less’ is a mantra they preach to their teams continuously, and which colors every decision they make. Yet few have a clear and consistent definition of productivity or an effective means of measuring it.
Productivity: a relative measure of the efficiency of a system in converting inputs into useful outputs. Productivity: the ratio of the real value of outputs to the combined input of labour and capital. Productivity: a measure relating a quantity and quality of output to the inputs required to produce it.
So, in simplest terms:
Productivity = Valuable Outputs / Costly Inputs
Be Careful What you Wish For While I am a ardent believer in the value of metrics, it is important to keep in mind the potential for unintended consequences. Individual and group behaviors will evolve to meet your measures – not necessarily your goals. So ensure that the measures you use promote the behaviors you desire. Most documented productivity metrics in software development define a unit of output as either a KLOC (thousand lines of code) or as a Function Point. Both have significant disadvantages. KLOCs are generally favored for the simple reason that they’re unambiguous and easy to measure. However, they can unintentionally promote poor design and coding practices. Elegant, efficient, maintainable software takes more time to write, and takes fewer lines of code. Therefore, a KLOC metric inadvertently creates a disincentive for building high-quality software, and rewards poorly thought through designs and shoddy craftsmanship. While Function Points don’t necessarily promote poor design and craftsmanship, they have their own unique challenge – they are famously difficult and expensive to calculate and measure. Perhaps even more importantly, both KLOCs and Function Points ignore the core aspect of ‘Value’ inherent in our definition of output – and therefore productivity. It has been stated that 64% of the features and function in the typical software system are rarely or never used (Standish 2002). Calling the code that delivers these features and functions “productive” may be a mischaracterization. Similarly, those features and functions that ARE commonly used are generally not of uniform value. The Pareto Principle would suggest that 80% of the value is delivered by 20% of the effort. Yet KLOC and Function Point-based metrics treat all features and functions (and code) delivered as interchangeable. This promotes focusing on the easiest and lowest technical risk; rather than the most valuable, most innovative and greatest business risk…
Output is a measure of the value delivered, not the effort expended.
A number of people in the Agile community have written about an alternative unit of output measure – Value Points.  While the simplicity and value focus of this model is appealing, it has challenges when scaled beyond a few teams. In order to be meaningful at the organizational level, you must normalize the relative value point scale across teams and programs – which can be difficult and expensive. Also, the Value Point approach does not easily translate to initiatives and/or divisions that may not be delivering in an Agile manner. Having a common measure of output, and therefore productivity, is critical to measuring the impact of your Agile investment. So, the approach we recommend is to associate a percentage of the total value of a given initiative to each Minimally Marketable Feature (MMF) or production release. These percentages can then be applied to a any monetary business justifications (ROI, NPV, Discounted Cashflow, etc.) to arrive at an expected dollar value realized from each release. Hence,
Productivity = Total Value Realized (delivered to production) /  Total Cost of production (labor)
Using this approach, your organization (be it a Scrum team, a multi-team Agile program, a waterfall project, or even an entire product development group) base-lines their productivity for a period of time and monitors the change over time. Example:
The division has 3 initiatives in progress:
  • Initiative A:  Total Expected Value $5MM; being delivered by a Scrum team with an iteration run-rate of $70,000.
  • Initiative B:  Total Expected Value of $25MM; being delivered by 5 Scrum teams with a combined iteration run-rate of $400,000.
  • Initiative C:  Total Expected Value of $50MM; being delivered according to a project plan and resource matrix that charges $2.5MM to the project in the 1st quarter.
In Q1:
  • Initiative A: Released to production monthly and delivered a total of 60% of Expected Value; or $3MM. 25% of the backlog has been burned-down in terms of story points. Total Cost $210,000. PRODUCTIVITY = 14.29
  • Initiative B: Released to production quarterly and delivered a total of 65% of Expected Value; or $16.25MM. 35% of the backlog has been burned-down in terms of story points. Total Cost $1.2MM. PRODUCTIVITY = 13.54
  • Initiative C: Completed requirements definition and is 50% done with Detailed Design, and delivered 0% of Total Expected Value. Total Cost $1.2MM. PRODUCTIVITY = 0
*undelivered work is WIP, and therefore not yet productive. Aggregate Division Productivity for Q1 = 7.38
As you can see, it would be relatively straightforward to predict Q2 productivity – at the initiative as well as division levels – by assessing the various product roadmaps and traditional project plans. Those projections could then be used to:
  • drive discussion about trade-offs on where to allocate limited capacity and maximize productivity
  • staff and fund initiatives where productive potential is high, and to cancel successful projects whose greatest productive potential has already been harvested
  • inform intelligent business decisions – which is WHY we’re measure outcomes
By understanding the productivity of the development organization (its efficiency in delivering features the business needs and delivering improved ROI), leaders can effectively drive improvements and business decisions that maximize productivity – Doing More with Less. The next topic I’ll address in the Measuring Impact series is Quality. Most organizations claim to have a Quality focus, yet few take a holistic view of Quality or appreciate the strong correlation between Quality and our other Outcome Metrics – Productivity, Responsiveness, Customer Satisfaction, Employee Satisfaction and Predictability. How are you measuring the impact of your agile investments? Please share your ideas and feedback in the comments.

Leave a Reply

What is 3 + 1 ?
Please leave these two fields as-is:
Please do this simple sum so I know you are human:)

There are 101 ways to approach anything.
To find the best way, sometimes you need expert help

What People Say

“Kelly revolutionised the way our digital department operated. A true advocate of agile principles, he quickly improved internal communication within our teams and our internal clients by aligning our business and creating a much enhanced sense of transparency in the decisions the business was making. Kelly also introduced a higher sense of empowerment to the development teams...”


“Kelly’s a leading program director with the ability to take charge from day one and keep strong momentum at both a program and project level driving prioritisation, resourcing and budgeting agendas. Kelly operates with an easy-going style and possesses a strong facilitation skill set. From my 5 months experience working with Kelly, I would recommend Kelly to program manage large scale, complex, cross company change programs both from a business and IT perspective.”


“Kelly is an extremely talented and visionary leader. As such he manages to inspire all around him to achieve their best. He is passionate about agile and has a wealth of experience to bring to bear in this area. If you're 'lucky' he might even tell you all about his agile blog. Above all this, Kelly is great fun to work with. He is always relaxed and never gets stressed - and trust me, he had plenty of opportunity here! If you get the chance to work with Kelly, don't pass it up.”


“Kelly is an Agile heavy-weight. He came in to assess my multi-million $ Agile development program which wasn’t delivering the right throughput. He interviewed most of the team and made some key recommendations that, when implemented, showed immediate results. I couldn’t ask for more than that except he’s a really nice guy as well.”


“Kelly and I worked together on a very large project trying to secure a new Insurer client. Kelly had fantastic commercial awareness as well as his technical expertise. Without him I would never had secured this client so I owe a lot to him. He is also a really great guy!”


“Kelly came to the department and has really made a huge impact on how the department communicates, collaborates and generally gets things done. We were already developing in an agile way, but Kelly has brought us even more into alignment with agile and scrum best practices, being eager to share information and willing to work with us to change our processes rather than dictate how things must be done. He is highly knowledgable about agile development (as his active blog proves) but his blog won't show what a friendly and knowledgeable guy he is. I highly recommend Kelly to anyone looking for a CTO or a seminar on agile/scrum practices - you won't be disappointed!”


“Kelly was a great colleague to work with - highly competent, trustworthy and generally a nice bloke.”


“Kelly was engaged as a Program Director on a complex business and technology transformation program for Suncorp Commercial Insurance. Kelly drew on his key capabilities and depth of experience to bring together disparate parties in a harmonised way, ensuring the initiate and concept phases of the program were understood and well formulated. Excellent outcome in a very short time frame. ”


“I worked with Kelly on many projects at IPC and I was always impressed with his approach to all of them, always ensuring the most commercially viable route was taken. He is great at managing relationships and it was always a pleasure working with him.”


“I worked with Kelly whilst at Thoughtworks and found him to be a most inspiring individual, his common-sense approach coupled with a deep understanding of Agile and business makes him an invaluable asset to any organisation. I can't recommend Kelly enough.”


“Kelly was a brilliant CTO and a great support to me in the time we worked together. I owe Kelly a great deal in terms of direction and how to get things done under sometimes difficult circumstances. Thanks Kelly.”