Lies, Damn Lies, and Statistics

This content is syndicated from LeadingAnswers: Leadership and Agile Project Management Blog by Mike Griffiths. To view the original post in full, click here.

"Calling Hallelujah Always Offends Someone"

Dangerous Statistics I am glad the PMI is finally recognizing agile methods, Ken Schwaber recently posted about the PMI Agile Certification, saying that he “…welcomes this and looks forward to PMI shifting from its previous approach to an agile approach. The test of this will be, of course, the success of the projects that adhere to its principles. In the past, the success (or yield) of their predictive approach has been less than 50% of projects (on time, on date, with the desired functionality.)”

He was quoting from the Standish CHAOS Report that comes out every couple of years and documents the success and failure rates of IT projects. The CHAOS reports have been published since 1994, the same year DSDM appeared and when many agile methods were getting going. Each year the results vary slightly, but the general theme is that many IT projects are challenged and results like the following are typical:

    * 32%   Successful (On Time, On Budget, Fully Functional)
    * 44%   Challenged (Late, Over Budget, And/Or Less than Promised Functionality)
    * 24%   Failed (Cancelled or never used)
    * 61%   Feature complete

It is interesting then that Ken attributes the poor success rates of IT projects since the start of agile to be a PMI problem. You would think that with the rise of agile methods and the success of all these Scrum, XP, FDD, and DSDM projects we hear about, that these statistics would have turned right around!

Maybe agile adoption is still too small and most companies are running single pass waterfall projects, or have command-and-control project managers? Well, not if you believe Forester Research and Dr Dobb’s. They indicate that agile methods are now used more than any other approach for IT projects. With 35% of companies using agile and a further 21% using some kind of iterative approach and only 13% using a waterfall approach.

  Agile Adoption
So if 56% of projects, back in 2009, were already getting the adaptive feedback benefits of an iterative approach why are the CHAOS report findings not getting any better? Also the increase in agile continues and last year Gartner predicted “By 2012 agile development methods will be utilized in 80% of all software development projects”. So why the poor results, and tell me again why the failures we see are PMI problems caused by “their predictive approach” when it appears these approaches are already in the minority?

I have been bothered by the apparent lack of improvement in CHAOS report findings for about 5 years; because I thought the increase in agile methods should have made a big difference. I also use them as an example of why there is a need for companies to switch to agile methods, if the current approach is not working so well, maybe we should try something else.

In the early days of the Agile 200x Conferences we had 150 to 300 attendees and they tended to be the innovators and early adopter organizations. However, in the last 5 years, the conferences have swelled to 1,200, 1,500 and more attendees as agile clearly crossed the chasm. Attendees are not just from software companies anymore, but local government, utilities and risk adverse insurance companies are all using agile methods, and yet those CHAOS report findings hardly budge.

Shown below are the CHAOS Report findings from 1994 to 2009.
CHAOS Results
We can see there has been a slight increase in Successful projects, but no “hockey-stick” shaped uptick to mark the arrival of agile or even anything close to the 35% increase associated with agile adoption by 2009. After doing some research it became apparent that their definition of success is quite different from that of the average sponsors.

The CHAOS reports definition of successful is not “functionality within +/- 20% of budget or schedule” as you might think. Instead they calculate a project’s success measure by counting the number of projects that have an initial forecast larger than the actual for cost and time, and one that’s smaller for functionality. This is divided by the total number of projects to calculate the success rates. Standish Group defines its success measure as a measure of estimation accuracy of cost, time, and functionality.

If that sounds confusing then it is appropriate, while their math might be statistically defendable, it is more a measure of estimate variance within a set of projects than happy or disgruntled customers. Maybe our measuring stick is broken, or has a funny scale?

  Bad Measure

That said many useful findings emerge from the CHAOS reports. I attended a great presentation by Jim Johnson from The Standish Group a few years back where he described their studies into the successful projects they found. Here we find some useful endorsements for agile methods. The following table shows the top ranked factors for project success.

Success Factors
User Involvement features prominently in agile methods as do Small Milestones. Also agile’s frequent interaction with business and product demos contribute to surfacing Clear Business Objectives. All of these factors contributing to successful projects feature prominently in agile methods.

Returning to Ken’s post, I agree with his initial comment of welcoming the PMI’s adoption of agile methods. There definitely is a need to embrace processes that better fit today’s knowledge worker environments. Predictive planning does not work in high change environments, and command-and-control management is no way to engage or motivate a team. In short, project management needs to move on.

Yet to attribute today’s IT failures, by CHAOS standards, to PMI processes is likely invalid, when agile is now the most widely used IT approach. Depending on your view this association can be seen as “optimistically naïve” because the causal link is weak, or “cynically canny” since the CHAOS Report findings will not change dramatically until they adjust their definitions of success.

For illustration a similarly afflicted statement would be: “I welcome Ken’s endorsement of the PMI initiative and think the test of this support will be, of course, the success of agile methods in solving global warming, which in the past has been less than successful”. The link is just not there.

This post is a bit of as departure for me; I don’t want to get engaged in flame wars and am indebted to Ken for teaching me about Scrum. Yet because we see the CHAOS report findings widely publicized as justification for change, I wanted to share some caution about their application. Also as agile methods become the norm, criticisms about the status quo are criticisms of agile.

I see the irony and duplicity here; when agile was small the CHAOS statistics were the rallying cry for change. Now they are addressing agile projects I am calling them questionable. As they say there are lies, damn lies, and statistics. I guess we need to choose our statistics well or be willing to flip-flop to sets that suit our arguments. Jim Johnson from the Standish group often gets asked what “CHAOS” stands for, one individual suggested “Calling Hallelujah Always Offends Someone” which seems apt here.

    1) The Rise and Fall of the CHAOS Report Figures, IEEE Software, 2010
    2) The CHAOS Chronicles, Jim Johnson, 2003

Leave a Reply

What is 4 + 8 ?
Please leave these two fields as-is:
Please do this simple sum so I know you are human:)

There are 101 ways to approach anything.
To find the best way, sometimes you need expert help

What People Say

“Kelly revolutionised the way our digital department operated. A true advocate of agile principles, he quickly improved internal communication within our teams and our internal clients by aligning our business and creating a much enhanced sense of transparency in the decisions the business was making. Kelly also introduced a higher sense of empowerment to the development teams...”


“Kelly’s a leading program director with the ability to take charge from day one and keep strong momentum at both a program and project level driving prioritisation, resourcing and budgeting agendas. Kelly operates with an easy-going style and possesses a strong facilitation skill set. From my 5 months experience working with Kelly, I would recommend Kelly to program manage large scale, complex, cross company change programs both from a business and IT perspective.”


“Kelly is an extremely talented and visionary leader. As such he manages to inspire all around him to achieve their best. He is passionate about agile and has a wealth of experience to bring to bear in this area. If you're 'lucky' he might even tell you all about his agile blog. Above all this, Kelly is great fun to work with. He is always relaxed and never gets stressed - and trust me, he had plenty of opportunity here! If you get the chance to work with Kelly, don't pass it up.”


“Kelly is an Agile heavy-weight. He came in to assess my multi-million $ Agile development program which wasn’t delivering the right throughput. He interviewed most of the team and made some key recommendations that, when implemented, showed immediate results. I couldn’t ask for more than that except he’s a really nice guy as well.”


“Kelly and I worked together on a very large project trying to secure a new Insurer client. Kelly had fantastic commercial awareness as well as his technical expertise. Without him I would never had secured this client so I owe a lot to him. He is also a really great guy!”


“Kelly came to the department and has really made a huge impact on how the department communicates, collaborates and generally gets things done. We were already developing in an agile way, but Kelly has brought us even more into alignment with agile and scrum best practices, being eager to share information and willing to work with us to change our processes rather than dictate how things must be done. He is highly knowledgable about agile development (as his active blog proves) but his blog won't show what a friendly and knowledgeable guy he is. I highly recommend Kelly to anyone looking for a CTO or a seminar on agile/scrum practices - you won't be disappointed!”


“Kelly was a great colleague to work with - highly competent, trustworthy and generally a nice bloke.”


“Kelly was engaged as a Program Director on a complex business and technology transformation program for Suncorp Commercial Insurance. Kelly drew on his key capabilities and depth of experience to bring together disparate parties in a harmonised way, ensuring the initiate and concept phases of the program were understood and well formulated. Excellent outcome in a very short time frame. ”


“I worked with Kelly on many projects at IPC and I was always impressed with his approach to all of them, always ensuring the most commercially viable route was taken. He is great at managing relationships and it was always a pleasure working with him.”


“I worked with Kelly whilst at Thoughtworks and found him to be a most inspiring individual, his common-sense approach coupled with a deep understanding of Agile and business makes him an invaluable asset to any organisation. I can't recommend Kelly enough.”


“Kelly was a brilliant CTO and a great support to me in the time we worked together. I owe Kelly a great deal in terms of direction and how to get things done under sometimes difficult circumstances. Thanks Kelly.”