Agile as a Solution for "Miscalibration Errors"

This content is syndicated from LeadingAnswers: Leadership and Agile Project Management Blog by Mike Griffiths. To view the original post in full, click here.

Error Malcolm Gladwell (author of Blink and Tipping Point) was in town a couple of weeks ago and I enjoyed a great presentation he gave on what happens when we think we have complete information on a subject.

The Problem
Gladwell asserts that the global economic crisis was largely caused by “Miscalibration Errors”. These are errors made by leaders who become over confident due to reliance on information. Those in charge of the major banks were smart, professional, and respected people at the top of their game; who, as it turns out, are prime candidates from miscalibration errors.

People who are incompetent make frequent, largely unimportant errors, and that is understandable. They are largely unimportant errors because people who are incompetent rarely get into positions of power. Yet those who are highly competent are susceptible to rare, but hugely significant errors. 

Think of the global economic crisis where bank CEOs were seemingly in denial of the impending collapse of the sub-prime mortgage market. (I don’t mean close to the end when they were secretly betting against the market while still recommending products to their clients, but earlier on when they were happy to bet their own firms on “AAA” rated derivatives that they knew were really just a collection of highly suspect subprime mortgages.)

Anyway, this phenomenon of educated, well informed leaders making rare, but catastrophic errors is not new and unlikely to go away soon, it seems to be a baked-in human flaw. When presented with increasing levels of information our perception of judgement accuracy increases when in reality their judgement may be very suspect. Let’s look at some examples:


An Example
In a study by Stuart Oskamp, psychiatrists given one paragraph of information about a patient and asked to make a diagnosis and rate their confidence in their diagnosis rated their confidence at about 25%. This was uncannily close to their actual diagnosis accuracy of about 25%.

The same psychiatrists were then given one page of notes about the patient and the process repeated. Their confidence in their diagnosis increased to 40% even though their success rates stayed pretty close to 25%. Each time more information was provided about the patient, 10 pages, and then their whole file, their confidence increased to 70% and 90% while their diagnosis accuracy crept up from 25%, to 27%, and eventually 29%. Now we have a dangerous position, highly skilled, respected professionals, at the top of their game, with few people to question them, highly confident of their flawed opinion.

As Gladwell was outlining this situation I thought about my time spent planning and scheduling traditional projects, before transitioning to agile. Scope was defined, detailed requirement documents created, estimates created and vetted by skilled teams and correlated with several different estimation approaches. With increasing data, rigour, and the application of software engineering best practice, my confidence in the plan increased. The plan that is so carefully crafted, resource levelled, with every known risks balanced with contingency; so that  even if the unexpected happens we had a plan for that, we must be prepared and positioned for success.

This was a classic miscalibration error, with increasing levels of information leading to overconfidence regarding an inherently unknowable domain (custom software development) that inevitably leads to the occasional failure or major project issue. 

Specifications are another area prone to miscalibration errors too. When we spend a lot of time gathering specifications, validating specifications and elaborating alternative flows and exceptions we build a sense of confidence in them.  Early in my career I worked on military software development where we used formal specification languages like “Z” and “OBJ”. These specifications were machine readable and machine checkable. By writing very tight, well defined specs where any omissions would be caught and traceability from requirement to feature to test was assured, we thought we had our bases covered. Yet unforeseen problems still occurred.

The Solution
Gladwell suggested ”frequent checkpoints with reality” as a solution for miscalibration errors. Rather than allowing ourselves to build on top of premises, plans and specifications, force a verification step. With a financial derivative this may mean assessing the risk of each component and comparing risks – a complicated process in the financial crisis as the documentary “Inside Job” reveals. However for project managers agile provides many opportunities.

The iteration demos and team velocity have a sobering impact of grandiose project plans. Yes, we may have planned to be at point “X” in the project by now, but if the business has only accepted functionality “Y” and the team velocity averages 10 widgets per month, then this is the reality we need to live with.

Likewise for specifications, they are at best today’s reasoned guess at what is required, but usage reveals true requirements. Rather than months cataloguing features we think are important, greater value can be extracted from a balance of specification and evaluation.

Misscalibration errors are especially difficult to catch because they are made by domain experts who are hard to argue with. Question the BA team on the specification and they can bury you in details about processes followed, checks and balances used, and customer signatures. Question the PM on a project plan and the same happens, every best practice can be followed, people can be consulted, all of which builds a sense of stakeholder confidence, yet the risks are there.

The more information we gather the more we increase our internal perception of certainty or diagnosis. Frequent checkpoints with reality can be humbling as we surface small, iteration-long miscalibration errors, but these are typically survivable and critical for avoiding larger project-based miscalibration errors that can be fatal.

Leave a Reply

What is 5 + 8 ?
Please leave these two fields as-is:
Please do this simple sum so I know you are human:)

There are 101 ways to approach anything.
To find the best way, sometimes you need expert help

What People Say

“Kelly revolutionised the way our digital department operated. A true advocate of agile principles, he quickly improved internal communication within our teams and our internal clients by aligning our business and creating a much enhanced sense of transparency in the decisions the business was making. Kelly also introduced a higher sense of empowerment to the development teams...”

PETER SILVA-JANKOWSKI
IPC MEDIA

“Kelly’s a leading program director with the ability to take charge from day one and keep strong momentum at both a program and project level driving prioritisation, resourcing and budgeting agendas. Kelly operates with an easy-going style and possesses a strong facilitation skill set. From my 5 months experience working with Kelly, I would recommend Kelly to program manage large scale, complex, cross company change programs both from a business and IT perspective.”

LUKE SHARKEY /STRATEGY & IMPLEMENTATION LEADER
SUNCORP

“Kelly is an extremely talented and visionary leader. As such he manages to inspire all around him to achieve their best. He is passionate about agile and has a wealth of experience to bring to bear in this area. If you're 'lucky' he might even tell you all about his agile blog. Above all this, Kelly is great fun to work with. He is always relaxed and never gets stressed - and trust me, he had plenty of opportunity here! If you get the chance to work with Kelly, don't pass it up.”

GILES BENTLEY, DEVELOPMENT & OPERATIONS DIRECTOR
TIME INC

“Kelly is an Agile heavy-weight. He came in to assess my multi-million $ Agile development program which wasn’t delivering the right throughput. He interviewed most of the team and made some key recommendations that, when implemented, showed immediate results. I couldn’t ask for more than that except he’s a really nice guy as well.”

DAN PULHAM, DIGITAL DIRECTOR
TELSTRA

“Kelly and I worked together on a very large project trying to secure a new Insurer client. Kelly had fantastic commercial awareness as well as his technical expertise. Without him I would never had secured this client so I owe a lot to him. He is also a really great guy!”

GINA MILLARD
GLASS'S INFORMATION SERVICES

“Kelly came to the department and has really made a huge impact on how the department communicates, collaborates and generally gets things done. We were already developing in an agile way, but Kelly has brought us even more into alignment with agile and scrum best practices, being eager to share information and willing to work with us to change our processes rather than dictate how things must be done. He is highly knowledgable about agile development (as his active blog proves) but his blog won't show what a friendly and knowledgeable guy he is. I highly recommend Kelly to anyone looking for a CTO or a seminar on agile/scrum practices - you won't be disappointed!”

ANDY JEFFRIES/TECHNICAL LEAD
IPC MEDIA

“Kelly was a great colleague to work with - highly competent, trustworthy and generally a nice bloke.”

HANNAH JOYCE
GLASS'S INFORMATION SERVICES

“Kelly was engaged as a Program Director on a complex business and technology transformation program for Suncorp Commercial Insurance. Kelly drew on his key capabilities and depth of experience to bring together disparate parties in a harmonised way, ensuring the initiate and concept phases of the program were understood and well formulated. Excellent outcome in a very short time frame. ”

BRUCE WEIR/EGM
SUNCORP

“I worked with Kelly on many projects at IPC and I was always impressed with his approach to all of them, always ensuring the most commercially viable route was taken. He is great at managing relationships and it was always a pleasure working with him.”

BEATRIZ MONTOYA/CONSUMER MARKETING DIRECTOR
IPC MEDIA

“I worked with Kelly whilst at Thoughtworks and found him to be a most inspiring individual, his common-sense approach coupled with a deep understanding of Agile and business makes him an invaluable asset to any organisation. I can't recommend Kelly enough.”

PETER THATCHER, SENIOR ACCOUNT DIRECTOR
ThoughtWorks

“Kelly was a brilliant CTO and a great support to me in the time we worked together. I owe Kelly a great deal in terms of direction and how to get things done under sometimes difficult circumstances. Thanks Kelly.”

JULIE PEEL
GLASS'S INFORMATION SERVICES