The Business Analysis Benchmark Study put out by IAG Consulting has become a great source of information for studying the maturity level of how organizations define and manage their requirements. Just recently the 2009 edition of the study was finalized and published, and we wanted to know a bit more information on the history of the study and the key findings the folks at IAG found to be most interesting and/or surprising.
Below are a few questions we asked of Keith Ellis, Vice President of Marketing and Strategic Alliances at IAG Consulting:
How many years has IAG Consulting been putting out this benchmark study? What initially triggered the idea of doing a study of this magnitude?
We’re in our second year in publishing the study. In 2008, we focused on the impact of requirements quality on project outcome in larger strategic projects. In 2009 we expanded the scope to look at the impact of requirements maturity across the spectrum of projects in IT and the impact of this on overall corporate competitiveness. We also made the 2009 study more ‘solution focused’ – subtitling it “the Path to Success”, and dedicated more of this year’s effort to uncovering exactly how performance change can be made in requirements definition and management maturity.
The idea came from seeing an alignment between:
- our company’s long-term passion in researching best practices,
- the skills of our executive team in creating a larger-scale research studies, and,
- a lack information for CIOs on both the impact of requirements definition and management and effective actions people can take to make improvement.
People intuitively understand that requirements definition and management quality is important – but people tend not to take effective action as a result of this understanding. It’s like weight loss; people know the benefits of a healthy lifestyle, but we face an unprecedented wave of increasing obesity as a result of ineffective action. Similarly, 70% of project failure can be attributed to poor requirements definition and management, yet despite people knowing requirements are important, we’re seeing increasing project failure rates in the recent past (Standish, Chaos Report) rather than decreasing rates. We wanted to give people the tools they need to diagnose where their organization is experiencing weakness, and be able to map out a strategy that more predictably brings success.
What are some of the most interesting or surprising results your team found in this year’s study?
The magnitude of the performance difference between low maturity companies and high maturity companies continues to blow me away – as does the strength of the correlation between requirements maturity level and project success rates (the higher the maturity, the higher the success rate). It’s staggering – but we also knew this to be the case from our research last year.
I think the big interesting ones this year were some of the myths we dispelled: for example finding that lower skilled people in a high maturity organization OUTPERFORM far better people in a low requirements maturity organization. This really showcases that requirements maturity is not about a deliverable, and, you can’t simply hire great analysts and expect the problem of poor quality requirements to simply go away. Requirements maturity is much more complex, and improving maturity is about a combination of people, process and technology coming together the right way, with the right organization support to achieve success. This same finding comes out loud and clear when we compared agile, iterative, waterfall and visualization/prototyping methods to each other and showed that there is an immaterial performance difference between methods for any given level of requirements maturity. WOW. Again, the findings scream: you need to bring together people, process and technology surrounding requirements definition and management maturity if you are to successfully enact performance improvement no matter what your SDLC or development framework.
There were also some interesting findings when we compared the stock market performance of high maturity firms: very large, publically traded companies, with institutionalized and consistent practices in requirements definition and management, vastly outperformed their peers. The ROA of level 4 requirements definition and management firms was 10% higher than the ROA of their peers. I think we’ll see a growing body of evidence over the next few years that more definitively shows that the maturity of internal practices and disciplines like requirements definition and management can lead to significant and sustainable competitive advantage. Simply give me 3X the sample size, a bit more historical data, and a market that’s not busy going through the biggest correction since the great depression, and we’d have some better insight in this area.
On the implementation side we demonstrated the impact of IAG’s Requirements Maturity Model as a roadmap for success. Again, not a surprise given last year’s study and all our client engagements using the model, but it’s important to get out there how to make improvement possible in this complex area and the report documents in detail key steps to achieve success. Check it out at http://www.iag.biz/benchmark
So, how are you seeing the level of maturity in requirements definition and management practices evolving over the past few years?
I think if we went back 5 to 10 years, you would see a huge difference. People have come a long way in their thinking. I’d bet, 10 years ago most companies would be level 0 or 1. The problem is, the massive breakthrough in performance change only really happens at level 3 or 3.5 and the vast majority of companies are in the 2ish range. You’ve got a situation where today there is far more intense focus on the area (86% of companies trying to make improvement here), but people are not swinging the bar far enough to really catalyze significant benefit. People have a lot of blood, sweat and tears tied up in what they’ve accomplished so far. What I hope is, rather than seeing all this effort yielding less benefit than they’d like and giving up, that people realize they’ve already pushed through some of the hardest barriers of organizational change and they may be closer than they think to realizing measurable performance gains. If the current level of focus on improvement is maintained, the next five to seven years could see substantial reductions in project failure rates in the market.
Where do you think companies should be investing their efforts to address requirement needs in the long-term?
I think the answer changes depending on the organization. You really need to assess the overall maturity of the organization across 6 competency areas (process, techniques, staff, deliverables, organization and technology) and identify where organizations are strong or weak. I’ve got clients that have great deliverables and technique (probably level 4 or 5), but weaknesses in staff, process and organization kill their effectiveness. I’ve got others that have pretty a strong staff, and have deliverables structure that’s killing them. I have others that are extremely strong in the later stages of the requirements definition and management lifecycle, but are very weak at the upfront requirements planning and elicitation layers. The point being, it’s essential to know the current strengths upon which you can build, and the weaknesses that require the greatest level of immediate focus to create an effective action plan.
I think another issue faced by companies is they really need to showcase the performance impact of great requirements maturity. You can never ignore that people need to experience a new process before they can internalize how it’s different. When people experience well facilitated requirements discovery sessions, see how their team is coming together, and get a few projects delivered on-time and on budget, then all of a sudden other business stakeholders will want to get on board. You’ll go from no one wanting an analyst on their project, to the corporate leadership clamoring for resources and wanting to get the best-of-the-best analyst team allocated to their project.
How does IAG plan to make use of the findings in the benchmark study in your own consulting efforts?
We’ve been using the Requirements Maturity Model for a number of years now when we’ve been putting together Requirements Centers of Excellence for Fortune 500 class companies. We have about five of these on-going at any given point in time although different companies call them different things: RCoE, Community of Practice, Requirements Management Office, Requirements Standards Office, etc… In these, the requirements maturity model helps us assemble the blueprint for organization change. With this research behind us, we’ve got high confidence that the path recommended not only yields better results, it is optimized in the path to achieve these results.
We’re also seeing more companies come to us looking to benchmark their organization’s maturity level. Many of these have taken great strides forward already, and want to get a more objective assessment that baselines the maturity of their organization, compares various parts of the organization against each other, and finds gaps. Seeing these kinds of engagements tells me there are a lot of organizations out there whose leadership are absolutely convinced of the value of great requirements definition and management maturity – and they’re taking it to the next level. Not only are these organizations already mature – they’re gunning to radically outperform everyone. What’s exciting to watch is just how much performance improvement they’re experiencing and the corporate commitment they’re getting as a result.