In statistics the name given to mistaking noise for signal is known as “overfitting”. When delivering new Information Systems the concept is also prevalent, particularly at the stage in delivery that sends shivers down every IS project managers spine, the requirements stage!
Trying to “overfit” evolving business needs into a clearly defined and agreed set of system requirements is one of the most common causes of barriers to business acceptance of new systems. Persuading a business that the requirements have now been baselined and what has been agreed is now what is going to be delivered is quickly becoming a key skill of an IS project manager. Guarding the requirements to avoid any scope creep whilst maintaining business engagement is quite possibly the Holy Grail for IS delivery success.
My organisation is a few short months away from delivering the single biggest system change in its history, and at the same time as a significant organisational restructure and a focus on the delivery culture of the organisation, guess which element of this huge change programme is considered by many to be the catalyst? Yes, the delivery of the new Information System.
The delivery of such a transformational information system has been underway in several forms for a number of years, at scope gathering levels, at business development levels and now at development and implementation stages. A significant amount of effort went into the delivery of what one supplier called “the Carlsberg of requirements specifications”. Regardless of this plaudit the day after the contract for delivery was signed (with a different supplier) it became clear that a procurement ready set of business requirements now needed to become a design specification, a logical data model and a set of business story boards.
How to go about this whilst avoiding “overfitting” has become the challenge, particularly against a backdrop of organisational change and flux. Guarding the requirements whilst ensuring that what is being built is still relevant is key. Many organisations report that the answer to this problem is to build within an Agile methodology, but we were not geared up to be able to do this, we need to keep doing the day job whilst the new system is built. So our answer has been to put in place the closest possible relationship between the supplier and the organisation, but, more importantly to ensure that “the business” (a differentia I detest) is as close as possible to all elements of the delivery.
But where does that leave us, we turned down the possibility of delivering this system using Agile as we were worried about the resource intensity, but, our solution has quite probably been as resource intensive as going down the Agile route. However what we have done, so far, is mitigate any “overfitting” as the resource working with the development and delivery function owns all elements of the requirements specification and therefore can maintain their own confidence that what is being built and demonstrated at each stage is what was specified originally and continues to meet the business needs.
The guard against “overfitting” though is not only at the business resource level. It also requires a robust and sensible change management process that is managed and believed in by both sides of the contract. We achieved this for our delivery a little later than we should have, and initially we had issues caused by this, but now with our supplier leading us by the nose we have this process in place and it works well enough to facilitate a sensible conversation at all project delivery levels about the difference between a contractual change requirement and the technical elaboration of a business requirement into a coded piece of delivery.
So, “overfitting” avoided, well we hope so, only implementation and business use will actually tell. As with the concept of “overfitting” in statistics if we have not avoided it we will have a system that delivers a double whammy of issues; a system that looks good on paper only and performs worse in the real world, issues that are very difficult to recover from, but we think we have them avoided, have you?