Myth #4: It Is Fair To Assume That Acceptable Credentials Are An Indication Of Future Success – Part #3 (Lesson 16)
Case studies
Case studies are useful to the extent that you can be certain that the supplier has completed a number of projects before, to some degree of success.
I always find it interesting to look at how many case studies a supplier has. Sometimes there are hundreds of case studies available on the supplier’s website, but somehow, only one or two are even remotely relevant.
Quantity is no substitute for relevant quality.
Consider as well the age of the case study. How long ago was this project actually completed? What version of the software does it relate to? From a technical perspective, what about the versions of server operating systems, databases, application servers, etc.? Does your supplier have experience that remains current?
One final caution regarding lists of case studies that detail successful implementations at other organisations. Make sure that you scratch below the surface to understand the exact scope of services and products that were implemented as part of that project.
The case study may reflect the scope of a pilot or initial launch of limited scope a long time ago.
Remember that case studies – whilst demonstrating some experience – are essentially marketing devices. Have you ever seen a case study that tells you how the supplier did a bad job?
In the same way that an individual’s résumé will always tell you how wonderful they are, so will a case study.
Case studies will always remain a useful and desirable part of your due diligence, but you need to ensure that you understand exactly what a case study is really telling you – and what it is not.
Reference site visits and reference calls
You know that you will never see a bad case study, but how well do reference calls and reference visits stand up to close scrutiny?
Whilst a case study is essentially a clinically constructed marketing device, a reference call or reference visit is ostensibly less controlled. Because it is more under your control, it is therefore more likely to give you the results that you want.
Once again though, you have to ask the question: will a reference call or reference visit ever yield a bad reference?
It is fairly certain that the reference site has been carefully selected and briefed, so the reference given will not be bad. In fact, a reference site visit could well be hosted by the supplier, who may guide you around between the various contacts at that site, carefully stage-managing the exercise at all times.
If you want to see just how open your supplier is, ask for a list of references where they had to overcome problems either with the product or the way the services were delivered.
Maintaining reference sites is not easy. The supplier’s client is being asked to repeatedly give up time to help their supplier’s sales process along.
Suppliers do not have an infinite number of reference sites available.
Indeed, if they do not manage the process well, the few relevant reference sites that they do have quickly start to suffer from ‘reference fatigue’ – being asked too many times.
Few available referees could also be a red flag. However, this may actually work to your advantage. A client who is frustrated with hosting so many reference calls or visits may become more candid than they otherwise might have been.
Additionally, whether you conduct reference calls or reference site visits largely depends on the supplier and the cooperation of their clients. The result is that sometimes, the reference site is not very well aligned to your line of business.
Wherever possible, onsite reference visits would be my preference. Looking into people’s eyes and seeing front-line staff actually using and talking about the system will always yield more information than a 15-45-minute telephone call.
If you can only secure a reference call, try to ensure that you do it via video conference, using a free service such as Zoom or Skype, or your own corporate solution.