Last week, I wrote an open letter to Ferris Research and Microsoft, in which I showed that the particular Ferris Research report being referenced by Microsoft didn't support their conclusion that "Notes systems can cost twice as much as Exchange 2007".  Well, after a number of escalations on different fronts, and a lot of finger-pointing, it turns out Microsoft was referencing the wrong Ferris report.  No wonder the story didn't add up.  The thing is, it still doesn't.

Now Microsoft refers to a different Ferris survey, from June 2008, titled "Email Support Staff Requirements and Costs".  On this one, I am a bit blind -- IBM doesn't have access to this report, so I've not read it.  It's still a report over a year old.  There were only 136 respondents, and we know from prior Ferris reports that their survey pool -- drawn from subscribers to their mailing list and webinars -- skews heavily to Exchange customers.  The link for the survey itself is still active, so we can learn a few things from it.  One, Notes/Domino 8 wasn't even an option on the list of systems, because Ferris Research began collecting data for this survey in November, 2006.  Two, it asks for information on semi-related staffing pools, such as LAN admins and help desk staff.  For Notes, the survey asked to assign a percentage of time from organizational application developers to "messaging activities", which will necessarily skew the results.  Exchange shops don't have application developers working on Exchange, for obvious reasons, and if you indicated Exchange as your mail system, you were never asked about allocating application developer staffing to messaging.  Last, the survey never asks for the fully loaded headcount costs for the staff identified -- for example if Notes professionals are paid differently than Windows "reboot monkeys".

In comments on the original Microsoft blog post, Julia White now writes:

This report is about the on-going costs of running Exchange vs Notes and this report does show that the cost of Exchange 2007 is half that of Notes 7.  I did not reference the Notes 8 costs in the report, because I consider the survey number too low and in my conversation with Ferris they agreed it should be caveated due to low response numbers.
Actually, Julia, it's not.  The report was about staffing headcount.  That is a component of the on-going costs of running Exchange vs. Notes.  It still does not support the original assertion of "Notes systems can cost twice as much as Exchange 2007" in and of itself.  And now all of Microsoft's corporate ethics are standing behind a flawed report that had 136 total self-selected responses, zero of which apply to the version of Notes that has been in market for two years.  Like the use of the pretty graphics from indeed.com, Julia White has a long way to go before she graduates from letting the statistics get in the way of the story.

Julia also chose to give Ferris Research a plug in all of this:
Despite Ed's remarks, in my experience, Ferris has demonstrated they are a knowledgeable firm with a solid grasp on the market and have always found them ethical and honest.
A few more details behind my last blog posting, where, like Julia, I defended Ferris's qualitative analysis, may be helpful.  When I first saw Julia's blog entry, I followed established IBM procedure around competitive use of analyst information.  I figured that an explanation from Ferris Research might help me understand the Microsoft citation in better context.  I don't feel comfortable publishing the e-mail verbatim since I was not directly a party to the interaction, but since the e-mail was relevant to my decision to blog, it's worth discussing.  What transpired in those emails was: 1) David indicated that he was comfortable with the Microsoft citation because of the use of the word "can"; 2) he asserted that from an e-mail/calendaring perspective, the costs of running both environments are much the same.  3) when IBM asked if IBM could publish a citation from the January, 2008 survey that Microsoft was referencing to refute the blog posting -- a report from when IBM was subscribing to Ferris research -- he asked for a fee.

What Ferris Research could have done, especially now that we know that Microsoft originally cited the wrong report, would have been to take an active role in moderating the blog dispute.  In fact, when initially contacted by IBM, David Ferris thought that it might be a mis-quote.  However, unlike other analyst firms who maintain ombudsman roles, he made no offer to have the facts corrected, or to use his own blog to represent Ferris Research's view of the situation.  Thus, I think the point of my original blog post still hold.  The analyst data was misused in this situation; the revised analyst data is still being mis-used; and the analyst themselves, who say that their perspective is different, stands by and lets it all happen.  

Post a Comment