The Black Book of Outsourcing – Invaluable Resource or Red Herring?

During this short week, I took the opportunity to take a hard look at the 2009 Black Book of Outsourcing, the source of innumerable vendor press releases touting so-called market leadership based on rankings provided by Brown and Wilson.  In Scott Wilson’s own words, “The Black Book is known as the leading independent unbiased ranking of vendors as it surveys qualified users and is externally audited.”  Okay, but is it worth anything?  Let’s delve deeper…

About the Publisher

In 2005, Brown-Wilson penned the Black Book of Outsourcing, which served as a solid introduction to outsourcing for buyers.  At the time, it was one of the few published resources that touched on a wide variety of issues, including outsourcing strategy, vendor selection, contract negotiation, and vendor management.  It offered no vendor rankings, but they were made available online.  Douglas Brown and Scott Wilson know what they are talking about, so the book was well written, although it offered few resources for buyers to leverage, such as templates.

Then, in 2006 Brown-Wilson issued their first rankings. It was a bewildering group of 90+ sub lists of “Top 10 Vendors” in categories ranging from Identity Management and Access Controls to Investor Relations and Outplacement Services to Mining Engineering Firms.  All made publicly available.  Besides it appearing like a high school reunion-like method to find an award for every vendor, it became a vendor list for RFPs, used by analysts, advisors, consults, and buyers. However, it was difficult to use because of the number of sublists and the clear issue that not all subcategories made good sense.  For example, there was a bias towards India and no inclusion of Canada, the Philippines or Costa Rica – all very hot outsourcing locations at the time of publication.

Brown-Wilson must have seen the usability problem because in 2007 they simplified things when they published their first “Top 50″ vendor list.  They also included 11 clearly understandable categories that helped organize a baffling 100+ sub “Top 10″ categories of outsourcing vendors which included Agribusiness, Process Consulting, pay-as-you-go call centers, and Outsourcing and Offshoring Research Vendors (which did not include Brown-Wilson).  In an even more bizarre move, Brown-Wilson also rated outsourcing advisors and consultants by creating 15 different categories of “Top 10″ lists (as if this niche needed sub-niches).  Finally, Brown-Wilson named the top 50 green outsourcing companies.  As if non-performance, governance, and proper requirements definition wasn’t enough, Brown-Wilson suggested that companies were actually “dumping” outsourcing vendors based on their efforts to clean up the environment.  (I won’t go into the seemingly strange combination of carbon credits and emissions trading where companies really outsourced their environmental requirements and then terminated vendors for failing to meet environmental standards.)

In 2008, Brown-Wilson kept the format the same.  An overall Top 50 vendor list, and two subcategories, advisors and vendors, that were broken into seven (instead of 15) and 90+ (instead of 100+) subcategories, respectively.  The same “Green” top 50 category was reported, too.

Then, in April 2009, Brown-Wilson sold their business to Datamonitor (which had acquired a whole host of other companies, as reported by SageCircle).  In the 2009 results, the same Top 50 Vendor list was published, along with a single category of “top 10″ advisors, and just 50 subcategories of vendors were published.  Of course, now Datamonitor sells specific reports
based on the data collected for thousands of dollars per report.  So, it better be worth something, right?

Analysis of the Results of The Black Book of Outsourcing

Now, let’s look at how the results changed from year to year to see if the results are reliable.

A quick look at a few intra-categories’ differences between 2006 and 2007 results shows some really big changes:

• Health insurance/payors outsourcing vendors:  Only four vendors appeared in both Top 10 lists.  EDS remained last, despite six new vendors entering the list.  TriZetto dropped from first to ninth.  ACS, who didn’t even appear on the 2006 list, beat out all comers and took the top ranking.
• Overall BPO: Only three vendors appeared in both Top 10 Lists.  Accenture moved from first to last.  Cap Gemini moved from ninth to third.  WNS Global, who also didn’t appear in the 2006 list, beat out all comers and took the top ranking.
• Enterprise Application Solutions Vendors: Only 5 vendors appeared on both lists.  Satyam took the top spot in both years, but #2 and #3 in 2007 were new vendors to the list and the 2006 #9 (HP) and #10 (EDS) moved into 4th and 5th, respectively.

Those are some pretty major changes.  It’s like the New England Patriots dropping from super bowl contention to middle-of-the-pack performers.  However, unlike in American football, outsourcing vendors sign clients for multiple year contracts.  Why is there such variation?

The Top 50 Lists between 2007-2009 show significant variation:

• Just 16 vendors were rated in top 50 all three years (17 if you include EDS after HP’s acquisition).
• The 2009 top 20 vendors includes just nine vendors in the 2008 top 20.  Only seven of the 2009 top 20 vendors were in the 2007 top 20.  Only three 2009 top 20 vendors were listed in the top 20 in all three years.
• Not a single vendor in the 2009 top 5 vendors had previously been in the top 5.  Only one had previously been in the top  10.
• Xerox, ranked #3 in 2009, had never been previously ranked.

Why is there such variation?

The Problem

Nothing Brown and Wilson do in their survey methodology cause some vendors to appear high while other vendors to appear low – it’s really the customers who cause the ratings to be high or low.

However, Brown and Wilson’s presentation of the data is awkward.  Some examples:

1. The Top 50 Compares Apples to Oranges – How can Accenture, IBM, and Genpact be compared to Xerox?  Xerox is a one-dimensional vendor focused on managing printers and outsourced print.  While one could certainly try to make arguments about the best athlete and compare across different sports, how could you compare Michael Phelps, Tiger Woods, and Michael Jordan?  Their stats are hard to compare across categories.  This causes great variation in interpretation.
2. Non-Standard Scoring Methodology Skews Results – Brown and Wilson made a poor choice when they decided to use an 11-point scale.  Likert and semantic differential questions always use an odd number, and very, very rarely use numbers like 0 and 10 (because of their emotional value) and seldom exceed a scale of 9.  The reason is that respondents have a hard time differentiating between the numbers.  For example, Strongly Agree, Agree, Neutral, Disagree, Strongly Disagree or Strongly Agree, Agree, Somewhat Agree, Neutral, Somewhat Disagree, Disagree, and Strongly Disagree.  Could someone please attach some verbal definitions between a 7 and 8 on Brown and Wilson’s 11-point scale?  Actually, the did it for you, and this is the shocking part: Satisfactory Performance is defined as a 7 AND a 8 on their scale.  So, they provide seven levels of dissatisfaction (0-6), two levels of satisfaction (7-8), and two levels of overwhelming satisfaction (9-10).  Then they average the scores, which makes the differences among vendors harder to differentiate.  As a result, one should expect the results at the top 50 vendors to be very tightly bunched together as a result, as they represent just 1% of the total vendors evaluated!  As a result, there are probably very, very small differences among the top 50 scores, making them highly susceptible to margin of error.
3. All Factors are Treated Equally – Brown and Wilson measure 31 different factors (they call them KPIs) in the most recent survey.  These factors are all treated equally in calculations, but few respondents likely feel they are equal.  For example, “Community Obligations and Stewardship” is weighted equally with “Results-Oriented Performance Culture.”  Furthermore, some of the factors are tablestakes, like “Lawful Conduct and Company Culture” and “Ethical Financial Leadership.”  If a company doesn’t do these things well, they are shut down by the government, regulators, or the courts.  These are “yes” and “no” type questions, but Brown and Wilson force them to be scored of a 11-point scale, too.  The net result here that truly important and differentiating factors are muted by less important or tablestakes questions.  Again, this causes respondents’ scores to be less valid and reliable.
4. How Much Do Vendor Managers Really Know – In certain total-outsource industries (like Payroll and AP), how much do the vendor managers really know?  For example, do you know how involved your payroll vendor is in the community or how much diversity they create?  With over 715,000  invitations to participate, how many of those vendor managers know the vendor well enough to differentiate across a 11-point scale on rarely encountered factors, including “Value, Mission, and Message Communication” and “Ethical Human Resources Leadership.”  How reliable are respondent’s scores in these areas?

Conclusion and Recommendations

Brown and Wilson have admirably attempted a nearly impossible task: asking vendor managers with different seniority, experience, and industries to compare almost 5,000 vendors who belong to one or more different outsourcing niches.  Kudos for the attempt.  However, their survey design methodology is deeply flawed and results in reliability and validity errors.  The result is that potential and current customers’ perspectives are deeply affected by these types of lists, as they generally form the starting point for vendor selection lists.  They also create false perspectives among the vendors.

However, here are my recommendations regarding the use of the top vendor lists:

1. Don’t believe the vendor hype regarding position in the results.  Scoring is not reliable or valid.
2. Don’t rely solely on the top vendors in each category – go deeper into the list to find the gems that may be 10 places below, but have a margin of difference among scores of less than 2%.
3. Use the factors Brown and Wilson ask respondents to evaluate to drive vendor selection, alignment and governance activities

Finally, here’s the recommendations for Datamonitor to consider:

1. Get scientific and use solid, proven survey design to drive value in the Black Book of Outsourcing
2. Consider using the Analytical Hierarchy Process to compare vendors’ performance.
3. Consider using an “importance” and “effectiveness” methodology, that asks respondents to rate both for each factor.  That would provide valuable feedback to customers and vendors, as well as create a basis for weighting factors differently.
4. Don’t allow multiple responses from one company.  In other words, don’t let Kraft submit 50 responses for one vendor.  Ask them to consolidate their responses as a corporate response to get better feedback from all levels in the organization.
5. Reevaluate some of the factors scored and remove tablestake factors, like lawful and ethical HR actions.

The book is great, and should be required reading for any outsourcing professional, buy it here:

Related posts:

1. The Business of Outsourcing Certifications
2. Offshore Outsourcing RFPs: Are They Encouraging Copy and Paste?
3. Offshore Outsourcing Vendor Governance Organizations
4. IT Outsourcing Metrics: A Good Example of Management Controls
5. Offshore Outsourcing Management – What’s the Problem?