360° Vendor Management

More on Outsourcing Vendor Metrics: Operational Metrics

With increasingly complex vendor management programs and more interest in developing outsourcing and services relationships, it is no surprise that measuring vendor performance is an increasingly important topic on vendor management executives’ agendas. We previously wrote a very popular article on Outsourcing Vendor Metrics. Our readers have provided plenty of positive feedback via email. Overwhelmingly, vendor managers have asked for more detail and examples of each category of metric to improve their service level agreements. In this article, which is part one of several upcoming articles, we delve deeper into detailed operational service level metrics outsourcing executives can use to manage their projects effectively

Operational service levels are the core metrics of any outsourcing relationship, as they carry incentives and penalties for vendor performance. They key structural components of operational service level metrics are:

Metric Definition
All metrics must be thoroughly defined and their method of calculation precisely described to remove all ambiguities that may lead to disputes. It must address what is being measured with a high level of precision.

For example, a frequently used call center metric is call quality. A novice would define the metric as, “Vendor shall produce a 95% call quality score each month.” There are a number of problems with this definition (we’ll address the other non-definitional problems in other sections below).

First, the metric definition fails to describe in detail what a “call quality score” is. A call quality score could address a variety of elements: accuracy of an agent’s response, an agent’s use of key call greetings and security protocol, an agent’s tone, an agent’s accent, an agent’s accurate entry of notes in a system, or the presence of static or clipping in the voice quality.

Second, the metric definition fails to describe how the metric is calculated. For example, the 95% target could be calculated by averaging all agent scores or by determining the percentage of passing scores.

Third, even the metric calculation methodology of determining a “score” is unclear. Do agents start with 100 points and lose a certain number of points for each type of error? Do agents start with 0 points and gain points for completing each task and, if so, what happens if an agent is not asked to address a particular area - how do they get points? Are there items which, if done incorrectly, could cause all points to be lost (e.g., use of profanity or failure to use appropriate security protocols). Also, consider acceptable use of rounding in calculations.

Lastly, which calls can be assessed for quality? If all calls are assessed, how do you provide scoring elements for calls that are incorrectly routed to the vendor call center (e.g., wrong numbers or customers trying alternative company contact numbers to get out-of-scope issues escalated)? If a caller involuntarily abandons a call due a system or network issue, can the call be assessed a quality score?

Precision definitions require extensive use of examples and inclusion of scoring parameters. We suggest that calculations be defined and examples given. If there are complex scoring guidelines, they should specifically attached in an exhibit or appendix where they can be appropriately managed in the case of changes.

Performance Objectives
Precise goals of metrics are essential for all parties to know the range of acceptable performance. In situations where bands of performance trigger penalties or incentives, precision is essential. The example given above fails to address this issue. In particular, it fails to identify if 95% is the only objective…and if 98% is acceptable!

Equally important is defining rounding parameters, as few metrics can be represented solely as integers. Is 94.5% a passing score? 94.49%? 94.445?

Also, remember to not leave gaps in ranges. For example, “Meets Expectations: 95.0%-97.0%; Exceeds Expectations: 98.0%-100.0%” leaves an undefined gap between 97.0% and 98.0%. This could be the difference between paying an incentive or not…

Measurement Methodology
Depending on the metric, different measurement methodologies must be used. In particular, the use of sampling is one of the most complex numerical issues vendor managers come across. Frankly, this is an area that many vendor managers lack the skills to understanding, but it is an area that can be easily taught by experienced professionals.

First, effective vendors managers must know when to use sampling and when not to use sampling. For example, sampling turnaround time, backlog or inventory metrics is a very poor idea because performance parameters can be dramatically skewed and random samples may not identify these issues. Never sample volume-sensitive metrics. Use sampling when assessing quality scores.

Second, vendor managers must fully understand that samples sizes and populations are not related. In fact, the calculation of sample sizes (with the exception of small populations) do not include a variable for population size. Instead, they focus on allowable error, desired confidence intervals, and performance variability. Small populations simply use a slightly different calculation. Vendors, vendor managers, and the customers of vendor managers frequently fail to understand this important calculation. In fact, we frequently hear vendors complain that “the sample was so small that one dumb mistake can cause us to miss the goal.” These vendors fail to understand that the mistake can be extrapolated to exist in much greater numbers in the total population. It is important the sampling calculation and acceptable variables in the calculation be defined in the contract.

Another factor vendor managers must account for is acceptable error ranges and the impact on high performance goals. An example is the goal of “99.5% accuracy” which is calculated using a sample that allows an error of +/- 3%. The level of precision in the sample is too wide to be acceptably used in this level of quality of expectation. Rather than tightening the acceptable error range in the sampling calculation, vendor managers are better off adopting a different goal. In particular, move to 6 sigma objectives which is a better metric for goals with high performance expectations.

Rebuttal Process, Roles, and Responsibilities
No matter how precisely you define vendor metrics, there will always be gray areas that are undefined. If you haven’t read our article on calibrating quality expectations, please do - as calibrations improve the mutual understanding of quality expectations.

While effective calibration processes decrease the number of items in a rebuttal process (especially over time as quality expectations are more precisely refined as a result of the rebuttal discussions), you must still determine a methodology of handling rebuttals. If left undefined, contracts will normally force unresolved issues into formal dispute resolution procedures (e.g., arbitration or law suits), which is frankly like using a nuclear weapon to remove the paint from your house. In fact, defined rebuttal processes should contain language that prevent rebuttal processes from entering into dispute resolution by giving the vendor management executive sole discretion regarding quality standards.

Rebuttal process definitions should describe the frequency, timing, and required participation of rebuttal meetings. They should be completed in time to allow the vendor to accurately invoice on time, they should occur with sufficient frequency to prevent backlogs of rebuttal items that could expose the vendors to additional risk until the rebutted item is resolved. Rebuttal processes should also describe necessary escalation to outsourcing governance executives.

Defined Excuses for Non-Performance
A vendor can fail to meet performance objectives for a wide variety of reasons. These could include the fault of the vendor, the vendor’s third party suppliers (e.g., telecommunications), the client’s systems, high transaction volumes, force majeure events, and events not included in force majeure clauses but have no obvious owner (e.g., Indian taxi or Philippine jeepney driver strikes). With as much detail as possible, a contract should define acceptable rationale for a vendor to not meet vendor service level performance expectations.

This rationale should be as narrow as possible, too. If there was a typhoon on Tuesday, the vendor shouldn’t be allowed to get off the hook for quality results on that day because quality has nothing to do with weather. However, the vendor could be excused for service levels that day. What is important, however, that the excuse not be so broad as to not hold the vendor accountable for a long term period. While a typhoon could cause an inventory backlog over a few days, failing to create an incentive for the vendor to implement its business continuity plans and work-down the aged inventory in a reasonable period will give the vendor an excuse to explain away other performance problems that are unrelated, but hard for vendor managers to detect.

If a major media event occurs which drives unexpected volumes, a vendor may not be able to meet turnaround times. It’s important to define acceptable ranges of volume spikes.

Do you have other ideas on measuring operational metrics? Let us know!

Update: We’ve posted another another article in this metrics collection: Key Performance Indicators.

This entry was posted on Tuesday, February 26th, 2008 at 8:29 am and is filed under Metrics, Outsourcing, Outsourcing SLAs, Outsourcing Vendor Management, Vendor Management. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.