Nick Krym of Pragmatic Outsourcing Provides a Great Example of IT Service Levels

In the world of Service Level Agreements and IT outsourcing, contractual metrics are difficult to conjure because the work is intangible, with the exception of uptime, help desk/desk side services, and application development milestones. However, metrics are the dashboard by which effective vendor managers can effectively manage outsourcing vendors.  Today, Nick Krym posted a great blog article on metrics that’s worth a look.

What makes his approach to IT outsourcing metrics noteworthy?

It begins with the clear segregation of the outsourcing vendors’ maintenance service levels from development service levels.  He recognizes up front that the internal stakeholders have different objectives and the contractual pricing is based on different outcomes.  Hence, he has two categories of service levels, not one lengthy set of service levels that measure everything or a short set of service levels that apply the same vendor management approach to each service type.  BPO vendor managers can use the same approach, as different transaction types require different service levels, but, by breaking the into clear categories, vendor performance of each transaction type can be evaluated.  For example, call center inbound customer service and call center inbound customer written correspondence.

The second sound aspect of Nick’s approach is his approach of leveraging a simple few metrics to measure the outsourcing vendors’ effectiveness.  Between 4-8 contract service levels is all that most organizations need, and he uses 4-5 in each category.  I know of one situation where the client measured 350+ metrics for an IT outsourcing engagement.  Not only was it incredibly time consuming to manage and calculate (and rebut!), but the effect of missing a single service level was essentially meaningless to the vendor.  Think about it: if you have 10% of your monthly contract value at risk, missing any of the 350 service levels on a $10M/month contract is worth exactly $2,857.14.  That’s not exactly going to get people motivated to fix things, right?  Having fewer metrics makes each metric more meaningful.

Finally, Nick uses control limits to communicate acceptable service level performance. In each chart, he uses high and low limits.  The majority of vendor managers I’ve met use a single line – the service level.  However, what most folks don’t understand is that by using high and low control limits, you can:

• Clearly communicate a range of acceptable performance.
• Understand if the vendor is over performing in some areas to offset performance in other areas.
• Display trends within the acceptable performance range to proactively manage impending failure before the vendor misses the SLA.
• Moderate expensive overperformance that the vendor need not do.  Remember – you pay for the extra quality, which may not be tangible.

Of course, I will note that Nick’s vendor is performing pretty horribly, but, as he says, he inherited the vendor through M&A activity.  This means the vendor needs an opportunity to meet expectations before terminating the relationship.  Besides this, here are some opportunities for improvement:

1. Never average daily turnaround time metrics to create a monthly metric (top left graph).  Averages distort performance.  Rather, create a monthly turnaround time based on all transactions and show daily/weekly trending using a month-to-date trend.  Then, bin items based on TAT and do root cause analysis on items that fall below TAT.  And then histogram based on counts of root cause analysis.
2. There are a few glitches.  First, the top left graph uses decimal places in the Y-axis, but the others don’t.  Second, the top right graph indicates a NEGATIVE response time between Oct 14 and Oct 21, probably because the line is formatted to be curved, a no-no in the data visualization world.  Third, consider rounding to a single decimal place, unless contract metrics require a 2nd decimal place.  Lastly, it appears that there are lots of zeros, which is probably due to volumes, so it may be important to indicate volumes using a secondary Y-axis to make this clearer.

Great job, Nick!

If you’re interested in 360° Vendor Management’s perspective on metrics, please visit our articles on Top Ten Service Level Agreement Considerations, Outsourcing Vendor Metrics, Operational Metrics, Key Performance Indicators, and Transformation Metrics.

A quick reminder: if you find this content interesting, please subscribe via Google’s automated delivery service to receive updates using the “Subscribe” field on the right side of the page or by clicking here.

Related posts:

1. Outsourcing Metrics: Key Performance Indicators
2. Transformational Metrics: Governing Outsourcing’s Lure
3. More on Outsourcing Vendor Metrics: Operational Metrics

Sustainable, successful outsourcing is all about leveraging other companies’ core competencies. Despite the labor arbitrage low cost country sourcing provides or the abundant availability skilled and unskilled labor in foreign countries, outsourcing is a fundamentally the purchase of another company’s superior service, technology, or product. Vertically integrated companies are simply non-existent. Our companies’ suppliers design products, manufacture components and finished goods, manage logistics and inventory, provide customer service, facilitate payments, and provide administrative human resources, finance, and IT support. While each new senior executive’s arrival will reopen the debate of his or her company’s core competencies (witness the CEO changes at Dell and Yahoo!), the simple fact is that no company can do everything, much less everything well.

Outsourcing provides your company the ability to obtain a level of specialization and performance it could otherwise never achieve – and with a shocking degree of immediacy. The challenge for clients who purchase these services is the transformational journey necessary to take advantage of the capabilities their suppliers provide. One of the chief gripes expressed by many vendors is that their clients fail to adopt some of the best practice processes they are capable of delivering. Vendors will offer free consulting assessments, networking and educational events with experts and other clients, and inexpensive pilot projects – almost anything to get clients to bite. As a result, clients leave quality, service and cost on the table – all elements that drive client satisfaction.

The rationale is simple. Companies who outsource typically see the capability as non-core. Others may outsource for cost considerations. Given these narrow perspectives, why would a company contemplate upgrading it’s outsourced operations?

The ramifications, however, are staggering. Just like an internal operation that lacks investment, quality, service, and cost factors plateau and begin to deteriorate. Picture the difference between call centers in 1988 and 2008. Twenty years of investment in technology, operations management, and educational standards has transformed this business process. When a company outsources its call center, the danger is that the capability it purchases today becomes obsolete in the next 4-8 years.

Oddly enough, this is the same length of the average outsourcing contract. Without transformation mandates or innovative clients, vendor operations will simple remain status quo. As the contract term completes, both parties are left with a level of dissatisfaction. Clients question why they pay so much for so little. Vendors question why their clients failed to adopt their innovations and process improvements. Negotiations begin, relationships flounder, and opportunities are lost.

Effective vendor managers manage and measure innovation in order to continuously drive improvement and deepen supplier relationships. These companies use the following key metrics:

Initial Versus Current Performance Objectives – Comparing baseline performance to post-transition performance demonstrates the initial value the vendor’s capability provides. Comparing the initial post-transition performance to current performance demonstrates the continued value the vendor’s capability expertise provides. Good vendor managers track these trends and seek opportunities to improve performance.

Initial Versus Current Return on Relationship Investment – Before a process is outsourced, the client expected a certain financial return on their capabilities’ performance. This could be measured in cost per transaction or net present value. When the process was outsourced, the client had certain financial expectations. However, good vendor managers recognize that this expectation must be constantly compared to the current costs to ensure that transformations are reflected in the financial comparisons. This isn’t to suggest costs shouldn’t go up. However, good vendors managers need to demonstrate that the vendor has continued to drive financial value for your company through transforming the operations after transition. Most importantly, sometimes this additional value may be the result of ancillary services. For example, a call center vendor may offer analytical services that make recommendations on cross-selling opportunities or customers retention strategies. The value created from these recommendations may need to be tracked independently from the initial goals of the program.

Achieved Transformational Milestones – Excellent outsourcing relationships should contain concrete, discrete, and required desired achievements. This may be as simple a detailed list of activities, tasks, or deliverables. This could be as complex as required step improvements in process performance. Regardless, these should be identified in the contract or other governance deliverable. Good vendor managers will track the both the % complete of each milestone and the timeliness of completion. Reviewing this list keeps both parties on track for desired improvements.

Customer Education Opportunities – What may seem like sleazy sales shtick, is actually an opportunity for effective vendor managers to educate themselves and their key stakeholders on opportunities. Great vendors create opportunities for their clients to learn from experts, other clients, and other sources. Vendors should be required to educate their clients as part of their services (not for extra fees), and vendor managers should record the number and types of events, numbers of hours of education, and satisfaction with these events.

Do you have other methods of measuring vendor innovation? Let us know by leaving a comment below or sending us a note.

Related posts:

1. Outsourcing Metrics: Key Performance Indicators
2. IT Outsourcing Metrics: A Good Example of Management Controls
3. Paying Outsourcing Vendors

The rationale is quite simple. Believe it or not, but outsourcing vendors have tremendous pricing discipline. They analyze Net Present Value and Internal Rate of Return on their investment in the client contract. All factors are included, such as labor, facilities, technology, telecommunications, etc. More importantly, savvy vendors develop probability models of anticipated performance against service level agreements (which is what they do with your internal performance metric data – if you give it to them). They use SOX-driven corporate guidelines when developing these models which force them to assume some degree of failure. This failure is then built into the price – essentially allowing the vendor to obtain desired financial performance even when performance fails to meet client expectations. This dramatic difference in rewards and incentives between clients and their vendors should encourage clients to drop “partnership” and “partner” from their lexicon, except for the fact that overcoming this financial gap requires true relationship building.

In a nutshell, service level penalties are insufficiently severe to drive vendor performance. That means successful vendor managers must use other levers to compel vendors to perform. One of those levers is a comprehensive key performance indicator program.

A key performance indicator program is a continuous process that focuses the vendor on a variety of leading and trailing indicators of process performance. The contractual operational metrics are a subset of these indicators – there are many other metrics and a clear process you should leverage.

Indicators – The choice of indicators you use largely depends on the business process or technology that you have outsourced. Besides the contractual operational metrics, business process outsourcing indicators should include forecasting accuracy, staffing accuracy, schedule adherence, attrition, inventory, quality calibration accuracy, loaded cost per transaction, and invoice accuracy.

KPI Achievement Index – With so many indicators to deal with, it’s hard to summarize vendor performance. A simple measure of success is an achievement index, which is calculated by dividing the number of metrics met or exceeded by the total number of metrics. There is no weighting of the indicators because it’s too complex given the number of metrics and leading indicators could cause other metrics to fail. With that said, measure the achievement index weekly or month to date, with a year to date measurement, too.. As severe as that may seem, that’s what a good operations manager should be doing, if not daily – being brilliant at the basics.

Formalize Review – In daily or weekly operations meetings, the indicators and achievement index should form the foundation of the meetings. Meeting between 1-3 times a week, even when achievement is high, should be required. Formalize the agenda to include discussion of all indicators. The discussion should not just focus on failure but also on deterioration of operational performance. Use formal root cause analysis processes. Never be lulled to sleep just because the metric is above the line. We recognize that the frequency and standing agenda may get boring over time, but excellent outsourcing executives understand that management is repetitive. It’s this repetition that drives performance.

Forced Remediation – Good outsourcing contracts give vendor managers the ability to obtain root cause analyses operational metrics, but sometimes leave out either key performance indicators and deterioration. Be sure to include language in your contract which requires the vendor to perform root cause analyses on missed or deteriorating operational metrics and key performance indicators. If your contract doesn’t contain the language, vendor managers can still manage using these metrics by explaining the need for this level of management with the vendor account management team. If they are still reluctant, escalate with a tone of “positive, collaborative focus on sustained process performance.” Eventually the vendor will agree that this level of client involvement will ensure better performance, which means increased profitability and better client satisfaction levels.

Do you use key performance indicators in your outsourcing contracts and vendor management process? Share your thoughts with us as a comment or send us a note.

Related posts:

1. Transformational Metrics: Governing Outsourcing’s Lure
2. IT Outsourcing Metrics: A Good Example of Management Controls
3. Outsourcing Vendor Metrics

Operational service levels are the core metrics of any outsourcing relationship, as they carry incentives and penalties for vendor performance. They key structural components of operational service level metrics are:

Metric Definition
All metrics must be thoroughly defined and their method of calculation precisely described to remove all ambiguities that may lead to disputes. It must address what is being measured with a high level of precision.

For example, a frequently used call center metric is call quality. A novice would define the metric as, “Vendor shall produce a 95% call quality score each month.” There are a number of problems with this definition (we’ll address the other non-definitional problems in other sections below).

First, the metric definition fails to describe in detail what a “call quality score” is. A call quality score could address a variety of elements: accuracy of an agent’s response, an agent’s use of key call greetings and security protocol, an agent’s tone, an agent’s accent, an agent’s accurate entry of notes in a system, or the presence of static or clipping in the voice quality.

Second, the metric definition fails to describe how the metric is calculated. For example, the 95% target could be calculated by averaging all agent scores or by determining the percentage of passing scores.

Third, even the metric calculation methodology of determining a “score” is unclear. Do agents start with 100 points and lose a certain number of points for each type of error? Do agents start with 0 points and gain points for completing each task and, if so, what happens if an agent is not asked to address a particular area – how do they get points? Are there items which, if done incorrectly, could cause all points to be lost (e.g., use of profanity or failure to use appropriate security protocols). Also, consider acceptable use of rounding in calculations.

Lastly, which calls can be assessed for quality? If all calls are assessed, how do you provide scoring elements for calls that are incorrectly routed to the vendor call center (e.g., wrong numbers or customers trying alternative company contact numbers to get out-of-scope issues escalated)? If a caller involuntarily abandons a call due a system or network issue, can the call be assessed a quality score?

Precision definitions require extensive use of examples and inclusion of scoring parameters. We suggest that calculations be defined and examples given. If there are complex scoring guidelines, they should specifically attached in an exhibit or appendix where they can be appropriately managed in the case of changes.

Performance Objectives
Precise goals of metrics are essential for all parties to know the range of acceptable performance. In situations where bands of performance trigger penalties or incentives, precision is essential. The example given above fails to address this issue. In particular, it fails to identify if 95% is the only objective…and if 98% is acceptable!

Equally important is defining rounding parameters, as few metrics can be represented solely as integers. Is 94.5% a passing score? 94.49%? 94.445?

Also, remember to not leave gaps in ranges. For example, “Meets Expectations: 95.0%-97.0%; Exceeds Expectations: 98.0%-100.0%” leaves an undefined gap between 97.0% and 98.0%. This could be the difference between paying an incentive or not…

Measurement Methodology
Depending on the metric, different measurement methodologies must be used. In particular, the use of sampling is one of the most complex numerical issues vendor managers come across. Frankly, this is an area that many vendor managers lack the skills to understanding, but it is an area that can be easily taught by experienced professionals.

First, effective vendors managers must know when to use sampling and when not to use sampling. For example, sampling turnaround time, backlog or inventory metrics is a very poor idea because performance parameters can be dramatically skewed and random samples may not identify these issues. Never sample volume-sensitive metrics. Use sampling when assessing quality scores.

Second, vendor managers must fully understand that samples sizes and populations are not related. In fact, the calculation of sample sizes (with the exception of small populations) do not include a variable for population size. Instead, they focus on allowable error, desired confidence intervals, and performance variability. Small populations simply use a slightly different calculation. Vendors, vendor managers, and the customers of vendor managers frequently fail to understand this important calculation. In fact, we frequently hear vendors complain that “the sample was so small that one dumb mistake can cause us to miss the goal.” These vendors fail to understand that the mistake can be extrapolated to exist in much greater numbers in the total population. It is important the sampling calculation and acceptable variables in the calculation be defined in the contract.

Another factor vendor managers must account for is acceptable error ranges and the impact on high performance goals. An example is the goal of “99.5% accuracy” which is calculated using a sample that allows an error of +/- 3%. The level of precision in the sample is too wide to be acceptably used in this level of quality of expectation. Rather than tightening the acceptable error range in the sampling calculation, vendor managers are better off adopting a different goal. In particular, move to 6 sigma objectives which is a better metric for goals with high performance expectations.

Rebuttal Process, Roles, and Responsibilities
No matter how precisely you define vendor metrics, there will always be gray areas that are undefined. If you haven’t read our article on calibrating quality expectations, please do – as calibrations improve the mutual understanding of quality expectations.

While effective calibration processes decrease the number of items in a rebuttal process (especially over time as quality expectations are more precisely refined as a result of the rebuttal discussions), you must still determine a methodology of handling rebuttals. If left undefined, contracts will normally force unresolved issues into formal dispute resolution procedures (e.g., arbitration or law suits), which is frankly like using a nuclear weapon to remove the paint from your house. In fact, defined rebuttal processes should contain language that prevent rebuttal processes from entering into dispute resolution by giving the vendor management executive sole discretion regarding quality standards.

Rebuttal process definitions should describe the frequency, timing, and required participation of rebuttal meetings. They should be completed in time to allow the vendor to accurately invoice on time, they should occur with sufficient frequency to prevent backlogs of rebuttal items that could expose the vendors to additional risk until the rebutted item is resolved. Rebuttal processes should also describe necessary escalation to outsourcing governance executives.

Defined Excuses for Non-Performance
A vendor can fail to meet performance objectives for a wide variety of reasons. These could include the fault of the vendor, the vendor’s third party suppliers (e.g., telecommunications), the client’s systems, high transaction volumes, force majeure events, and events not included in force majeure clauses but have no obvious owner (e.g., Indian taxi or Philippine jeepney driver strikes). With as much detail as possible, a contract should define acceptable rationale for a vendor to not meet vendor service level performance expectations.

This rationale should be as narrow as possible, too. If there was a typhoon on Tuesday, the vendor shouldn’t be allowed to get off the hook for quality results on that day because quality has nothing to do with weather. However, the vendor could be excused for service levels that day. What is important, however, that the excuse not be so broad as to not hold the vendor accountable for a long term period. While a typhoon could cause an inventory backlog over a few days, failing to create an incentive for the vendor to implement its business continuity plans and work-down the aged inventory in a reasonable period will give the vendor an excuse to explain away other performance problems that are unrelated, but hard for vendor managers to detect.

If a major media event occurs which drives unexpected volumes, a vendor may not be able to meet turnaround times. It’s important to define acceptable ranges of volume spikes.

Do you have other ideas on measuring operational metrics? Let us know!

Update: We’ve posted another another article in this metrics collection: Key Performance Indicators.

Related posts:

1. IT Outsourcing Metrics: A Good Example of Management Controls
2. Outsourcing Vendor Metrics
3. Vendor Inventory Metrics

In short, there are operational service level metrics, key performance indicators, and transformational metrics.

Operational Service Level Metrics – These are the core metrics of any outsourcing relationship. These metrics are described in detail in the contract and frequently are tied to penalties and incentives for vendor performance. Typical metrics include quality, timeliness, inventory, and satisfaction. These metrics typically measure operational performance of a business process.

Key Performance Indicators (KPIs) – KPIs can be described in the contract or in the business requirements. They are typically very detailed metrics that are necessary for managing a vendor relationship. Examples include schedule adherence, staffing accuracy, and training completion rate. KPIs are reported by the vendor, but do not have contractual penalties or incentives.

Transformational Metrics – Transformational metrics describe overall business outcome goals. They differ from KPIs in that they provide contractual incentives for performance, but also differ from operational service level metrics in that they indicate end-state objectives. Examples include project milestones, process performance objectives, and implementation goals.

The trick for vendor managers is to keep the big picture of in mind when building a balanced scorecard. Many vendor managers, after coming from deep operational mindsets, focus too keenly on KPIs and rarely give praise for operational service level achievements. They completely ignore transformational goals. Project managers who become vendor managers, ignore KPIs, loosely monitor SLA performance, and focus keenly on transformational metrics. Process improvement experts who become vendor managers focus on SLAs, but forget all other metrics…

Successful vendor managers understand that KPIs are leading indicators of operational service level metric performance – and track them rigorously to predict SLA achievement. SLAs are trended to indicate operational performance and consistency. Transformation metrics are reviewed regularly and never ignored. As you can see, good vendor managers use all available data, but realize that they cannot hammer vendors for failure to achieve KPIs. Rather, good vendor managers use KPIs to ensure vendors react early to negative trends and ensure vendors perform.

A good resource to use is Dashboardspy, which provide many, many examples of good metrics and dashboards. Another good resource is Stephen Few’s Perceptual Edge.

How do you use metrics to manage outsourcing vendor performance? Let us know!

Update: We’ve published a more detailed analysis on operational vendor metrics. Please be sure to read it for additional tips on structuring service level agreements and metrics.

Related posts:

1. Outsourcing Metrics: Key Performance Indicators
2. More on Outsourcing Vendor Metrics: Operational Metrics
3. Vendor Inventory Metrics

More outsourcing contracts should address inventory metrics.  If nothing else, addressing this dimension of back office operations gives a vendor manager visibility.  With penalties on the line, vendors will be more likely to consistently meet timeliness metrics (which are more important) because inventories will be better managed.  Finally, managing inventories effectively are essential to creating a positive customer experience.

We wouldn’t suggest that inventory metrics  should be weighted the same as quality and timeliness, but they deserve a seat at the table.  Good examples inventory metrics are average inventory age, days work on hand, and percent of inventory older than 14 days.

Related posts:

1. Week in Review: Scheduling, Forecasting, Requirements, and Inventory
2. More on Outsourcing Vendor Metrics: Operational Metrics
3. Outsourcing Vendor Metrics

1. MECE – SLAs should be Mutually Exclusive and Completely Exhaustive (MECE). By this, no two SLAs should measure the same thing, and there should be a SLA for every important aspect of the program. Too often we see SLAs that overlap, creating double jeopardy situations and misleading positive or negative performance reports. In addition, we often see important areas that are left unmeasured due to oversight.
2. Defined – Service levels must be thoroughly defined. For example, abandonment rate within a call center ought to be defined so that parties know whether blocked calls (trunk blockage), calls that terminate in the IVR, calls that terminate in the first 5 seconds (we don’t agree with this, but someone people do), or calls that answered by agents but are the “wrong number” are considered abandoned calls. Check this Q&A section out if you need some idea of how varied definitions are. If you don’t define the service level, you don’t know what you’re measuring.
3. Calculated – Even though you spend time defining service levels, you need to define any calculations. For example, if the service level is abandonment rate, the calculation should defined as “the number of Abandoned Calls divided by the total number of Offered Calls”. Where we use capitalized terms, these are predefined earlier in the SLA section. Vendors and vendor managers should never, ever be surprised by calculations. Give examples in your written SLAs.
4. Measurable – Don’t waste anyone’s time with SLAs that can’t be measured. For example, if you’re measuring customer satisfaction in a call center environment (typically done via a 3rd party after the primary call is concluded), but you don’t have the means to measure customer satisfaction (e.g., your call center doesn’t have the ability to use 3rd parties or your agents aren’t trained to collect satisfaction data), its a waste of time. Typically, angry customers will whip out unmeasurable SLAs and argue that vendors failed to achieve them, which is a huge waste of effort.
5. Easily Measurable – It’s one thing to measure something, its another thing to spend oodles of dollars to measure the same thing. If the cost of measurement doesn’t warrant the benefits of the SLA, don’t use it. The best example of this was a measure where customers call to complain about a bad transaction they’ve recently received in the mail. The vendor didn’t manage the call center, just the backoffice transactions. So, the call center needed to take notes on bad transactions and track them in a manner that allowed auditors to identify if the vendor was the cause of the bad transaction or not. Since the vendor only handled one of six steps in the backoffice process and the mainframe systems didn’t track transaction history, it was impossible to determine who caused the error without significant system modifications.
6. Time Frames – SLAs should cover a specific time period. Daily, weekly, monthly, quarterly, annually, etc. They should also only be assessed once. The example provided in #4 is a bad example of this, since customers could call to complain months after the transaction was completed, making it difficult to understand when to assess a month’s quality number. Essentially, the vendor would be in jeopardy forever, since a customer could complain at any time about a month – and every complaint would only lower the quality score, until the vendor had to pay penalties.
7. Singled Barrels – A SLA should contain only one measure, not two, three, or even four measures. If you’re SLA is “99% of transactions must meet quality standards and achieve customer satisfaction requirements” you need to track both conditions, which is nightmarishly difficult. In questionaire terminology, these are called double-barreled situations, and typically provide misleading or inaccurate pictures of operational performance.
8. Serve a Purpose – In some contracts, a minimum number of SLAs are required (to reduce the vendor’s risk, of course). That’s great when you need 3 or 4 SLAs, but what if you only need two SLAs and are therefore required to make-up another one or two to meet the contractual guidelines? These typically become “gimmes” and are a waste of time. Every SLA should serve a purpose.
9. Actionable – Every SLA should be capable of being influenced through actions of the vendor or the company. If the SLA can’t be influenced, don’t bother. A bad example may be measures of employee satisfaction with compensation in a HR outsourcing relationship, where the vendor has no control over compensation or employee /supervisor communication/training program. Since all employees will naturally dislike their compensation to some degree, the vendor has very little ability to create positive results.
10. Realistic – Look, we all want to be perfect, but those who belong to the cult of zero defects don’t understand contracting and real-life BPO and ITO. Achieving 100% of anything is simply unrealistic in most situations. Your goals can be aggressive (or evenly progressively more aggressive over time), but they should achievable.

We want to hear your comments! Let us know what you’re thinking by commenting below.

Related posts:

1. Outsourcing Service Level Agreements: The Monthly Close
2. Another Tale from “When You Don’t Have Vendor Management Governance”