Nick Krym of Pragmatic Outsourcing Provides a Great Example of IT Service Levels

In the world of Service Level Agreements and IT outsourcing, contractual metrics are difficult to conjure because the work is intangible, with the exception of uptime, help desk/desk side services, and application development milestones. However, metrics are the dashboard by which effective vendor managers can effectively manage outsourcing vendors.  Today, Nick Krym posted a great blog article on metrics that’s worth a look.

What makes his approach to IT outsourcing metrics noteworthy?

It begins with the clear segregation of the outsourcing vendors’ maintenance service levels from development service levels.  He recognizes up front that the internal stakeholders have different objectives and the contractual pricing is based on different outcomes.  Hence, he has two categories of service levels, not one lengthy set of service levels that measure everything or a short set of service levels that apply the same vendor management approach to each service type.  BPO vendor managers can use the same approach, as different transaction types require different service levels, but, by breaking the into clear categories, vendor performance of each transaction type can be evaluated.  For example, call center inbound customer service and call center inbound customer written correspondence.

The second sound aspect of Nick’s approach is his approach of leveraging a simple few metrics to measure the outsourcing vendors’ effectiveness.  Between 4-8 contract service levels is all that most organizations need, and he uses 4-5 in each category.  I know of one situation where the client measured 350+ metrics for an IT outsourcing engagement.  Not only was it incredibly time consuming to manage and calculate (and rebut!), but the effect of missing a single service level was essentially meaningless to the vendor.  Think about it: if you have 10% of your monthly contract value at risk, missing any of the 350 service levels on a $10M/month contract is worth exactly $2,857.14.  That’s not exactly going to get people motivated to fix things, right?  Having fewer metrics makes each metric more meaningful.

Finally, Nick uses control limits to communicate acceptable service level performance. In each chart, he uses high and low limits.  The majority of vendor managers I’ve met use a single line – the service level.  However, what most folks don’t understand is that by using high and low control limits, you can:

• Clearly communicate a range of acceptable performance.
• Understand if the vendor is over performing in some areas to offset performance in other areas.
• Display trends within the acceptable performance range to proactively manage impending failure before the vendor misses the SLA.
• Moderate expensive overperformance that the vendor need not do.  Remember – you pay for the extra quality, which may not be tangible.

Of course, I will note that Nick’s vendor is performing pretty horribly, but, as he says, he inherited the vendor through M&A activity.  This means the vendor needs an opportunity to meet expectations before terminating the relationship.  Besides this, here are some opportunities for improvement:

1. Never average daily turnaround time metrics to create a monthly metric (top left graph).  Averages distort performance.  Rather, create a monthly turnaround time based on all transactions and show daily/weekly trending using a month-to-date trend.  Then, bin items based on TAT and do root cause analysis on items that fall below TAT.  And then histogram based on counts of root cause analysis.
2. There are a few glitches.  First, the top left graph uses decimal places in the Y-axis, but the others don’t.  Second, the top right graph indicates a NEGATIVE response time between Oct 14 and Oct 21, probably because the line is formatted to be curved, a no-no in the data visualization world.  Third, consider rounding to a single decimal place, unless contract metrics require a 2nd decimal place.  Lastly, it appears that there are lots of zeros, which is probably due to volumes, so it may be important to indicate volumes using a secondary Y-axis to make this clearer.

Great job, Nick!

If you’re interested in 360° Vendor Management’s perspective on metrics, please visit our articles on Top Ten Service Level Agreement Considerations, Outsourcing Vendor Metrics, Operational Metrics, Key Performance Indicators, and Transformation Metrics.

A quick reminder: if you find this content interesting, please subscribe via Google’s automated delivery service to receive updates using the “Subscribe” field on the right side of the page or by clicking here.

Related posts:

1. Offshore Outsourcing Vendors, Customers, and Advisors: They Should Know Better
2. Vendor Management Organizations Are a Bad Design
3. Offshore Outsourcing Management – What’s the Problem?

More outsourcing contracts should address inventory metrics.  If nothing else, addressing this dimension of back office operations gives a vendor manager visibility.  With penalties on the line, vendors will be more likely to consistently meet timeliness metrics (which are more important) because inventories will be better managed.  Finally, managing inventories effectively are essential to creating a positive customer experience.

We wouldn’t suggest that inventory metrics  should be weighted the same as quality and timeliness, but they deserve a seat at the table.  Good examples inventory metrics are average inventory age, days work on hand, and percent of inventory older than 14 days.

Related posts:

1. Week in Review: Scheduling, Forecasting, Requirements, and Inventory
2. Scheduling: One Reason Outsourcing Deals Fail
3. IT Outsourcing Metrics: A Good Example of Management Controls

1. MECE – SLAs should be Mutually Exclusive and Completely Exhaustive (MECE). By this, no two SLAs should measure the same thing, and there should be a SLA for every important aspect of the program. Too often we see SLAs that overlap, creating double jeopardy situations and misleading positive or negative performance reports. In addition, we often see important areas that are left unmeasured due to oversight.
2. Defined – Service levels must be thoroughly defined. For example, abandonment rate within a call center ought to be defined so that parties know whether blocked calls (trunk blockage), calls that terminate in the IVR, calls that terminate in the first 5 seconds (we don’t agree with this, but someone people do), or calls that answered by agents but are the “wrong number” are considered abandoned calls. Check this Q&A section out if you need some idea of how varied definitions are. If you don’t define the service level, you don’t know what you’re measuring.
3. Calculated – Even though you spend time defining service levels, you need to define any calculations. For example, if the service level is abandonment rate, the calculation should defined as “the number of Abandoned Calls divided by the total number of Offered Calls”. Where we use capitalized terms, these are predefined earlier in the SLA section. Vendors and vendor managers should never, ever be surprised by calculations. Give examples in your written SLAs.
4. Measurable – Don’t waste anyone’s time with SLAs that can’t be measured. For example, if you’re measuring customer satisfaction in a call center environment (typically done via a 3rd party after the primary call is concluded), but you don’t have the means to measure customer satisfaction (e.g., your call center doesn’t have the ability to use 3rd parties or your agents aren’t trained to collect satisfaction data), its a waste of time. Typically, angry customers will whip out unmeasurable SLAs and argue that vendors failed to achieve them, which is a huge waste of effort.
5. Easily Measurable – It’s one thing to measure something, its another thing to spend oodles of dollars to measure the same thing. If the cost of measurement doesn’t warrant the benefits of the SLA, don’t use it. The best example of this was a measure where customers call to complain about a bad transaction they’ve recently received in the mail. The vendor didn’t manage the call center, just the backoffice transactions. So, the call center needed to take notes on bad transactions and track them in a manner that allowed auditors to identify if the vendor was the cause of the bad transaction or not. Since the vendor only handled one of six steps in the backoffice process and the mainframe systems didn’t track transaction history, it was impossible to determine who caused the error without significant system modifications.
6. Time Frames – SLAs should cover a specific time period. Daily, weekly, monthly, quarterly, annually, etc. They should also only be assessed once. The example provided in #4 is a bad example of this, since customers could call to complain months after the transaction was completed, making it difficult to understand when to assess a month’s quality number. Essentially, the vendor would be in jeopardy forever, since a customer could complain at any time about a month – and every complaint would only lower the quality score, until the vendor had to pay penalties.
7. Singled Barrels – A SLA should contain only one measure, not two, three, or even four measures. If you’re SLA is “99% of transactions must meet quality standards and achieve customer satisfaction requirements” you need to track both conditions, which is nightmarishly difficult. In questionaire terminology, these are called double-barreled situations, and typically provide misleading or inaccurate pictures of operational performance.
8. Serve a Purpose – In some contracts, a minimum number of SLAs are required (to reduce the vendor’s risk, of course). That’s great when you need 3 or 4 SLAs, but what if you only need two SLAs and are therefore required to make-up another one or two to meet the contractual guidelines? These typically become “gimmes” and are a waste of time. Every SLA should serve a purpose.
9. Actionable – Every SLA should be capable of being influenced through actions of the vendor or the company. If the SLA can’t be influenced, don’t bother. A bad example may be measures of employee satisfaction with compensation in a HR outsourcing relationship, where the vendor has no control over compensation or employee /supervisor communication/training program. Since all employees will naturally dislike their compensation to some degree, the vendor has very little ability to create positive results.
10. Realistic – Look, we all want to be perfect, but those who belong to the cult of zero defects don’t understand contracting and real-life BPO and ITO. Achieving 100% of anything is simply unrealistic in most situations. Your goals can be aggressive (or evenly progressively more aggressive over time), but they should achievable.

We want to hear your comments! Let us know what you’re thinking by commenting below.

Related posts:

1. Another Tale from “When You Don’t Have Vendor Management Governance”