Caller Experience Index

SpeechCycle recently introduced al new way to look at the quality of interaction in voice automated systems. It is called the Caller Experience Index (CEI).

Typically caller experience is evaluated by asking several questions to callers a few hours, or a few days, after they actually interacted with an IVR. The problem is that caller surveys do not provide much insight into what the problems are with the system. Some caller responses may be affected by past non satisfactory interactions with other voice systems or by their general sentiment about automated customer care or about the specific service company they are dealing with. As surprising as it seems, some callers cannot even remember if they interacted with a “computer voice” by pushing keys or by speaking. So, what to do if the result of the survey turns out to be negative? Have callers been truly objective in their evaluation? Did they, at the time they participated into the survey, actually remember their experience? Are there other factors that influenced the survey results? And, if the survey results are positive, does it really mean that everything is all right and the system is perfect?

Without eliminating the need to take surveys and the high informational value they provide, the Caller Experience Index is a new scientific method for gauging the quality of a voice system with respect to the user experience. CEI corresponds to a numerical value, between 1 and 5, which rates the overall quality of the experience, 1 being very bad and 5 very good. But that’s not all. The number represents a summary of a dozen or so of measurements taken on a large number of call logs and call recordings, each measurement representing a specific dimension affecting the caller experience. The overall duration of the call, the average number of speech recognition errors and re-prompts, and the number of times the voice of the caller is not heard are some examples of those measurable quantities. While the values of those quantities can be computed automatically based on the availability of call logs, there are few other values that require subjective qualitative analysis. In particular we are looking at caller cooperation and interaction experience. Qualitative scores on both dimensions are assigned to each of a restricted set of call recordings—typically a couple of hundreds—which are selected to represent an unbiased sample of all calls. Annotators who are trained on qualitative judgment of these dimensions listen to the calls and score them. Inter-annotator agreement techniques are in place to make sure that scoring is consistent across listeners.

Thus CEI is computed as a combination of quantitative and qualitative measurements, each one of them giving insight with respect to a particular dimension affecting the overall caller experience. The information collected during the CEI evaluation constitute the basis of a set of recommendations that, if implemented, would improve the caller experience, and consequently increase the value of CEI for the next release of the system.

The Caller Experience Index is the first industry attempt to improve the quality of the interaction using a scientific method based on data and consequently help increase the adoption of voice automated systems. Speech recognition has been often claimed to be an art. We, at SpeechCycle, are committed to make it a science.

More on the Caller Experience Index at SpeechTechMag and SpeechCycle's latest press release.