A Way Out of Automated Phone Hell

Getting transferred from one automated message to another while stuck in a company’s convoluted telephone system is enough to make even the most unflappable individual’s blood boil.

By Louise Knapp

Story location: http://www.wired.com/news/gizmos/0,1452,62184,00.html

A solution that may prevent violence against handsets comes in the form of a new software program designed to detect callers’ frustration and transfer them to a human operator.

The system works by analyzing not only what callers say, but also how they say it. Callers get transferred if they start to spit out expletives or if they simply sound angry.

"We are not just interested in what is being said, but how it is being conveyed," said program creator Shrikanth Narayanan, professor in the Speech Analysis and Interpretation Laboratory at the University of Southern California.

Narayanan’s program parses speech by transforming it into electrical waveforms. "If you plot these waveforms given off by speech — those wiggly things — a high energy will give a greater amplitude, which affects the way the waves come out," Narayanan said.

The angrier the caller becomes, the greater the energy that will be apparent in the wave pattern. Once the wave pattern hits the level the computer is programmed to recognize as the frustration cutoff point, the caller will immediately be transferred to the operator.

Context is also important in the system’s programming — it not only determines the kind of words the computer is programmed to be on guard for, but also the type of emotion it is set up to recognize.

"We don’t need to know about all the emotions, just the ones that are relative to that domain," Narayanan said. "For the call center it is frustration — you don’t want to lose the customer because they are becoming frustrated."

Narayanan’s program is set up to continually learn from the examples it is given. "The computer learns through pattern recognition," Narayanan said. "It’s computer algorithm technology that relies on learning from example. When it sees new examples it can react appropriately."

Narayanan’s system learned what to look for by training on nearly 1,400 real phone calls.

The calls came from an airline company that collected the recordings in order to analyze how their representatives handled complaints. Narayanan had a team analyze the calls, rating on a scale of one to five how frustrated the caller sounded. The calls were examined for word content, and speech waves were examined to determine patterns that reflect frustration.

"Examples of when people get frustrated were taken from what people perceived as frustration in the caller," Narayanan said. "From this we devised a number of identification tags and programmed the computer to recognize these, so the machine attempts to replicate the human interpretation."

Narayanan said his system gets it right 80 percent of the time, with an equal number of false positives and false negatives.

Elsa Martin, executive of international and domestic affairs for Apex Voice Communications of Sherman Oaks, California, said the system sounds interesting.

"I think that the company would use this. We deal with a lot of important clients all over the world, and they focus on customer care — their priority is to make sure the customer is happy," Martin said.

Martin said her company would want to see an improvement on the system’s accuracy, though she said it would not have a problem with the false positives.

"If the customer was being transferred when not frustrated, this would not be a problem, as it would be a bonus for the customer to speak to a real person," Martin said.

Philip Resnik, professor in the computational linguistics department at the University of Maryland Institute for Advanced Computer Studies, agreed.

"False positives would probably be OK. When I am on the phone with one of these, I tend to hit zero over and over again to get a human. So the fact you are transferred to a human when not frustrated probably will not upset people," Resnik said.

Resnik also approved of Narayanan’s training techniques.

"It sounds like they have done things right in terms of the training data — others have used actors to simulate emotions, but they are using naturally occurring data, and this is the right thing to do," Resnik said.

Resnik said he thinks that Narayanan could improve the accuracy of the system by using more data.

John Hansen, professor at the University of Colorado, Boulder’s Center for Spoken Language Research, said Narayanan’s system would probably have trouble coping with all the different accents and ages of the callers.

"You also have the problem that the caller may be calling from a cell phone, a cordless phone, that callers have different dialects," Hansen said. "You don’t want to assume they are mad only because they are not speaking normally. They could stutter, for example."

Hansen said it is not hard to build a system like this, but it’s trickier to get the system to cope with all these variables.

"I would say, in terms of a commercial product, I would be skeptical that they would have something reliable enough for market," Hansen said.

Narayanan admits there are obstacles to overcome.

"We base our interpretation on probability, so it’s not certain," Narayanan said. "We are still trying to figure out what are good indicators of emotion and how we can make machines learn these and respond automatically."

But Narayanan said the more data that is available to analyze, the better his system will become and the more able to cope with all the variables.

"This is a good start," Narayanan said. "In two years’ time we could have a preliminary version of the program available."