Automated Empathy

Wading through voice prompts when calling an airline, bank, or utility is exasperating, especially as the machine cannot comprehend the rising anxiety in your voice, or your desire to speak to a sympathetic human being.

Professor Julia Hirschberg and her colleagues in the Department of Computer Science at The Fu Foundation School of Engineering and Applied Science are working to change all that. They are conducting research into identifying recognition errors, user corrections, and human emotion in spoken dialogue systems such as the ones users encounter when calling Amtrak or American Express.

As head of the Human-Computer Interface Research Department at AT&T Laboratories-Research, Hirschberg developed programs that generated natural-sounding speech from text input and programs that browsed and searched speech as people browse and search text. Since coming to Columbia in 2002, she has focused on computational linguistics, the writing of software that generates and interprets text and speech. Her specialty is the study of intonation in human speech, the production of intonational variation in automatic text-to-speech systems that imitates human performance, and the automatic identification and interpretation of features of human speech.

The results may benefit corporations and individuals alike.

“Companies are worried about their public image,” she said. “They don’t want to annoy their customers, so they want their automated systems to sound as natural as possible. If a caller’s emotional state can be identified automatically, the system will know when a call must be handed off to a person for individual attention.”

To achieve this, Hirschberg and her fellow researchers are gathering reams of digital data on speech patterns, analyzing the samples to find reliable predictors of particular emotions such as frustration or uncertainty. Their objective is to write software that recognizes emotion through intonation and amplitude variation, allowing a machine to infer a caller’s emotional state from his or her speech.

“No one has been able to do this in a practical way yet,” she said. “This is the essence of research. It’s a pretty wild experiment, but the potential payoff is enormous.”