Can machine learning (ML) tools be used to identify which mood we’re in – and, if so, how accurate are these predictions? A team of researchers has investigated if very short audio segments are enough for ML models to tell how we feel, independent from the words that are spoken. They found that certain models can identify emotion in sound segments with approximately the same accuracy as humans. These models could enable continuous emotion classification in real-time scenarios, the researchers said.

Words are important to express ourselves. What we don’t say, however, may be even more instrumental in conveying emotions. Humans can often tell how people around them feel through non-verbal cues embedded in our voice.

Now, researchers in Germany wanted to find out if technical tools, too, can accurately predict emotional undertones in fragments of voice recordings. To do so, they compared three ML models’ accuracy to recognize diverse emotions in audio excepts. Their results were published in Frontiers in Psychology.

“Here we show that machine learning can be used to recognize emotions from audio clips as short as 1.5 seconds,” said the article’s first author Hannes Diemerling, a researcher at the Center for Lifespan Psychology at the Max Planck Institute for Human Development. “Our models achieved an accuracy similar to humans when categorizing meaningless sentences with emotional coloring spoken by actors.”

Hearing how we feel

The researchers drew nonsensical sentences from two datasets – one Canadian, one German – which allowed them to investigate whether ML models can accurately recognize emotions regardless of language, cultural nuances, and semantic content. Each clip was shortened to a length of 1.5 seconds, as this is how long humans need to recognize emotion in speech. It is also the shortest possible audio length in which overlapping of emotions can be avoided. The emotions included in the study were joy, anger, sadness, fear, disgust, and neutral.

Based on training data, the researchers generated ML models which worked one of three ways: Deep neural networks (DNNs) are like complex filters that analyze sound components like frequency or pitch – for example when a voice is louder because the speaker is angry – to identify underlying emotions. Convolutional neural networks (CNNs) scan for patterns in the visual representation of soundtracks, much like identifying emotions from the rhythm and texture of a voice. The hybrid model (C-DNN) merges both techniques, using both audio and its visual spectrogram to predict emotions. The models then were tested for effectiveness on both datasets.

“We found that DNNs and C-DNNs achieve a better accuracy than only using spectrograms in CNNs,” Diemerling said. “Regardless of model, emotion classification was correct with a higher probability than can be achieved through guessing and was comparable to the accuracy of humans.”

Read and download the original article

As good as any human

“We wanted to set our models in a realistic context and used human prediction skills as a benchmark,” Diemerling explained. “Had the models outperformed humans, it could mean that there might be patterns that are not recognizable by us.” The fact that untrained humans and models performed similarly may mean that both rely on resembling recognition patters, the researchers said.

The present findings also show that it is possible to develop systems that can instantly interpret emotional cues to provide immediate and intuitive feedback in a wide range of situations. This could lead to scalable, cost-efficient applications in various domains where understanding emotional context is crucial, such as therapy and interpersonal communication technology.

The researchers also pointed to some limitations in their study, for example, that actor-spoken sample sentences may not convey the full spectrum of real, spontaneous emotion. They also said that future work should investigate audio segments that last longer or shorter than 1.5 seconds to find out which duration is optimal for emotion recognition.

REPUBLISHING GUIDELINES: Open access and sharing research is part of Frontiers’ mission. Unless otherwise noted, you can republish articles posted in the Frontiers news site — as long as you include a link back to the original research. Selling the articles is not allowed.