arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2203.15326cs.SDcs.AIeess.AS
收藏

基于多层次声学信息的协同注意语音情感识别

Speech Emotion Recognition with Co-Attention based Multi-level Acoustic Information

Nanyang Technological University(南洋理工大学);Tianjin University(天津大学)

Heqing Zou, Yuke Si, Chen Chen, Deepu Rajan, Eng Siong Chng

摘要:

语音情感识别(SER)的目的是帮助机器仅从音频信息中理解人类的主观情感。然而,提取和利用全面深入的音频信息仍然是一项具有挑战性的任务。在本文中,我们提出了一个端到端的语音情感识别系统,使用多层次的声学信息与一个新设计的共同注意模块。我们首先提取多层次的声学信息,包括MFCC,频谱图,以及嵌入的高级声学信息,分别与CNN,BiLSTM和wav2vec2。然后,这些提取的特征被视为多模态输入,并融合所提出的共同注意力机制。在IEMOCAP数据集上进行了实验,我们的模型在两种不同的非特定人交叉验证策略下都取得了很好的效果。我们的代码可以在GitHub上找到。

英文摘要:

Speech Emotion Recognition (SER) aims to help the machine to understand human's subjective emotion from only audio information. However, extracting and utilizing comprehensive in-depth audio information is still a challenging task. In this paper, we propose an end-to-end speech emotion recognition system using multi-level acoustic information with a newly designed co-attention module. We firstly extract multi-level acoustic information, including MFCC, spectrogram, and the embedded high-level acoustic information with CNN, BiLSTM and wav2vec2, respectively. Then these extracted features are treated as multimodal inputs and fused by the proposed co-attention mechanism. Experiments are carried on the IEMOCAP dataset, and our model achieves competitive performance with two different speaker-independent cross-validation strategies. Our code is available on GitHub.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑