SECap：基于大语言模型的语音情感描述
SECap: Speech Emotion Captioning with Large Language Model
AI总结：
针对语音情感分类不足以表达复杂情感的问题，提出SECap框架，利用HuBERT提取特征、Q-Former桥接、LLaMA生成自然语言情感描述，客观评估优于基线，主观评分与人类相当。
摘要：
语音情感在人类交流中至关重要，并广泛用于语音合成和自然语言理解等领域。以往的大多数研究，如语音情感识别，将语音情感归类为固定的类别集合。然而，人类语音中表达的情感往往复杂多样，将其归类为预定义组可能不足以充分代表语音情感。相反，直接通过自然语言描述语音情感可能是一种更有效的方法。遗憾的是，专注于这一方向的研究并不多。因此，本文提出了一种名为SECap的语音情感描述框架，旨在使用自然语言有效描述语音情感。得益于大语言模型在语言理解和文本生成方面的出色能力，SECap采用LLaMA作为文本解码器，以生成连贯的语音情感描述。此外，SECap利用HuBERT作为音频编码器提取通用语音特征，并利用Q-Former作为桥接网络，为LLaMA提供与情感相关的语音特征。为此，Q-Former利用互信息学习来解耦情感相关语音特征和语音内容，同时实施对比学习以提取更多情感相关语音特征。客观和主观评估结果表明：1）SECap框架在所有客观评估中均优于HTSAT-BART基线；2）SECap能够生成高质量的语音情感描述，在主观平均意见得分测试中达到与人类标注者相当的性能。
英文摘要：
Speech emotions are crucial in human communication and are extensively used in fields like speech synthesis and natural language understanding. Most prior studies, such as speech emotion recognition, have categorized speech emotions into a fixed set of classes. Yet, emotions expressed in human speech are often complex, and categorizing them into predefined groups can be insufficient to adequately represent speech emotions. On the contrary, describing speech emotions directly by means of natural language may be a more effective approach. Regrettably, there are not many studies available that have focused on this direction. Therefore, this paper proposes a speech emotion captioning framework named SECap, aiming at effectively describing speech emotions using natural language. Owing to the impressive capabilities of large language models in language comprehension and text generation, SECap employs LLaMA as the text decoder to allow the production of coherent speech emotion captions. In addition, SECap leverages HuBERT as the audio encoder to extract general speech features and Q-Former as the Bridge-Net to provide LLaMA with emotion-related speech features. To accomplish this, Q-Former utilizes mutual information learning to disentangle emotion-related speech features and speech contents, while implementing contrastive learning to extract more emotion-related speech features. The results of objective and subjective evaluations demonstrate that: 1) the SECap framework outperforms the HTSAT-BART baseline in all objective evaluations; 2) SECap can generate high-quality speech emotion captions that attain performance on par with human annotators in subjective mean opinion score tests.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
