arXiv 2312.151852023-12-23cs.CLcs.HCcs.MMcs.SDeess.AS

emotion2vec：用于语音情感表征的自监督预训练

emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation

Shanghai Jiao Tong University（上海交通大学）；Fudan University（复旦大学）；The Chinese University of Hong Kong（香港中文大学）；Alibaba（阿里巴巴）

Ziyang Ma, Zhisheng Zheng, Jiaxin Ye, Jinchao Li, Zhifu Gao, Shiliang Zhang, Xie Chen

备注：: Code, checkpoints, and extracted features are available at https://github.com/ddlBoJack/emotion2vec

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出emotion2vec，首个通用语音情感表征模型，通过自监督在线蒸馏结合话语级和帧级损失预训练，在IEMOCAP及10种语言数据集上超越现有模型，并适用于多种情感任务。

摘要：

我们提出了emotion2vec，一个通用的语音情感表征模型。emotion2vec通过自监督在线蒸馏在开源无标签情感数据上进行预训练，在预训练过程中结合了话语级损失和帧级损失。通过在主流IEMOCAP数据集上仅训练线性层进行语音情感识别任务，emotion2vec超越了最先进的预训练通用模型和情感专家模型。此外，emotion2vec在10种不同语言的语音情感识别数据集上表现出一致的改进。emotion2vec在其他情感任务上也表现出色，如歌曲情感识别、对话中的情感预测和情感分析。对比实验、消融实验和可视化全面展示了所提出的emotion2vec的通用能力。据我们所知，emotion2vec是各种情感相关任务中的首个通用表征模型，填补了该领域的空白。

英文摘要：

We propose emotion2vec, a universal speech emotion representation model. emotion2vec is pre-trained on open-source unlabeled emotion data through self-supervised online distillation, combining utterance-level loss and frame-level loss during pre-training. emotion2vec outperforms state-of-the-art pre-trained universal models and emotion specialist models by only training linear layers for the speech emotion recognition task on the mainstream IEMOCAP dataset. In addition, emotion2vec shows consistent improvements among 10 different languages of speech emotion recognition datasets. emotion2vec also shows excellent results on other emotion tasks, such as song emotion recognition, emotion prediction in conversation, and sentiment analysis. Comparison experiments, ablation experiments, and visualization comprehensively demonstrate the universal capability of the proposed emotion2vec. To the best of our knowledge, emotion2vec is the first universal representation model in various emotion-related tasks, filling a gap in the field.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。