通过自监督表示探索情感语音与音乐中的声学相似性
Exploring Acoustic Similarity in Emotional Speech and Music via Self-Supervised Representations
AI总结：
本研究通过分析SSL模型在语音和音乐情感识别中的逐层行为及跨领域微调，揭示了情感语音与音乐间的声学相似性及模型情感偏差，并证明参数高效微调能提升跨域识别性能。
摘要：
由于语音和音乐在声学上的重叠，从两者中识别情感具有相似性，这引发了在两个领域间进行知识迁移的研究兴趣。然而，鉴于语音和音乐的自监督学习（SSL）模型很少被应用于跨领域研究，语音和音乐之间共享的声学线索，特别是那些由SSL模型编码的线索，在很大程度上仍未被探索。在这项工作中，我们重新审视了情感语音和音乐之间的声学相似性，首先分析了用于语音情感识别（SER）和音乐情感识别（MER）的SSL模型的逐层行为。此外，我们通过在两阶段微调过程中比较几种方法来进行跨领域适应，研究了利用音乐提升SER以及利用语音提升MER的有效方式。最后，我们使用针对单个情感的Frechet音频距离探索了情感语音和音乐之间的声学相似性，揭示了语音和音乐SSL模型中存在的情感偏差问题。我们的研究结果表明，虽然语音和音乐的SSL模型确实捕获了共享的声学特征，但由于其训练策略和领域特异性，它们的行为会因不同情感而有所变化。此外，参数高效微调可以通过利用彼此的知识来提升SER和MER的性能。本研究为情感语音和音乐之间的声学相似性提供了新见解，并突出了跨领域泛化以改进SER和MER系统的潜力。
英文摘要：
Emotion recognition from speech and music shares similarities due to their acoustic overlap, which has led to interest in transferring knowledge between these domains. However, the shared acoustic cues between speech and music, particularly those encoded by Self-Supervised Learning (SSL) models, remain largely unexplored, given the fact that SSL models for speech and music have rarely been applied in cross-domain research. In this work, we revisit the acoustic similarity between emotion speech and music, starting with an analysis of the layerwise behavior of SSL models for Speech Emotion Recognition (SER) and Music Emotion Recognition (MER). Furthermore, we perform cross-domain adaptation by comparing several approaches in a two-stage fine-tuning process, examining effective ways to utilize music for SER and speech for MER. Lastly, we explore the acoustic similarities between emotional speech and music using Frechet audio distance for individual emotions, uncovering the issue of emotion bias in both speech and music SSL models. Our findings reveal that while speech and music SSL models do capture shared acoustic features, their behaviors can vary depending on different emotions due to their training strategies and domain-specificities. Additionally, parameter-efficient fine-tuning can enhance SER and MER performance by leveraging knowledge from each other. This study provides new insights into the acoustic similarity between emotional speech and music, and highlights the potential for cross-domain generalization to improve SER and MER systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
