arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.12948cs.CLcs.SDeess.AS
收藏

语音基础模型没有学到关于语音的什么?

What Do Speech Foundation Models Not Learn About Speech?

Carnegie Mellon University(卡内基梅隆大学);MBZUAI(穆罕默德·本·扎耶德人工智能大学)

Abdul Waheed, Hanin Atwany, Bhiksha Raj, Rita Singh

AI总结:

本研究分析了Whisper等语音基础模型在Dynamic-SUPERB基准上对非语言线索的捕捉与逐层表示,通过零样本评估和逐层特征微调,揭示了模型的泛化能力、层间表示特征及下游任务适配程度。

摘要:

理解语音基础模型如何捕捉非语言线索对于提升其在各种任务中的可解释性和适应性至关重要。在我们的研究中,我们分析了Whisper、Seamless、Wav2Vec、HuBERT和Qwen2-Audio等几个著名模型,重点关注它们在Dynamic-SUPERB基准中的副语言和非副语言任务上学习到的表示。我们的研究探讨了三个关键问题:(1) 捕捉了哪些非语言线索(例如说话人意图、情感、环境上下文)?(2) 这些线索在模型的不同层中是如何表示的?(3) 这些表示在多大程度上能被有效适配到下游任务?为回答这些问题,我们首先在零样本设置下评估模型,随后对这些模型提取的逐层特征进行微调。我们的结果深入揭示了模型的泛化能力、逐层表示的特征,以及下游任务适配所需的转换程度。研究结果表明,尽管未针对某些任务进行显式训练,其中一些模型在零样本设置下的多种任务上表现良好。我们还观察到,零样本性能与学习到的更好表示相关。对逐层特征的分析表明,部分模型在学习表示的可分性与模型深度之间呈现凸关系,不同层捕捉特定任务的特征。

英文摘要:

Understanding how speech foundation models capture non-verbal cues is crucial for improving their interpretability and adaptability across diverse tasks. In our work, we analyze several prominent models such as Whisper, Seamless, Wav2Vec, HuBERT, and Qwen2-Audio focusing on their learned representations in both paralinguistic and non-paralinguistic tasks from the Dynamic-SUPERB benchmark. Our study addresses three key questions: (1) What non-verbal cues (e.g., speaker intent, emotion, environmental context) are captured? (2) How are these cues represented across different layers of the models? and (3) To what extent can these representations be effectively adapted to downstream tasks? To answer these questions, we first evaluate the models in a zero-shot setting, followed by fine-tuning on layer-wise features extracted from these models. Our results provide insights into the models' capacity for generalization, the characteristics of their layer-wise representations, and the degree of transformation required for downstream task adaptation. Our findings suggest that some of these models perform well on various tasks in zero-shot settings, despite not being explicitly trained for those tasks. We also observe that zero-shot performance correlates with better-learned representations. The analysis of layer-wise features demonstrates that some models exhibit a convex relationship between the separability of the learned representations and model depth, with different layers capturing task-specific features.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑