利用脑激活细化自监督学习语音表征
Refining Self-Supervised Learnt Speech Representation using Brain Activations
AI总结:
本文提出利用fMRI脑激活对齐wav2vec2.0表征以细化自监督语音模型,并在SUPERB上验证其对说话人验证、语音识别和意图分类等任务的有效性。
摘要:
已有文献表明,自监督预训练模型提取的语音表征与人类语音感知的脑激活存在相似性,并且在下游任务上微调语音表征模型可以进一步提高这种相似性。然而,这种相似性是否可用于优化预训练语音模型仍不清楚。因此,在本工作中,我们提出利用功能磁共振成像(fMRI)记录的脑激活来细化常用的wav2vec2.0模型,通过将模型表征向人类神经响应对齐来实现。在SUPERB上的实验结果表明,这一操作对多个下游任务有益,例如说话人验证、自动语音识别、意图分类。因此,所提出的方法可被视为改进自监督语音模型的一种新替代方案。
英文摘要:
It was shown in literature that speech representations extracted by self-supervised pre-trained models exhibit similarities with brain activations of human for speech perception and fine-tuning speech representation models on downstream tasks can further improve the similarity. However, it still remains unclear if this similarity can be used to optimize the pre-trained speech models. In this work, we therefore propose to use the brain activations recorded by fMRI to refine the often-used wav2vec2.0 model by aligning model representations toward human neural responses. Experimental results on SUPERB reveal that this operation is beneficial for several downstream tasks, e.g., speaker verification, automatic speech recognition, intent classification.One can then consider the proposed method as a new alternative to improve self-supervised speech models.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。