arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.01737eess.AScs.CLcs.SD
收藏

基于Whisper迁移学习的微观可懂度预测

Transfer Learning from Whisper for Microscopic Intelligibility Prediction

Université de Toulon(土伦大学);Aix Marseille Univ(艾克斯-马赛大学);CNRS(法国国家科学研究中心);LIS(信息与系统科学实验室)

Paul Best, Santiago Cuervo, Ricard Marxer

AI总结:

针对微观可懂度预测缺乏大规模深度学习迁移学习方法的问题,本文提出将Whisper模型迁移学习用于词汇响应级微观可懂度预测,在零样本及微调场景下均优于基线,最高相对提升66%,验证了该类方法的潜力。

摘要:

宏观可懂度模型用于预测给定带噪语音刺激下人类的预期词错误率。与之不同,微观可懂度模型旨在对听众的感知做出细粒度预测,例如预测语音或词汇响应。当前最优的宏观模型采用大规模语音处理深度学习模型的迁移学习方法,而这类方法极少用于微观建模。本文研究了将当前最优的自动语音识别深度学习模型Whisper的迁移学习用于词汇响应层面的微观可懂度预测。我们的方法优于所有考虑的基线模型,即便在零样本设置下也是如此;在微调以预测听众响应时,相对提升最高可达66%。研究结果表明,基于大规模深度学习的方法在微观可懂度预测领域具有应用前景。

英文摘要:

Macroscopic intelligibility models predict the expected human word-error-rate for a given speech-in-noise stimulus. In contrast, microscopic intelligibility models aim to make fine-grained predictions about listeners' perception, e.g. predicting phonetic or lexical responses. State-of-the-art macroscopic models use transfer learning from large scale deep learning models for speech processing, whereas such methods have rarely been used for microscopic modeling. In this paper, we study the use of transfer learning from Whisper, a state-of-the-art deep learning model for automatic speech recognition, for microscopic intelligibility prediction at the level of lexical responses. Our method outperforms the considered baselines, even in a zero-shot setup, and yields a relative improvement of up to 66\% when fine-tuned to predict listeners' responses. Our results showcase the promise of large scale deep learning based methods for microscopic intelligibility prediction.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑