Macroscopic intelligibility models predict the expected human word-error-rate for a given speech-in-noise stimulus. In contrast, microscopic intelligibility models aim to make fine-grained predictions about listeners' perception, e.g. predicting phonetic or lexical responses. State-of-the-art macroscopic models use transfer learning from large scale deep learning models for speech processing, whereas such methods have rarely been used for microscopic modeling. In this paper, we study the use of transfer learning from Whisper, a state-of-the-art deep learning model for automatic speech recognition, for microscopic intelligibility prediction at the level of lexical responses. Our method outperforms the considered baselines, even in a zero-shot setup, and yields a relative improvement of up to 66\% when fine-tuned to predict listeners' responses. Our results showcase the promise of large scale deep learning based methods for microscopic intelligibility prediction.
Speech Intelligibility Assessment with Uncertainty-Aware Whisper Embeddings and sLSTM
Research Center for Information Technology Innovation, Academia Sinica(资讯科技创新研究中心,学术院);Social Network and Human Centered Computing, Institute of Information Science, Academia Sinica(社交网络与人本计算,信息科学研究所,学术院);Institute of Information Science, Academia Sinica(信息科学研究所,学术院)
Enhancing Intelligibility for Generative Target Speech Extraction via Joint Optimization with Target Speaker ASR
School of Information Science and Engineering, Shandong University(信息科学与工程学院,山东大学);Institute of Artificial Intelligence (TeleAI), China Telecom(人工智能研究所(TeleAI),中国电信)
Objective and Subjective Evaluation of Diffusion-Based Speech Enhancement for Dysarthric Speech
Delft University of Technology(代尔夫特理工大学);Erasmus Medical Centre (EMC)(埃因霍温医学院);Indian Inst. of Information Technology Allahabad(阿利哈巴德信息科技学院);Dept. of Plastic and Reconstructive Surgery(整形外科与重建外科系);Dept. of Electronics and Communication(电子与通信系)