利用预训练AV-HuBERT的视听特征进行多模态构音障碍语音重建
Exploiting Audio-Visual Features with Pretrained AV-HuBERT for Multi-Modal Dysarthric Speech Reconstruction
AI总结：
针对重度构音障碍语音在复杂噪声环境下重建难度大的问题，提出融合唇部视觉信息、基于AV-HuBERT的多模态DSR框架，经UASpeech语料库验证，可显著提升语音可懂度与自然度，重度患者词错率绝对下降42.1%。
摘要：
构音障碍语音重建（DSR）旨在通过提升可懂度和自然度，将构音障碍语音转化为正常语音。这是一项极具挑战性的任务，尤其是针对重度构音障碍患者且处于复杂嘈杂声学环境中的说话场景。为应对这些挑战，我们提出一种新型多模态框架，在DSR中利用视觉信息（如唇部动作）作为重建高度异常发音的额外线索。该多模态框架包含：（i）多模态编码器，用于从带辅助视觉特征的构音障碍语音中提取鲁棒的音素嵌入；（ii）方差适配器，用于从提取的音素嵌入中推断正常音素时长和音高轮廓；（iii）说话人编码器，用于编码说话人的嗓音特征；（iv）梅尔解码器，用于基于提取的音素嵌入、韵律特征和说话人嵌入生成重建后的梅尔频谱图。在常用的UASpeech语料库上开展的客观与主观评估均表明，我们提出的方法在语音可懂度和自然度方面相较基线系统实现了显著提升，尤其是针对症状更严重的说话人。与原始构音障碍语音相比，重建语音针对更重度构音障碍水平的患者实现了42.1%的绝对词错误率下降。
英文摘要：
Dysarthric speech reconstruction (DSR) aims to transform dysarthric speech into normal speech by improving the intelligibility and naturalness. This is a challenging task especially for patients with severe dysarthria and speaking in complex, noisy acoustic environments. To address these challenges, we propose a novel multi-modal framework to utilize visual information, e.g., lip movements, in DSR as extra clues for reconstructing the highly abnormal pronunciations. The multi-modal framework consists of: (i) a multi-modal encoder to extract robust phoneme embeddings from dysarthric speech with auxiliary visual features; (ii) a variance adaptor to infer the normal phoneme duration and pitch contour from the extracted phoneme embeddings; (iii) a speaker encoder to encode the speaker's voice characteristics; and (iv) a mel-decoder to generate the reconstructed mel-spectrogram based on the extracted phoneme embeddings, prosodic features and speaker embeddings. Both objective and subjective evaluations conducted on the commonly used UASpeech corpus show that our proposed approach can achieve significant improvements over baseline systems in terms of speech intelligibility and naturalness, especially for the speakers with more severe symptoms. Compared with original dysarthric speech, the reconstructed speech achieves 42.1\% absolute word error rate reduction for patients with more severe dysarthria levels.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
