面向视听多通道语音分离与识别的联合说话人特征学习
Joint Speaker Features Learning for Audio-visual Multichannel Speech Separation and Recognition
AI总结：
针对视听多通道语音分离与识别的零样本自适应，本文通过专用融合模块联合训练xVector与ECAPA-TDNN说话人编码器，显著提升分离识别效果并大幅降低WER。
摘要：
本文提出了用于视听多通道语音分离与识别系统零样本自适应的联合说话人特征学习方法。xVector与ECAPA-TDNN说话人编码器通过专用融合模块连接，并与完整系统训练紧密集成。在LRS3-TED数据模拟的多通道重叠语音上开展的实验表明，与未进行联合说话人特征估计的基线相比，联合说话人特征学习能够持续提升语音分离和识别性能。进一步分析显示，性能提升与采用余弦相似度衡量的说话人间区分度增强密切相关。性能最佳的联合说话人特征学习自适应系统在融合WavLM特征和视频模态后，在Dev集和Test集上较经微调的WavLM基线模型取得具有统计显著性的WER降低，绝对降幅分别为21.6%和25.3%（相对降幅分别为67.5%和83.5%）。
英文摘要：
This paper proposes joint speaker feature learning methods for zero-shot adaptation of audio-visual multichannel speech separation and recognition systems. xVector and ECAPA-TDNN speaker encoders are connected using purpose-built fusion blocks and tightly integrated with the complete system training. Experiments conducted on LRS3-TED data simulated multichannel overlapped speech suggest that joint speaker feature learning consistently improves speech separation and recognition performance over the baselines without joint speaker feature estimation. Further analyses reveal performance improvements are strongly correlated with increased inter-speaker discrimination measured using cosine similarity. The best-performing joint speaker feature learning adapted system outperformed the baseline fine-tuned WavLM model by statistically significant WER reductions of 21.6% and 25.3% absolute (67.5% and 83.5% relative) on Dev and Test sets after incorporating WavLM features and video modality.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
