赋能 Whisper 作为联合多说话人和目标说话人语音识别系统
Empowering Whisper as a Joint Multi-Talker and Target-Talker Speech Recognition System
AI总结：
本文提出一种开创性方法，通过冻结 Whisper 并插入 Sidecar 分离器、引入 Target Talker Identifier 及软提示微调，使其能同时处理多说话人与目标说话人语音识别任务，在多个数据集上超越以往方法并具备零样本能力。
摘要：
多说话人语音识别和目标说话人语音识别均涉及多说话人语境下的转录，仍是重大挑战。然而，现有方法很少尝试同时解决这两项任务。本研究提出一种开创性方法，赋能作为语音基础模型的 Whisper，以处理联合多说话人与目标说话人语音识别任务。具体而言，(i) 我们冻结 Whisper 并在其编码器中插入 Sidecar 分离器，以分离多说话人的混合嵌入；(ii) 引入 Target Talker Identifier，动态识别目标说话人的嵌入流，仅需三秒注册语音作为提示；(iii) 探索解码器的软提示微调以更好地适应任务。我们的方法在双说话人和三说话人的 LibriMix 和 LibriSpeechMix 数据集上，两项任务均超越以往方法，并在 AishellMix 普通话数据集的多说话人 ASR 上提供了可接受的零样本性能。
英文摘要：
Multi-talker speech recognition and target-talker speech recognition, both involve transcription in multi-talker contexts, remain significant challenges. However, existing methods rarely attempt to simultaneously address both tasks. In this study, we propose a pioneering approach to empower Whisper, which is a speech foundation model, to tackle joint multi-talker and target-talker speech recognition tasks. Specifically, (i) we freeze Whisper and plug a Sidecar separator into its encoder to separate mixed embedding for multiple talkers; (ii) a Target Talker Identifier is introduced to identify the embedding flow of the target talker on the fly, requiring only three-second enrollment speech as a cue; (iii) soft prompt tuning for decoder is explored for better task adaptation. Our method outperforms previous methods on two- and three-talker LibriMix and LibriSpeechMix datasets for both tasks, and delivers acceptable zero-shot performance on multi-talker ASR on AishellMix Mandarin dataset.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
