通过提示微调扩展 Whisper 以实现目标说话人自动语音识别
Extending Whisper with prompt tuning to target-speaker ASR
AI总结：
本文提出利用提示微调将 Whisper 扩展至目标说话人 ASR，仅用约 1% 的任务特定参数即可达到与完全训练相当的性能，并保留时间戳等原有特性。
摘要：
目标说话人自动语音识别（ASR）旨在从多说话人重叠语音中转录出目标说话人的期望语音。现有的大多数目标说话人 ASR（TS-ASR）方法要么从头训练，要么对预训练模型进行完全微调，这导致显著的训练成本，并且不适用于大型基础模型。本文利用提示微调这一参数高效的微调方法，将大规模单说话人 ASR 模型 Whisper 扩展到 TS-ASR。我们探索并优化了多种提示微调方法的变体及其配置，以适配 TS-ASR 任务。实验结果表明，提示微调能够取得与最先进的完全训练方法相当的性能，同时仅需约 1% 的任务特定模型参数。值得注意的是，原始 Whisper 的特性（如反向文本归一化和时间戳标记）在目标说话人 ASR 中得以保留，从而使生成的转录文本保持自然且信息丰富。
英文摘要：
Target-speaker automatic speech recognition (ASR) aims to transcribe the desired speech of a target speaker from multi-talker overlapped utterances. Most of the existing target-speaker ASR (TS-ASR) methods involve either training from scratch or fully fine-tuning a pre-trained model, leading to significant training costs and becoming inapplicable to large foundation models. This work leverages prompt tuning, a parameter-efficient fine-tuning approach, to extend Whisper, a large-scale single-talker ASR model, to TS-ASR. Variants of prompt tuning approaches along with their configurations are explored and optimized for TS-ASR.Experimental results show that prompt tuning can achieve performance comparable to state-of-the-art full training approaches while only requiring about 1\% of task-specific model parameters. Notably, the original Whisper's features, such as inverse text normalization and timestamp tagging, are retained in target-speaker ASR, keeping the generated transcriptions natural and informative.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
