AMPS：基于多模态复述监督的自动语音识别
AMPS: ASR with Multimodal Paraphrase Supervision
AI总结：
针对多语言对话式语音给ASR带来的挑战，本研究提出AMPS技术，通过复述监督增强多语言多模态ASR系统，结合SeamlessM4T模型实现WER最高5%的相对降低，并完成详细评估。
摘要：
自发或对话式多语言语音给最先进的自动语音识别（ASR）系统带来诸多挑战。本研究提出新技术AMPS，通过基于复述的监督增强多语言多模态ASR系统，以改善印地语、马拉地语、马拉雅拉姆语、卡纳达语和尼扬贾语等多种语言的对话式ASR性能。训练多模态ASR模型时，我们将参考转录的复述作为额外监督，并针对ASR性能较差的 utterance 选择性调用该复述目标。将AMPS与最先进的多模态模型SeamlessM4T结合使用，我们实现了单词错误率（WER）的显著相对降低，最高达5%。我们使用客观和人类评估指标对系统进行了详细分析。
英文摘要：
Spontaneous or conversational multilingual speech presents many challenges for state-of-the-art automatic speech recognition (ASR) systems. In this work, we present a new technique AMPS that augments a multilingual multimodal ASR system with paraphrase-based supervision for improved conversational ASR in multiple languages, including Hindi, Marathi, Malayalam, Kannada, and Nyanja. We use paraphrases of the reference transcriptions as additional supervision while training the multimodal ASR model and selectively invoke this paraphrase objective for utterances with poor ASR performance. Using AMPS with a state-of-the-art multimodal model SeamlessM4T, we obtain significant relative reductions in word error rates (WERs) of up to 5%. We present detailed analyses of our system using both objective and human evaluation metrics.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
