arXiv 2310.148062023-10-23cs.CLcs.SDeess.AS

利用时间戳信息进行序列化联合流式识别与翻译

Leveraging Timestamp Information for Serialized Joint Streaming Recognition and Translation

Microsoft（微软）

Sara Papi, Peidong Wang, Junkun Chen, Jian Xue, Naoyuki Kanda, Jinyu Li, Yashesh Gaur

备注：: \c{opyright} 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出一种流式 Transformer-Transducer 模型，利用基于时间戳的词元级序列化输出训练方法，首次实现单解码器生成多对一和一对多联合 ASR 与 ST 输出。

摘要：

随着全球交流和跨语言互动的增加，对即时语音转录和翻译的需求日益增长。这使得在用户应用中提供多语言翻译变得至关重要。传统的自动语音识别（ASR）和语音翻译（ST）方法通常依赖独立系统，导致计算资源效率低下，并增加了实时同步的复杂性。本文提出了一种流式 Transformer-Transducer（T-T）模型，能够使用单个解码器联合生成多对一和一对多的转录与翻译。我们引入了一种基于时间戳信息的词元级序列化输出训练新方法，以在流式设置中有效生成 ASR 和 ST 输出。在 {it,es,de}->en 上的实验证明了我们方法的有效性，首次实现了使用单个解码器生成一对多联合输出。

英文摘要：

The growing need for instant spoken language transcription and translation is driven by increased global communication and cross-lingual interactions. This has made offering translations in multiple languages essential for user applications. Traditional approaches to automatic speech recognition (ASR) and speech translation (ST) have often relied on separate systems, leading to inefficiencies in computational resources, and increased synchronization complexity in real time. In this paper, we propose a streaming Transformer-Transducer (T-T) model able to jointly produce many-to-one and one-to-many transcription and translation using a single decoder. We introduce a novel method for joint token-level serialized output training based on timestamp information to effectively produce ASR and ST outputs in the streaming setting. Experiments on {it,es,de}->en prove the effectiveness of our approach, enabling the generation of one-to-many joint outputs with a single decoder for the first time.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。