学习何时说话:使用离线模型进行同声语音到语音翻译的延迟与质量权衡
Learning When to Speak: Latency and Quality Trade-offs for Simultaneous Speech-to-Speech Translation with Offline Models
AI总结:
本文提出面向真实场景的同声 S2ST 系统,支持 57 种语言译入英语,并通过四种发声决策策略在接近离线准确率的同时实现可调低延迟。
摘要:
近期语音到语音翻译(S2ST)的研究主要集中在离线场景,即在给出任何输出之前,完整的输入话语都已可用。然而,这在许多真实世界场景中并不合理。在对延迟敏感的应用中,翻译不应等待完整话语,而应在输入中的信息出现后尽快说出。在本工作中,我们介绍了一个面向真实世界用例的同声 S2ST 系统。我们的系统支持从 57 种语言到英语的翻译,并提供可调参数以动态调整输出延迟,其中包括四种用于决定何时说出输出序列的策略。我们表明,这些策略达到了离线级别的准确率,同时相对于 Greedy(wait-k)基线,延迟仅有极小增加。我们开源了评估代码和交互式测试脚本,以帮助未来的 SimulS2ST 研究与应用开发。
英文摘要:
Recent work in speech-to-speech translation (S2ST) has focused primarily on offline settings, where the full input utterance is available before any output is given. This, however, is not reasonable in many real-world scenarios. In latency-sensitive applications, rather than waiting for the full utterance, translations should be spoken as soon as the information in the input is present. In this work, we introduce a system for simultaneous S2ST targeting real-world use cases. Our system supports translation from 57 languages to English with tunable parameters for dynamically adjusting the latency of the output -- including four policies for determining when to speak an output sequence. We show that these policies achieve offline-level accuracy with minimal increases in latency over a Greedy (wait-$k$) baseline. We open-source our evaluation code and interactive test script to aid future SimulS2ST research and application development.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。