SimulEval：同声传译评估工具包
SimulEval: An Evaluation Toolkit for Simultaneous Translation
摘要：
文本和语音的同步翻译专注于实时和低延迟的场景，其中模型在读取完整的源输入之前开始翻译。评估同声翻译模型比离线模型更复杂，因为除了翻译质量之外，延迟是另一个需要考虑的因素。尽管研究界越来越关注同声翻译的新建模方法，但目前缺乏一个通用的评估程序。因此，我们提出了SimulEval，一个易于使用的通用评估工具包，用于同时进行文本和语音翻译。引入服务器-客户端方案来创建同声翻译场景，其中服务器发送源输入并接收用于评估的预测，而客户端执行定制的策略。给定一个策略，它会自动执行同步解码，并共同报告几个流行的延迟指标。我们还调整了从文本同声翻译到语音任务的延迟度量。此外，SimulEval还配备了可视化界面，以便更好地了解系统的同步解码过程。SimulEval已经被广泛用于IWIEF 2020同步语音翻译的共享任务。代码将在发布时发布。
英文摘要：
Simultaneous translation on both text and speech focuses on a real-time and low-latency scenario where the model starts translating before reading the complete source input. Evaluating simultaneous translation models is more complex than offline models because the latency is another factor to consider in addition to translation quality. The research community, despite its growing focus on novel modeling approaches to simultaneous translation, currently lacks a universal evaluation procedure. Therefore, we present SimulEval, an easy-to-use and general evaluation toolkit for both simultaneous text and speech translation. A server-client scheme is introduced to create a simultaneous translation scenario, where the server sends source input and receives predictions for evaluation and the client executes customized policies. Given a policy, it automatically performs simultaneous decoding and collectively reports several popular latency metrics. We also adapt latency metrics from text simultaneous translation to the speech task. Additionally, SimulEval is equipped with a visualization interface to provide better understanding of the simultaneous decoding process of a system. SimulEval has already been extensively used for the IWSLT 2020 shared task on simultaneous speech translation. Code will be released upon publication.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
