DiariST：带说话人分角色的流式语音翻译
DiariST: Streaming Speech Translation with Speaker Diarization
AI总结：
本文提出首个集成 token 级序列化输出训练与 t-vector 的流式语音翻译和说话人分角色系统 DiariST，并构建 DiariST-AliMeeting 数据集及新 BLEU 指标。
摘要：
面向会话录音的端到端语音翻译（ST）面临若干尚未得到充分研究的挑战，例如在缺少精确词级时间戳的情况下进行说话人分角色（SD），以及以流式方式处理重叠语音。在本工作中，我们提出 DiariST，这是首个流式 ST 与 SD 解决方案。它构建在基于神经转导器的流式 ST 系统之上，并集成了 token 级序列化输出训练和 t-vector，这两项技术最初是为多说话人语音识别开发的。由于该领域缺乏评估基准，我们通过将 AliMeeting 语料库的中文参考转录文本翻译成英文，构建了一个新的评估数据集 DiariST-AliMeeting。我们还提出了称为 speaker-agnostic BLEU 和 speaker-attributed BLEU 的新指标，用于在考虑 SD 准确率的同时衡量 ST 质量。与基于 Whisper 的离线系统相比，我们的系统在对重叠语音执行流式推理的同时，取得了强大的 ST 与 SD 能力。为促进这一新方向的研究，我们发布了评估数据、离线基线系统和评估代码。
英文摘要：
End-to-end speech translation (ST) for conversation recordings involves several under-explored challenges such as speaker diarization (SD) without accurate word time stamps and handling of overlapping speech in a streaming fashion. In this work, we propose DiariST, the first streaming ST and SD solution. It is built upon a neural transducer-based streaming ST system and integrates token-level serialized output training and t-vector, which were originally developed for multi-talker speech recognition. Due to the absence of evaluation benchmarks in this area, we develop a new evaluation dataset, DiariST-AliMeeting, by translating the reference Chinese transcriptions of the AliMeeting corpus into English. We also propose new metrics, called speaker-agnostic BLEU and speaker-attributed BLEU, to measure the ST quality while taking SD accuracy into account. Our system achieves a strong ST and SD capability compared to offline systems based on Whisper, while performing streaming inference for overlapping speech. To facilitate the research in this new direction, we release the evaluation data, the offline baseline systems, and the evaluation code.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
