利用检索与演示方法优化直接语音翻译中的罕见词准确率
Optimizing Rare Word Accuracy in Direct Speech Translation with a Retrieval-and-Demonstration Approach
AI总结：
本文提出检索与演示方法，通过改造直接语音翻译模型并结合跨模态检索器获取相似示例，显著提升罕见词翻译准确率。
摘要：
直接语音翻译（ST）模型常常难以处理罕见词。这些词的错误翻译可能带来严重后果，影响翻译质量和用户信任。虽然由于学习信号稀疏，罕见词翻译对神经模型而言本质上具有挑战性，但现实场景中通常可以获取相似主题下过往录音的翻译。为了利用这些宝贵资源，我们提出一种检索与演示方法，以提升直接ST模型中的罕见词翻译准确率。首先，我们改造现有ST模型，使其能够结合检索到的示例进行罕见词翻译，从而让模型从前置示例中受益，类似于上下文学习。随后，我们开发了一个跨模态（语音到语音、语音到文本、文本到文本）检索器来定位合适的示例。我们证明，标准ST模型可以被有效改造以利用示例进行罕见词翻译，在使用黄金示例时将罕见词翻译准确率较基线提升17.6%，使用检索示例时提升8.5%。此外，我们的语音到语音检索方法优于其他模态，并对未见过的说话人表现出更高的鲁棒性。我们的代码已公开（https://github.com/SiqiLii/Retrieve-and-Demonstration-ST）。
英文摘要：
Direct speech translation (ST) models often struggle with rare words. Incorrect translation of these words can have severe consequences, impacting translation quality and user trust. While rare word translation is inherently challenging for neural models due to sparse learning signals, real-world scenarios often allow access to translations of past recordings on similar topics. To leverage these valuable resources, we propose a retrieval-and-demonstration approach to enhance rare word translation accuracy in direct ST models. First, we adapt existing ST models to incorporate retrieved examples for rare word translation, which allows the model to benefit from prepended examples, similar to in-context learning. We then develop a cross-modal (speech-to-speech, speech-to-text, text-to-text) retriever to locate suitable examples. We demonstrate that standard ST models can be effectively adapted to leverage examples for rare word translation, improving rare word translation accuracy over the baseline by 17.6% with gold examples and 8.5% with retrieved examples. Moreover, our speech-to-speech retrieval approach outperforms other modalities and exhibits higher robustness to unseen speakers. Our code is publicly available (https://github.com/SiqiLii/Retrieve-and-Demonstration-ST).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
