CoSTA：使用对齐语音-文本交错的语码转换语音翻译
CoSTA: Code-Switched Speech Translation using Aligned Speech-Text Interleaving
AI总结：
本文提出端到端模型 CoSTA，利用预训练 ASR 与 MT 模块及对齐语音-文本交错融合，解决印度语码转换语音英译的数据稀缺问题，并发布多语种评测基准，最高提升 3.5 BLEU。
摘要：
语码转换是印度等多语言社会中广泛存在的语言现象。由于数据集可用性有限，为语码转换语音构建语音到文本模型颇具挑战。在本文中，我们聚焦于将印度语言的语码转换语音进行口语翻译（ST）为英语文本的问题。我们提出了一种新的端到端模型架构 COSTA，该架构以预训练的自动语音识别（ASR）和机器翻译（MT）模块为支撑（这些模块在许多语言中更易获得）。语音和 ASR 文本表征通过对齐交错方案进行融合，并进一步作为输入馈送至预训练 MT 模块；随后使用合成创建的 ST 数据对整个流水线进行端到端口语翻译训练。我们还发布了一个新的评估基准，用于语码转换的孟加拉语-英语、印地语-英语、马拉地语-英语和泰卢固语-英语语音到英语文本的翻译。COSTA 显著优于许多具有竞争力的级联和端到端多模态基线，BLEU 分数最高提升 3.5 分。
英文摘要：
Code-switching is a widely prevalent linguistic phenomenon in multilingual societies like India. Building speech-to-text models for code-switched speech is challenging due to limited availability of datasets. In this work, we focus on the problem of spoken translation (ST) of code-switched speech in Indian languages to English text. We present a new end-to-end model architecture COSTA that scaffolds on pretrained automatic speech recognition (ASR) and machine translation (MT) modules (that are more widely available for many languages). Speech and ASR text representations are fused using an aligned interleaving scheme and are fed further as input to a pretrained MT module; the whole pipeline is then trained end-to-end for spoken translation using synthetically created ST data. We also release a new evaluation benchmark for code-switched Bengali-English, Hindi-English, Marathi-English and Telugu- English speech to English text. COSTA significantly outperforms many competitive cascaded and end-to-end multimodal baselines by up to 3.5 BLEU points.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
