翻译器理解你的意思：由语音翻译辅助的端到端口语理解
The Interpreter Understands Your Meaning: End-to-end Spoken Language Understanding Aided by Speech Translation
AI总结：
本文提出利用语音翻译作为预训练任务，提升端到端口语理解模型在单语和跨语言场景下的性能，并在多个基准上验证了其有效性。
摘要：
端到端口语理解（SLU）即使在当前基于文本和语音的大型预训练语言模型下仍然难以实现，尤其是在多语言场景中。机器翻译已被确立为一种强大的文本预训练目标，因为它使模型能够捕捉输入话语的高级语义以及不同语言之间的关联，这对于处理较低层次声学帧的语音模型来说是非常理想的。特别受跨语言SLU任务的启发，我们证明语音翻译（ST）任务是在内语言和跨语言场景下对端到端SLU语音模型进行预训练的一种有效手段。通过引入ST，我们的模型在单语和多语意图分类以及口语问答任务上，使用SLURP、MINDS-14和NMSQA基准，达到了优于基线模型的性能。为了验证我们方法的有效性，我们还从合成和真实来源创建了新的基准数据集，用于语音摘要以及从英语到法语或西班牙语的低资源/零样本迁移。我们进一步展示了保留ST预训练任务知识对于更好的下游性能的价值，可能通过使用贝叶斯迁移正则化器来实现。
英文摘要：
End-to-end spoken language understanding (SLU) remains elusive even with current large pretrained language models on text and speech, especially in multilingual cases. Machine translation has been established as a powerful pretraining objective on text as it enables the model to capture high-level semantics of the input utterance and associations between different languages, which is desired for speech models that work on lower-level acoustic frames. Motivated particularly by the task of cross-lingual SLU, we demonstrate that the task of speech translation (ST) is a good means of pretraining speech models for end-to-end SLU on both intra- and cross-lingual scenarios. By introducing ST, our models reach higher performance over baselines on monolingual and multilingual intent classification as well as spoken question answering using SLURP, MINDS-14, and NMSQA benchmarks. To verify the effectiveness of our methods, we also create new benchmark datasets from both synthetic and real sources, for speech summarization and low-resource/zero-shot transfer from English to French or Spanish. We further show the value of preserving knowledge for the ST pretraining task for better downstream performance, possibly using Bayesian transfer regularizers.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
