arXiv 2310.020502023-10-03cs.CLcs.CV

面向端到端语音翻译的大语言模型调优

Tuning Large language model for End-to-end Speech Translation

University of Information Engineering（信息工程大学）；Tsinghua University（清华大学）；Zhengzhou University（郑州大学）

Hao Zhang, Nianwen Si, Yaqi Chen, Wenlin Zhang, Xukui Yang, Dan Qu, Xiaolin Jiao

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对多模态大模型在端到端语音翻译任务上表现不佳的问题，提出由语音前端、适配器和LLM后端组成的LST模型，经两阶段训练后在MuST-C基准上刷新了英德、英法、英西方向的SOTA性能。

摘要：

随着大语言模型（LLM）的兴起，基于LLM的多模态模型已展现出巨大潜力。LLaSM、X-LLM、SpeechGPT等模型在理解和生成人类指令方面表现出令人瞩目的能力，但在面对端到端语音翻译（E2E-ST）这类跨语言、跨模态的复杂任务时，性能往往会出现下滑。在这类场景中，多模态模型的表现落后于单模态模型。\n本文提出LST——一种专为E2E-ST任务优化的大型多模态模型。LST由语音前端、适配器和LLM后端三部分组成，其训练分为两个阶段：（1）模态调整阶段，调优适配器使语音表示与文本嵌入空间对齐；（2）下游任务微调阶段，同时训练适配器与LLM模型，优化E2E-ST任务的性能。\n在MuST-C语音翻译基准上的实验结果表明，LST-13B在英德/英法/英西语言对上取得了30.39/41.55/35.33的BLEU分数，超越了此前的模型，创下新的 state-of-the-art 水平。此外，我们还对单模态模型选择及训练策略的影响进行了深入分析，为后续研究奠定了基础。我们将在审稿结束后开源代码与模型。

英文摘要：

With the emergence of large language models (LLMs), multimodal models based on LLMs have demonstrated significant potential. Models such as LLaSM, X-LLM, and SpeechGPT exhibit an impressive ability to comprehend and generate human instructions. However, their performance often falters when faced with complex tasks like end-to-end speech translation (E2E-ST), a cross-language and cross-modal translation task. In comparison to single-modal models, multimodal models lag behind in these scenarios. This paper introduces LST, a Large multimodal model designed to excel at the E2E-ST task. LST consists of a speech frontend, an adapter, and a LLM backend. The training of LST consists of two stages: (1) Modality adjustment, where the adapter is tuned to align speech representation with text embedding space, and (2) Downstream task fine-tuning, where both the adapter and LLM model are trained to optimize performance on the E2EST task. Experimental results on the MuST-C speech translation benchmark demonstrate that LST-13B achieves BLEU scores of 30.39/41.55/35.33 on En-De/En-Fr/En-Es language pairs, surpassing previous models and establishing a new state-of-the-art. Additionally, we conduct an in-depth analysis of single-modal model selection and the impact of training strategies, which lays the foundation for future research. We will open up our code and models after review.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。