arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.15415cs.CL
收藏

LLaST:利用大语言模型改进的端到端语音翻译系统

LLaST: Improved End-to-end Speech Translation System Leveraged by Large Language Models

The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shanghai AI Laboratory(上海人工智能实验室);Nara Institute of Science and Technology(奈良先端科学技术大学院大学)

Xi Chen, Songyang Zhang, Qibing Bai, Kai Chen, Satoshi Nakamura

AI总结:

本文提出LLaST框架,通过架构设计、ASR增强训练、多语言数据增强和双LoRA优化,构建高性能LLM语音翻译系统,在CoVoST-2上表现优越。

摘要:

我们介绍了LLaST,一个用于构建高性能基于大语言模型的语音到文本翻译系统的框架。我们通过探索针对LLM定制的模型架构设计和优化技术,解决了端到端语音翻译(E2E ST)模型的局限性。我们的方法包括基于LLM的语音翻译架构设计、ASR增强训练、多语言数据增强和双LoRA优化。我们的方法在CoVoST-2基准上展示了优越的性能,并展示了由LLM驱动的卓越扩展能力。我们相信这种有效的方法将作为语音翻译的强有力基线,并为未来基于LLM的语音翻译框架的改进提供见解。我们在https://github.com/openaudiolab/LLaST发布了数据、代码和模型。

英文摘要:

We introduces LLaST, a framework for building high-performance Large Language model based Speech-to-text Translation systems. We address the limitations of end-to-end speech translation(E2E ST) models by exploring model architecture design and optimization techniques tailored for LLMs. Our approach includes LLM-based speech translation architecture design, ASR-augmented training, multilingual data augmentation, and dual-LoRA optimization. Our approach demonstrates superior performance on the CoVoST-2 benchmark and showcases exceptional scaling capabilities powered by LLMs. We believe this effective method will serve as a strong baseline for speech translation and provide insights for future improvements of the LLM-based speech translation framework. We release the data, code and models in https://github.com/openaudiolab/LLaST.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑