LLaST:利用大语言模型改进的端到端语音翻译系统
LLaST: Improved End-to-end Speech Translation System Leveraged by Large Language Models
AI总结:
本文提出LLaST框架,通过架构设计、ASR增强训练、多语言数据增强和双LoRA优化,构建高性能LLM语音翻译系统,在CoVoST-2上表现优越。
摘要:
我们介绍了LLaST,一个用于构建高性能基于大语言模型的语音到文本翻译系统的框架。我们通过探索针对LLM定制的模型架构设计和优化技术,解决了端到端语音翻译(E2E ST)模型的局限性。我们的方法包括基于LLM的语音翻译架构设计、ASR增强训练、多语言数据增强和双LoRA优化。我们的方法在CoVoST-2基准上展示了优越的性能,并展示了由LLM驱动的卓越扩展能力。我们相信这种有效的方法将作为语音翻译的强有力基线,并为未来基于LLM的语音翻译框架的改进提供见解。我们在https://github.com/openaudiolab/LLaST发布了数据、代码和模型。
英文摘要:
We introduces LLaST, a framework for building high-performance Large Language model based Speech-to-text Translation systems. We address the limitations of end-to-end speech translation(E2E ST) models by exploring model architecture design and optimization techniques tailored for LLMs. Our approach includes LLM-based speech translation architecture design, ASR-augmented training, multilingual data augmentation, and dual-LoRA optimization. Our approach demonstrates superior performance on the CoVoST-2 benchmark and showcases exceptional scaling capabilities powered by LLMs. We believe this effective method will serve as a strong baseline for speech translation and provide insights for future improvements of the LLM-based speech translation framework. We release the data, code and models in https://github.com/openaudiolab/LLaST.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。