EMMeTT：高效的多模态机器翻译训练
EMMeTT: Efficient Multimodal Machine Translation Training
AI总结：
本文提出多模态训练框架EMMeTT，通过均衡采样、高效数据迭代和2D分桶方案提升训练效率，使模型在保留神经机器翻译能力的同时，在自动语音翻译任务上超越基线。
摘要：
对基础语言模型进行模态扩展的日益关注，引发了对最有效且高效的多模态训练方法的讨论。本工作聚焦于神经机器翻译（NMT），提出了一种Speech-LLM的联合多模态训练机制以纳入自动语音翻译（AST）。我们研究了两种不同的基础模型架构，即decoder-only GPT和encoder-decoder T5，并使用Canary-1B的语音编码器进行扩展。为处理联合多模态训练，我们提出了一种名为EMMeTT的新颖训练框架。EMMeTT通过以下方式提升训练效率：跨语言、数据集和模态的均衡采样；高效的序列数据迭代；以及一种用于多模态数据的新型2D分桶方案，并辅以批大小优化器（OOMptimizer）。我们表明，多模态训练对两种架构均有持续帮助。此外，使用EMMeTT训练的SALM-T5保留了原始NMT能力，同时在FLORES和FLEURS的四语言子集上优于AST基线。最终的多模态翻译模型同时产生了强大的文本和语音翻译结果。
英文摘要：
A rising interest in the modality extension of foundation language models warrants discussion on the most effective, and efficient, multimodal training approach. This work focuses on neural machine translation (NMT) and proposes a joint multimodal training regime of Speech-LLM to include automatic speech translation (AST). We investigate two different foundation model architectures, decoder-only GPT and encoder-decoder T5, extended with Canary-1B's speech encoder. To handle joint multimodal training, we propose a novel training framework called EMMeTT. EMMeTT improves training efficiency with the following: balanced sampling across languages, datasets, and modalities; efficient sequential data iteration; and a novel 2D bucketing scheme for multimodal data, complemented by a batch size optimizer (OOMptimizer). We show that a multimodal training consistently helps with both architectures. Moreover, SALM-T5 trained with EMMeTT retains the original NMT capability while outperforming AST baselines on four-language subsets of FLORES and FLEURS. The resultant Multimodal Translation Model produces strong text and speech translation results at the same time.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
