arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.03145cs.CL
收藏

通过在平行数据上持续预训练提升大语言模型的翻译准确性

Enhancing Translation Accuracy of Large Language Models through Continual Pre-Training on Parallel Data

University of Tsukuba(筑波大学);NTT Corporation(NTT公司)

Minato Kondo, Takehito Utsuro, Masaaki Nagata

AI总结:

本文提出先在大规模平行数据上持续预训练、再用少量高质量平行数据微调大语言模型的两阶段翻译方法,并证明交替排列源目标句、保持预训练与推理语序一致及为源句加标签可显著提升日英互译准确性。

摘要:

本文提出了一种两阶段训练方法:先让预训练大语言模型在平行数据上进行持续预训练,再使用少量高质量平行数据进行监督微调。为验证所提方法的有效性,我们使用一个3.8B参数模型和八种不同格式的平行数据进行了持续预训练,并在十三个测试集上评估了日译英和英译日翻译。结果表明,在持续预训练中使用平行数据时,交替排列源语言句子和目标语言句子至关重要。此外,我们证明只有当持续预训练数据与推理时源语言和目标语言句子的顺序一致时,相应翻译方向的翻译准确性才会提升。我们还证明,与监督编码器-解码器模型相比,基于大语言模型的翻译模型在翻译口语方面更稳健,并且在使用更少训练数据时也能达到更高准确性。我们还发现,当持续预训练数据由交错的源语言句子和目标语言句子组成,并且为源语言句子添加标签时,能够取得最高的翻译准确性。

英文摘要:

In this paper, we propose a two-phase training approach where pre-trained large language models are continually pre-trained on parallel data and then supervised fine-tuned with a small amount of high-quality parallel data. To investigate the effectiveness of our proposed approach, we conducted continual pre-training with a 3.8B-parameter model and parallel data across eight different formats. We evaluate these methods on thirteen test sets for Japanese-to-English and English-to-Japanese translation. The results demonstrate that when utilizing parallel data in continual pre-training, it is essential to alternate between source and target sentences. Additionally, we demonstrated that the translation accuracy improves only for translation directions where the order of source and target sentences aligns between continual pre-training data and inference. In addition, we demonstrate that the LLM-based translation model is more robust in translating spoken language and achieves higher accuracy with less training data compared to supervised encoder-decoder models. We also show that the highest accuracy is achieved when the data for continual pre-training consists of interleaved source and target sentences and when tags are added to the source sentences.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑