arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.15061cs.CLcs.LG
收藏

面向领域特定机器翻译的大语言模型微调

Fine-tuning Large Language Models for Domain-specific Machine Translation

School of Informatics, Xiamen University(厦门大学信息学院);xFusion Digital Technologies Co., Ltd(星网锐捷数字技术有限公司)

Jiawei Zheng, Hanghai Hong, Feiyan Liu, Xiaoli Wang, Jingsong Su, Yonggui Liang, Shikai Wu

AI总结:

本文提出DragFT微调框架,通过词典增强提示、RAG示例选择和少样本微调,显著提升大语言模型在领域特定机器翻译上的性能,超越GPT-3.5和GPT-4o。

摘要:

大语言模型(LLMs)在领域特定机器翻译(MT)中展现出巨大潜力。然而,一个主要问题是,在通用领域语料上预训练的LLMs可能因缺乏领域特定知识而无法很好地泛化到特定领域。为解决这一问题,本文聚焦于通过提供高质量训练数据集并提出一种新颖的微调框架DragFT,来增强LLMs的领域特定机器翻译能力。DragFT通过三种技术增强LLMs:(i)词典增强提示将词典信息整合到提示中,以改善领域特定术语的翻译;(ii)基于RAG的少样本示例选择提供模拟领域和风格特征的高质量示例;(iii)使用领域内示例进行少样本微调进一步提升了性能。我们在三个具有13B训练参数的知名LLM骨干上部署DragFT以验证其有效性。在三个领域特定数据集上的结果表明,DragFT实现了显著的性能提升,并展现出优于GPT-3.5和GPT-4o等先进模型的卓越性能。DragFT相较于现有LLMs的巨大性能改进可归因于在缓解噪声的同时整合了相关知识。

英文摘要:

Large language models (LLMs) have shown great potential in domain-specific machine translation (MT). However, one major issue is that LLMs pre-trained on general domain corpus might not generalize well to specific domains due to the lack of domain-specific knowledge. To address this issue, this paper focuses on enhancing the domain-specific MT capability of LLMs, by providing high-quality training datasets and proposing a novel fine-tuning framework denoted by DragFT. DragFT augments LLMs via three techniques: (i) Dictionary-enhanced prompting integrates dictionary information into prompts to improve the translation of domain-specific terminology.; (ii) RAG-based few-shot example selection provides high-quality examples that simulate both the domain and style characteristics; (iii) Fine-tuning with few-shot examples further enhances performance when using in-domain examples. We deploy DragFT on three well-known LLM backbones with 13B training parameters to validate its effectiveness. The results on three domain-specific datasets show that DragFT achieves a significant performance boost and shows superior performance compared to advanced models such as GPT-3.5 and GPT-4o. The drastic performance improvement of DragFT over existing LLMs can be attributed to incorporating relevant knowledge while mitigating noise.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑