arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.20059cs.CL
收藏

基于偏好对齐是否总是提升基于大语言模型的翻译质量的最佳选项?一项实证分析

Is Preference Alignment Always the Best Option to Enhance LLM-Based Translation? An Empirical Analysis

Artefact Research Center(Artefact 研究中心);Unbabel;Equall;MICS, CentraleSupélec, Université Paris-Saclay(巴黎-萨克雷大学中央理工-高等电力学院数学与计算机科学实验室);Instituto de Telecomunicações(电信研究所);Instituto Superior Técnico & Universidade de Lisboa(里斯本大学高等技术学院)

Hippolyte Gisserot-Boukhlef, Ricardo Rei, Emmanuel Malherbe, Céline Hudelot, Pierre Colombo, Nuno M. Guerreiro

AI总结:

本研究通过实证分析探讨了基于偏好的对齐在提升大语言模型翻译质量中的效果,发现CPO在高质量数据上表现优异,但可能在下游度量上存在不稳定性,同时基础模型生成翻译的表现与多个外部系统相当且更一致。

摘要:

神经度量在机器翻译(MT)评估中变得越来越重要,因为它们比传统词汇度量在与人类判断的相关性上更优越。研究人员因此利用质量导向的解码策略使用神经度量,取得了比基于似然的方法更好的结果。随着大语言模型(LLMs)的兴起,基于偏好的对齐技术因其通过直接优化模型权重来提高翻译质量的潜力而受到关注。本研究聚焦于对比偏好优化(CPO),并进行了广泛的实验来评估基于偏好的对齐对翻译质量的影响。我们的发现表明,虽然CPO在高质量数据上始终优于监督微调(SFT)在对齐度量上,但在下游评估度量上可能会导致不稳定,特别是在神经和词汇度量之间。此外,我们证明仅依赖基础模型生成候选翻译的表现可以与使用多个外部系统相媲美,同时确保在下游度量上的一致性更好。

英文摘要:

Neural metrics for machine translation (MT) evaluation have become increasingly prominent due to their superior correlation with human judgments compared to traditional lexical metrics. Researchers have therefore utilized neural metrics through quality-informed decoding strategies, achieving better results than likelihood-based methods. With the rise of Large Language Models (LLMs), preference-based alignment techniques have gained attention for their potential to enhance translation quality by optimizing model weights directly on preferences induced by quality estimators. This study focuses on Contrastive Preference Optimization (CPO) and conducts extensive experiments to evaluate the impact of preference-based alignment on translation quality. Our findings indicate that while CPO consistently outperforms Supervised Fine-Tuning (SFT) on high-quality data with regard to the alignment metric, it may lead to instability across downstream evaluation metrics, particularly between neural and lexical ones. Additionally, we demonstrate that relying solely on the base model for generating candidate translations achieves performance comparable to using multiple external systems, while ensuring better consistency across downstream metrics.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑