arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.03352cs.CL

PEGRL: 通过后编辑引导的强化学习改进机器翻译

PEGRL: Improving Machine Translation by Post-Editing Guided Reinforcement Learning

National Key Laboratory for Novel Software Technology, Nanjing University（新型软件技术国家重点实验室，南京大学）；China Mobile Research Beijing, China（北京中国移动研究院）

Yunzhi Shen, Hao Zhou, Xin Huang, Xue Han, Junlan Feng, Shujian Huang

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出PEGRL框架，通过后编辑作为辅助任务稳定训练并引导整体优化，提升机器翻译性能。

摘要：

强化学习（RL）在基于大语言模型（LLM）的机器翻译中展现出强劲的潜力，近期方法如GRPO已取得显著进展；然而，面向翻译的RL仍然受到来自蒙特卡洛回报估计的噪声学习信号以及庞大的轨迹空间的挑战，后者倾向于全局探索而非细粒度的局部优化。我们引入PEGRL，一种两阶段的RL框架，利用后编辑作为辅助任务来稳定训练并引导整体优化。在每次迭代中，翻译输出被采样以构建后编辑输入，使后编辑阶段的回报估计能够受益于对当前翻译行为的条件化，同时共同支持全局探索和细粒度的局部优化。一个任务特定的加权方案进一步平衡翻译和后编辑目标的贡献，从而获得一个偏倚但更样本高效的估计器。在英语→芬兰语、英语→土耳其语以及英语↔中文的实验中，PEGRL在RL基线上表现出一致的提升，对于英语→土耳其语，其在COMET-KIWI上的性能与先进的LLM基系统（DeepSeek-V3.2）相当。我们的代码和一组代表性的预训练模型已公开在https://github.com/NJUNLP/peg-rl和https://huggingface.co/collections/DGME/pegrl。

英文摘要：

Reinforcement learning (RL) has shown strong promise for LLM-based machine translation, with recent methods such as GRPO demonstrating notable gains; nevertheless, translation-oriented RL remains challenged by noisy learning signals arising from Monte Carlo return estimation, as well as a large trajectory space that favors global exploration over fine-grained local optimization. We introduce \textbf{PEGRL}, a \textit{two-stage} RL framework that uses post-editing as an auxiliary task to stabilize training and guide overall optimization. At each iteration, translation outputs are sampled to construct post-editing inputs, allowing return estimation in the post-editing stage to benefit from conditioning on the current translation behavior, while jointly supporting both global exploration and fine-grained local optimization. A task-specific weighting scheme further balances the contributions of translation and post-editing objectives, yielding a biased yet more sample-efficient estimator. Experiments on English$\to$Finnish, English$\to$Turkish, and English$\leftrightarrow$Chinese show consistent gains over RL baselines, and for English$\to$Turkish, performance on COMET-KIWI is comparable to advanced LLM-based systems (DeepSeek-V3.2). Our code and a set of representative pretrained models are publicly available at \url{https://github.com/NJUNLP/peg-rl} and \url{https://huggingface.co/collections/DGME/pegrl}

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑