从教师模型细化进行强化学习：机器翻译中的渐进模仿学习
RL from Teacher-Model Refinement: Gradual Imitation Learning for Machine Translation
AI总结：
本文提出RLfR方法，通过教师模型细化和复合奖励机制提升机器翻译的语义质量和泛化能力。
摘要：
针对机器翻译（MT）中偏好学习方法（如直接偏好优化DPO）依赖大规模精心编纂的偏好三元组且泛化能力有限的问题，本文提出了一种基于教师模型细化的强化学习方法（RLfR）。该方法将静态三元组替换为由冻结教师模型生成的在线策略条件细化。在每一步中，演员生成候选翻译，教师对每个草稿进行最小局部编辑，演员通过结合缩放后的负编辑距离（用于词汇和结构保真度）与COMET（用于语义充分性）的复合奖励机制进行强化，以缩小差距。这种形式生成了稳定且模型感知的学习信号，无需显式偏好数据集。在FLORES-200（英语到德语、西班牙语、中文、韩语和日语）上的实验表明，RLfR在MT-SFT、DPO和固定参考RL基线中表现一致更优，提升了语义质量和实体保留，并在基于LLM的评估中也实现了更优表现。
英文摘要：
Preference-learning methods for machine translation (MT), such as Direct Preference Optimization (DPO), have shown strong gains but typically rely on large, carefully curated preference triplets and often struggle to generalize beyond their tuning domains. We propose Reinforcement Learning from Teacher-Model Refinement (RLfR), which replaces static triplets with on-policy, actor-conditioned refinements produced by a frozen teacher. At each step, the actor samples candidate translations, the teacher performs a minimal local edit of each draft, and the actor is reinforced to close the gap using a composite reward that combines scaled negative edit distance for lexical and structural fidelity with COMET for semantic adequacy. This formulation yields a stable, model-aware learning signal without requiring explicit preference datasets. Experiments on FLORES-200 (English to German, Spanish, Chinese, Korean, and Japanese) show that RLfR consistently outperforms strong MT-SFT, DPO, and fixed-reference RL baselines, improving semantic quality and entity preservation, and also achieves superior performance under LLM-based judge evaluations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
