RLMR：面向创意写作的混合奖励强化学习
RLMR: Reinforcement Learning with Mixed Rewards for Creative Writing
AI总结：
针对创意写作中主观质量与客观约束难以兼顾的问题，提出RLMR方法，通过动态混合奖励机制在GRPO训练中惩罚违反约束的样本，在指令遵循和写作质量上均取得显著提升。
摘要：
大型语言模型在创意写作应用中被广泛使用。创意写作需要在主观写作质量（如文学性和情感表达）与客观约束遵循（如格式要求和字数限制）之间取得平衡。现有方法难以兼顾这两个方面：单一奖励策略无法同时提升两种能力，而固定权重的混合奖励方法缺乏对不同写作场景的适应能力。为解决这一问题，我们提出了混合奖励强化学习（RLMR），利用一个动态混合奖励系统，该系统由评估主观写作质量的写作奖励模型和评估客观约束遵循的约束验证模型组成。约束遵循奖励权重根据采样组内的写作质量动态调整，确保违反约束的样本在GRPO中获得负优势，从而在训练过程中受到惩罚，这是本方法的关键创新。我们在从8B到72B参数的多种模型家族上进行了自动化和人工评估。此外，我们构建了一个名为WriteEval的真实写作基准，用于全面评估。结果表明，我们的方法在指令遵循（IFEval从83.36%提升至86.65%）和写作质量（在WriteEval上人工专家成对评估中达到72.75%的胜率）两方面均取得了一致的改进。据我们所知，RLMR是首个在在线强化学习训练中将主观偏好与客观验证相结合的工作，为多维创意写作优化提供了有效解决方案。
英文摘要：
Large language models are extensively utilized in creative writing applications. Creative writing requires a balance between subjective writing quality (e.g., literariness and emotional expression) and objective constraint following (e.g., format requirements and word limits). Existing methods find it difficult to balance these two aspects: single reward strategies fail to improve both abilities simultaneously, while fixed-weight mixed-reward methods lack the ability to adapt to different writing scenarios. To address this problem, we propose Reinforcement Learning with Mixed Rewards (RLMR), utilizing a dynamically mixed reward system from a writing reward model evaluating subjective writing quality and a constraint verification model assessing objective constraint following. The constraint following reward weight is adjusted dynamically according to the writing quality within sampled groups, ensuring that samples violating constraints get negative advantage in GRPO and thus penalized during training, which is the key innovation of this proposed method. We conduct automated and manual evaluations across diverse model families from 8B to 72B parameters. Additionally, we construct a real-world writing benchmark named WriteEval for comprehensive evaluation. Results illustrate that our method achieves consistent improvements in both instruction following (IFEval from 83.36% to 86.65%) and writing quality (72.75% win rate in manual expert pairwise evaluations on WriteEval). To the best of our knowledge, RLMR is the first work to combine subjective preferences with objective verification in online RL training, providing an effective solution for multi-dimensional creative writing optimization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
