Writer-R1: 通过记忆增强的回放策略优化提升大语言模型的生成写作
Writer-R1: Enhancing Generative Writing in LLMs via Memory-augmented Replay Policy Optimization
AI总结：
本文提出基于Grounded Theory的多智能体协作流程，结合记忆增强的回放策略优化算法，实现生成写作的可控迭代改进和端到端优化，实验表明其性能接近人工标注。
摘要：
作为典型的开放性生成任务，创造性写作缺乏可验证的参考答案，长期以来限制了奖励建模和自动评估，由于人工标注成本高、评估偏见和粗略反馈信号。为解决这些挑战，本文首先设计基于Grounded Theory的多智能体协作流程，进行问题的维度分解和层次归纳，动态产生可解释和可重用的细粒度标准。此外，我们提出记忆增强的回放策略优化（MRPO）算法：一方面，无需额外训练，MRPO引导模型基于动态标准进行自我反思，实现受控的迭代改进；另一方面，我们采用结合监督微调与强化学习的训练范式，将评估标准转化为奖励信号，实现端到端优化。实验结果表明，自动构建的标准在性能上与人工标注相当。使用此方法训练的Writer-R1-4B模型在多个创造性写作任务中优于基线模型，并超越了一些1000亿参数的开源模型。
英文摘要：
As a typical open-ended generation task, creative writing lacks verifiable reference answers, which has long constrained reward modeling and automatic evaluation due to high human annotation costs, evaluative bias, and coarse feedback signals. To address these challenges, this paper first designs a multi-agent collaborative workflow based on Grounded Theory, performing dimensional decomposition and hierarchical induction of the problem to dynamically produce interpretable and reusable fine-grained criteria. Furthermore, we propose the Memory-augmented Replay Policy Optimization (MRPO) algorithm: on the one hand, without additional training, MRPO guides models to engage in self-reflection based on dynamic criteria, enabling controlled iterative improvement; on the other hand, we adopt the training paradigm that combines supervised fine-tuning with reinforcement learning to convert evaluation criteria into reward signals, achieving end-to-end optimization. Experimental results demonstrate that the automatically constructed criteria achieve performance gains comparable to human annotations. Writer-R1-4B models trained with this approach outperform baselines across multiple creative writing tasks and surpass some 100B+ parameter open-source models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
