奖励创造力：一种人类对齐的生成奖励模型用于故事创作中的强化学习
Rewarding Creativity: A Human-Aligned Generative Reward Model for Reinforcement Learning in Storytelling
AI总结：
本文提出了一种人类对齐的生成奖励模型和强化学习框架，用于提升故事创作的质量和稳定性。
摘要：
尽管大型语言模型（LLMs）能够生成流畅的文本，但生成高质量的创意故事仍然具有挑战性。强化学习（RL）提供了一个有前途的解决方案，但面临两个关键障碍：设计可靠的奖励信号以评估主观故事质量以及缓解训练不稳定。本文介绍了用于创意故事创作的强化学习（RLCS）框架，以系统地解决这两个挑战。首先，我们开发了一个生成奖励模型（GenRM），它能够提供多维分析和对故事偏好的显式推理，通过在演示上进行监督微调训练，随后通过GRPO方法在扩展的偏好数据上进行细化。其次，我们引入了一种基于熵的奖励塑造策略，该策略动态优先学习于自信的错误和不确定的正确预测，防止在已经掌握的模式上过拟合。实验表明，GenRM在人类创造力判断中实现了68%的匹配度，而RLCS在整体故事质量上显著优于包括Gemini-2.5-Pro在内的强大基线。这项工作提供了一个将RL应用于创意领域的实用流程，有效应对了奖励建模和训练稳定性的双重挑战。
英文摘要：
While Large Language Models (LLMs) can generate fluent text, producing high-quality creative stories remains challenging. Reinforcement Learning (RL) offers a promising solution but faces two critical obstacles: designing reliable reward signals for subjective storytelling quality and mitigating training instability. This paper introduces the Reinforcement Learning for Creative Storytelling (RLCS) framework to systematically address both challenges. First, we develop a Generative Reward Model (GenRM) that provides multi-dimensional analysis and explicit reasoning about story preferences, trained through supervised fine-tuning on demonstrations with reasoning chains distilled from strong teacher models, followed by GRPO-based refinement on expanded preference data. Second, we introduce an entropy-based reward shaping strategy that dynamically prioritizes learning on confident errors and uncertain correct predictions, preventing overfitting on already-mastered patterns. Experiments demonstrate that GenRM achieves 68\% alignment with human creativity judgments, and RLCS significantly outperforms strong baselines including Gemini-2.5-Pro in overall story quality. This work provides a practical pipeline for applying RL to creative domains, effectively navigating the dual challenges of reward modeling and training stability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
