激发小型语言模型的创意写作：LLM-as-a-Judge 与多智能体精炼奖励的比较
Igniting Creative Writing in Small Language Models: LLM-as-a-Judge versus Multi-Agent Refined Rewards
AI总结：
本文在 RLAIF 框架下比较多智能体拒绝采样奖励模型与原则引导、对抗反思优化的 LLM-as-a-Judge，证明后者更能提升 7B SLM 的中文祝福语创意生成质量与训练效率。
摘要：
大型语言模型（LLM）已展现出卓越的创意写作能力，但其巨大的计算需求阻碍了广泛应用。增强小型语言模型（SLM）提供了一种有前景的替代方案，但监督微调（SFT）等现有方法在新颖性方面表现不佳，而基于人类反馈的强化学习（RLHF）成本高昂。本文在基于 AI 反馈的强化学习（RLAIF）框架内探索两种不同的 AI 驱动奖励策略，以激发一个 7B 参数 SLM 的创意写作能力，具体用于生成中文祝福语。第一种策略采用一个在高质量偏好数据上训练的 RM，这些数据由专为创意任务设计的新型多智能体拒绝采样框架筛选得到。第二种、也更具创新性的策略利用原则引导的 LLM-as-a-Judge 直接提供奖励信号，其奖励函数通过带有反思机制的对抗训练方案进行优化。综合实验表明，尽管两种方法都相较基线显著提升了创意输出，但原则引导的 LLM-as-a-Judge 可证明地产生了更优的生成质量。此外，它在训练效率和降低对人工标注数据依赖方面具有显著优势，为构建创意 SLM 提供了一条更具可扩展性且更有效的路径。我们的自动化评估方法也与人类判断表现出很强的一致性。代码和数据公开发布于 https://github.com/weixiaolong94-hub/Igniting-Creative-Writing-in-Small-Language-Models。
英文摘要：
Large Language Models (LLMs) have demonstrated remarkable creative writing capabilities, yet their substantial computational demands hinder widespread use. Enhancing Small Language Models (SLMs) offers a promising alternative, but current methods like Supervised Fine-Tuning (SFT) struggle with novelty, and Reinforcement Learning from Human Feedback (RLHF) is costly. This paper explores two distinct AI-driven reward strategies within a Reinforcement Learning from AI Feedback (RLAIF) framework to ignite the creative writing of a 7B-parameter SLM, specifically for generating Chinese greetings. The first strategy employs a RM trained on high-quality preference data curated by a novel multi-agent rejection sampling framework designed for creative tasks. The second, more novel strategy utilizes a principle-guided LLM-as-a-Judge, whose reward function is optimized via an adversarial training scheme with a reflection mechanism, to directly provide reward signals. Comprehensive experiments reveal that while both approaches significantly enhance creative output over baselines, the principle-guided LLM-as-a-Judge demonstrably yields superior generation quality. Furthermore, it offers notable advantages in training efficiency and reduced dependency on human-annotated data, presenting a more scalable and effective path towards creative SLMs. Our automated evaluation methods also exhibit strong alignment with human judgments. Our code and data are publicly available at https://github.com/weixiaolong94-hub/Igniting-Creative-Writing-in-Small-Language-Models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
