EvoRubric: 用于开放生成的自我进化评分标准驱动强化学习
EvoRubric: Self-Evolving Rubric-Driven RL for Open-Ended Generation
AI总结：
提出EvoRubric，一种单策略共进化强化学习框架，通过动态交替生成响应和评分标准，并引入多层验证机制，解决开放生成任务中缺乏明确奖励的问题，在医学、写作和科学领域超越传统静态和外部LLM驱动方法。
摘要：
强化学习（RL）在可验证领域显著提升了大型语言模型（LLM），但由于缺乏明确的奖励，为开放生成任务对齐模型仍然极具挑战性。当前的基于评分标准的RL方法通过使用显式标准来缓解这一问题；然而，它们严重依赖于静态的人工标注评分标准，这不可避免地导致策略滞后，或者依赖昂贵的外部专有模型进行动态更新。在本文中，我们提出了EvoRubric，一种新颖的单策略共进化RL框架，消除了对静态标准和外部评分标准生成器的依赖。通过将响应生成和评分标准生成统一在单一参数化策略下，EvoRubric在推理器和评分标准生成器之间动态交替。为了防止奖励黑客攻击并确保生成信号的可信度，我们引入了一个多层验证流程，包括元验证器、零方差剪枝和留一法同行共识机制。经过验证的标准被动态归档到记忆池中，产生密集的多目标奖励，以持续共同优化两个角色。在医学、写作和科学领域的广泛实验表明，EvoRubric始终优于传统的静态和外部LLM驱动的对齐方法。值得注意的是，我们的框架与人类专家先验知识兼容。当使用专家标注的评分标准初始化时，EvoRubric能够进一步发现新颖的、有区分度的维度，从而实现比仅依赖静态专家标注更好的性能。
英文摘要：
Reinforcement Learning (RL) has advanced Large Language Models (LLMs) in verifiable domains, while open-ended generation remains challenging due to the absence of definitive rewards. Rubric-based RL provides explicit evaluation criteria, but learning to construct these criteria remains challenging when final-answer correctness is not verifiable. We propose EvoRubric, a co-evolutionary RL framework that combines criterion-validity feedback, response discrimination, and peer agreement to learn rubrics for open-ended generation. A shared policy acts as both a Reasoner and a Rubric Generator, using its current responses and historical rubrics to discover new evaluation dimensions. To combine adaptive rubric discovery with a stable validity check, a frozen copy of the initial policy serves as the Meta-Verifier, while a frozen Grader scores responses against the retained criteria. Discriminative feedback, Leave-One-Out peer consensus, and a persistent memory pool transform this feedback into complementary rewards that jointly optimize both policy roles, closing the loop between response improvement and rubric discovery. EvoRubric improves over matched static and external evolving-rubric baselines across five benchmarks in Medical, Writing, and Science. Across three training seeds, it achieves five-benchmark averages of 56.28 at 8B and 61.13 at 14B, exceeding the strongest matched baselines by 3.09 and 2.06 points, respectively. Human audits assess criterion validity and response quality, and experiments with expert-initialized rubrics demonstrate compatibility with human priors.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
