arXiv 2605.080612026-05-08cs.AI

基于评分标准的强化学习：用于通用推理的结构化评判奖励

Rubric-Grounded RL: Structured Judge Rewards for Generalizable Reasoning

Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室）

Manish Bhattarai, Ismael Boureima, Nishath Rajiv Ranasinghe, Scott Pakin, Dan O'Malley

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出基于评分标准的强化学习框架，通过LLM评判器生成多标准奖励，提升模型在通用推理任务中的表现。

摘要：

我们主张将奖励分解为加权、可验证的标准，并利用LLM评判器对这些标准进行评分，提供部分信用优化信号：而不是二元结果或单一整体评分，每个响应会根据多个任务特定的标准进行评分。我们正式化了基于评分标准的强化学习（RL）：一种框架，其中策略在由冻结的LLM评判器生成的结构化、多标准奖励下进行优化，该评判器基于辅助信息对策略从未见过的领域进行条件判断。我们通过从科学和技术信息办公室（OSTI）衍生的约10万份科学和技术文档语料库中推导出评分标准，并使用Group Relative Policy Optimization（GRPO）训练Llama-3.1-8B-Instruct来实例化该框架。通过GRPO训练，模型在保留的评分评估中实现了71.7%的归一化奖励。GRPO调优的策略在四个未从训练语料库中衍生的推理基准测试中也优于基础模型——GSM8K、MATH、GPQA Main和GPQA Diamond。这些结果提供了证据，表明结构化、文档基础的奖励可以提高保留的评分表现，并诱导出超出训练环境所用语料库的可转移推理行为。

英文摘要：

We argue that decomposing reward into weighted, verifiable criteria and using an LLM judge to score them provides a partial-credit optimization signal: instead of a binary outcome or a single holistic score, each response is graded along multiple task-specific criteria. We formalize \emph{rubric-grounded reinforcement learning (RL)}: a framework in which the policy is optimized against a structured, multi-criterion reward produced by a frozen LLM judge that conditions on auxiliary grounding the policy never sees. We instantiate the framework by deriving rubrics from an Office of Scientific and Technical Information (OSTI)-derived corpus of roughly 100,000 scientific and technical documents and training Llama-3.1-8B-Instruct with Group Relative Policy Optimization (GRPO). With GRPO-based training, the model achieves $71.7\%$ normalized reward on held-out rubric evaluation. The GRPO-tuned policy also improves over the base model on four reasoning benchmarks not derived from the training corpus -- GSM8K, MATH, GPQA Main, and GPQA Diamond. These results provide evidence that structured, document-grounded rewards can improve held-out rubric performance and induce transferable reasoning behaviors beyond the corpus used to construct the training environment.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

AI总结：

摘要：

英文摘要：

相关论文