CoCA：基于评分标准的强化学习的准则感知信用分配
CoCA: Criterion-Aware Credit Assignment for Rubric-Based Reinforcement Learning
AI总结：
针对基于评分标准的强化学习中结构化监督未被充分利用的问题，提出准则感知信用分配方法 CoCA，在医学和科学基准上实现平均 4.2% 的性能提升，提升了开放式任务的学习效果。
摘要：
基于评分标准的强化学习（RL）依据明确准则评估响应以构建奖励，为在开放式任务上训练大语言模型提供了可行途径。然而，标准方法将准则级反馈压缩为标量奖励并分配均匀的 token 优势，导致结构化评分标准监督未被充分利用。近期研究尝试将评分标准作为特权信息推导自蒸馏目标以获取更细粒度的指导，但生成的学习信号未基于准则结果，可能导致奖励退化。这促使我们将问题重新表述为准则感知信用分配，其中特权评分标准信息作为信用分配的相关性线索，而非模仿目标的来源。据此，我们提出 CoCA，它从组相对结果中推导准则级信用，并基于准则诱导的分布偏移将其分配到各个 token，从而产生既基于结果又细粒度的学习信号。实验表明，在医学和科学基准上，CoCA 相较于标准的基于评分标准的 RL 平均提升了 4.2%，证明其在开放式任务上的学习更有效。
英文摘要：
Rubric-based reinforcement learning (RL) evaluates responses against explicit criteria to construct rewards, offering a viable way to train large language models on open-ended tasks. However, standard methods compress criterion-level feedback into scalar rewards and assign uniform token advantages, leaving structured rubric supervision underutilized. Recent work seeks finer-grained guidance by treating rubrics as privileged information to derive self-distillation targets, but the resulting learning signal is not grounded in criterion outcomes and can lead to reward degradation. This motivates us to reframe the problem as criterion-aware credit assignment, where privileged rubric information serves as a relevance cue for credit allocation rather than a source of imitation targets. Accordingly, we propose CoCA, which derives criterion-level credits from group-relative outcomes and allocates them across tokens based on criterion-induced distribution shifts, yielding both outcome-grounded and fine-grained learning signals. Empirically, CoCA achieves an average gain of +4.2% over standard rubric-based RL across medicine and science benchmarks, demonstrating more effective learning on open-ended tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
