SibylSense：通过记忆调优和对抗探测实现自适应评分标准学习
SibylSense: Adaptive Rubric Learning via Memory Tuning and Adversarial Probing
AI总结：
SibylSense通过记忆调优和对抗探测实现自适应评分标准学习，提升开放性任务下的强化学习性能。
摘要：
设计与开放性生成对齐且稳健的奖励仍然是训练后强化学习的关键障碍。评分标准提供结构化、可解释的监督，但扩大评分标准构建的难度在于：专家评分标准成本高，提示生成的评分标准往往表面化或不一致，而固定池辨别性评分标准容易饱和和漂移，导致奖励黑客问题。我们提出了SibylSense，一种推理时的学习方法，通过可调的记忆银行来适应冻结的评分标准生成器。记忆通过基于验证者的项目奖励进行更新，这些奖励通过少量示例的参考-候选答案辨别差距来测量。SibylSense交替进行记忆调优和评分标准对抗性策略更新，生成满足评分标准的候选答案，缩小辨别差距并推动评分标准生成器捕捉新的质量维度。在两个开放性任务上的实验表明，SibylSense产生的评分标准更具辨别性，并在静态和非自适应基线之上提升了下游强化学习性能。
英文摘要：
Designing aligned and robust rewards for open-ended generation remains a key barrier to RL post-training. Rubrics provide structured, interpretable supervision, but scaling rubric construction is difficult: expert rubrics are costly, prompted rubrics are often superficial or inconsistent, and fixed-pool discriminative rubrics can saturate and drift, enabling reward hacking. We present SibylSense, an inference-time learning approach that adapts a frozen rubric generator through a tunable memory bank of validated rubric items. Memory is updated via verifier-based item rewards measured by reference-candidate answer discriminative gaps from a handful of examples. SibylSense alternates memory tuning with a rubric-adversarial policy update that produces rubric-satisfying candidate answers, shrinking discriminative gaps and driving the rubric generator to capture new quality dimensions. Experiments on two open-ended tasks show that SibylSense yields more discriminative rubrics and improves downstream RL performance over static and non-adaptive baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
