RubricArmor：对抗演化改进基于LLM的评分标准生成
RubricArmor: Adversarial Evolution Improves LLM-Based Rubric Generation
- National University of Singapore（新加坡国立大学）
- Beijing University of Chemical Technology（北京化工大学）
- Zhejiang University（浙江大学）
- University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
- Southeast University（东南大学）
- Xiaohongshu（小红书）
- Peking University（北京大学）
- MBZUAI（穆罕默德·本·扎耶德人工智能大学）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
RubricArmor通过对抗演化在评分标准生成阶段主动暴露并修复奖励黑客漏洞，提升基于LLM的评分标准质量，进而增强下游强化学习效果。
AI中文摘要：
基于评分标准的强化学习（RL）通过针对查询特定的评估标准来评估响应，为对齐大型语言模型（LLMs）提供可解释的奖励。为了大规模构建评分标准，一种直接的基于LLM的评分标准生成方法是提示LLM直接从查询中生成评分标准。然而，由LLM直接生成的评分标准容易受到奖励黑客攻击，因为遗漏或未充分指定的标准使得策略能够以低质量的响应获得高评分标准奖励。现有的基于LLM的评分标准生成方法提高了生成标准的细粒度和覆盖范围，但并未主动防范奖励黑客攻击。为解决这一局限性，我们提出了RubricArmor，一个对抗性框架，在评分标准生成阶段暴露并缓解潜在的奖励黑客攻击，从而在后续RL中避免其发生。具体而言，RubricArmor执行对抗演化，其中攻击步骤和修复步骤在多轮中交替进行。攻击步骤通过构建满足当前评分标准但未能正确完成任务的对抗性响应来模拟策略的奖励黑客行为。修复步骤随后修订评分标准，以检测攻击步骤暴露的响应缺陷，同时保留其他有效标准。大量实验表明，RubricArmor优于竞争性的评分标准生成基线，并转化为更有效的下游基于评分标准的RL。
英文摘要：
Rubric-based reinforcement learning (RL) provides interpretable rewards for aligning large language models (LLMs) by evaluating responses against query-specific evaluation criteria. To construct rubrics at scale, a straightforward approach to LLM-based rubric generation is to prompt an LLM to generate a rubric directly from the query. However, rubrics directly generated by LLMs are vulnerable to reward hacking, since omitted or underspecified criteria allow the policy to obtain high rubric rewards with low-quality responses. Existing LLM-based rubric generation methods improve the granularity and coverage of the generated criteria but do not proactively guard against reward hacking. To address this limitation, we propose RubricArmor, an adversarial framework that exposes and mitigates potential reward hacking at the rubric generation stage before it occurs in subsequent RL. Specifically, RubricArmor performs adversarial evolution, in which an attack step and a repair step alternate over multiple rounds. The attack step simulates the reward hacking of the policy by constructing adversarial responses that satisfy the current rubric but fail to properly complete the task. The repair step then revises the rubric to detect the response defects exposed by the attack step while preserving other valid criteria. Extensive experiments demonstrate that RubricArmor outperforms competitive rubric generation baselines and translates into more effective downstream rubric-based RL.
