RHyVE：基于能力的验证与阶段感知部署用于LLM生成的奖励假说
RHyVE: Competence-Aware Verification and Phase-Aware Deployment for LLM-Generated Reward Hypotheses
AI总结：
研究提出RHyVE框架，通过能力感知验证和阶段感知部署提升LLM生成奖励假说的可靠性，实验显示在低能力阶段奖励排名不可靠，但经过任务依赖阈值后变得有效。
摘要：
大型语言模型（LLM）使强化学习中的奖励设计更具可扩展性，但生成的奖励并非自动可靠的训练目标。现有研究主要关注生成、进化或选择奖励候选，而较少关注何时可以验证和部署这些候选。本文研究部署时间问题，将生成的奖励视为奖励假说，其效用取决于当前策略的能力和训练阶段。提出RHyVE，一种能力感知验证和阶段感知部署协议，通过共享策略检查点比较小的奖励假说集，使用短周期分支验证。实验显示，在低能力阶段奖励排名不可靠，但经过任务依赖阈值后变得有效。在稀疏操作任务中，阶段感知部署在锁定协议下提升峰值和保留性能。更新的LLM生成奖励候选实验显示候选家族依赖行为：生成池可表现出阶段依赖的胜利变化，但无固定预热计划是普遍最优。预留计划选择、保守选择器基线、计算匹配对照和规模对照进一步表明，RHyVE应被视为验证驱动的部署协议而非通用调度器。密集和所有失败边界实验界定方法范围。这些结果表明，奖励生成和奖励部署应作为耦合问题研究：生成的奖励必须在变化的策略能力下验证和部署。
英文摘要：
Large language models (LLMs) make reward design in reinforcement learning substantially more scalable, but generated rewards are not automatically reliable training objectives. Existing work has focused primarily on generating, evolving, or selecting reward candidates, while paying less attention to when such candidates can be verified and deployed during policy optimization. We study this deployment-time problem by treating generated rewards as reward hypotheses whose utility depends on the competence of the current policy and the phase of training. We propose \textsc{RHyVE}, a competence-aware verification and phase-aware deployment protocol that compares small sets of reward hypotheses from shared policy checkpoints using short-horizon fork verification. Our experiments show that reward rankings are unreliable at low competence but become informative after task-dependent thresholds. On a sparse manipulation task, phase-aware deployment improves peak and retained performance under a locked protocol. Updated LLM-generated reward-candidate experiments show candidate-family-dependent behavior: generated pools can exhibit phase-dependent winner changes, but no fixed warm-up schedule is universally optimal. Held-out schedule selection, conservative selector baselines, compute-matched controls, and scale controls further show that \textsc{RHyVE} is best understood as a verification-informed deployment protocol rather than a universal scheduler. Dense and all-failure boundary experiments delimit the scope of the method. Together, these results suggest that reward generation and reward deployment should be studied as coupled problems: generated rewards must be verified and deployed under changing policy competence.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
