arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新，休息一下吧，周末愉快~~
arXiv 2602.03584cs.CLcs.AIcs.LG
收藏

$V_0$: 一种通用的价值模型，用于任何在零状态下的策略

$V_0$: A Generalist Value Model for Any Policy at State Zero

Yi-Kai Zhang, Zhiyuan Yao, Hongyan Hao, Yueqing Sun, Qi Gu, Hui Su, Xunliang Cai, De-Chuan Zhan, Han-Jia Ye

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出$V_0$，一种无需参数更新即可估计任意模型在未见提示上的预期性能的价值模型。通过将策略的动态能力作为显式上下文输入，$V_0$在策略优化中作为关键资源调度器，提升采样预算分配效率和部署时的路由性能。

摘要：

策略梯度方法依赖基线来衡量动作的相对优势，确保模型强化优于当前平均能力的行为。在使用Actor-Critic方法（如PPO）训练大型语言模型（LLMs）时，此基线通常由价值模型（Critic）估计，通常与策略模型大小相当。然而，随着策略不断演变，价值模型需要昂贵的同步增量训练来准确跟踪策略的转变能力。为避免此开销，Group Relative Policy Optimization（GRPO）通过使用一组轨迹的平均奖励作为基线，消除了耦合的价值模型，但此方法需要大量采样以维持估计稳定性。本文提出$V_0$，一种通用价值模型，能够估计任何模型在未见提示上的预期性能，而无需参数更新。我们通过将策略的动态能力作为显式上下文输入重新框架价值估计；具体而言，我们利用一组指令-性能对的历史来动态配置模型，脱离传统依赖参数拟合来感知能力转变的范式。聚焦于价值估计在零状态（即初始提示，因此为$V_0$），我们的模型作为关键资源调度器。在GRPO训练中，$V_0$在轨迹展开前预测成功率，允许高效分配采样预算；在部署中，它作为路由器，将指令路由到最经济且合适的模型。实验结果表明，$V_0$显著优于启发式预算分配，并在LLM路由任务中实现性能与成本之间的帕累托最优权衡。

英文摘要：

Policy gradient methods rely on a baseline to measure the relative advantage of an action, ensuring the model reinforces behaviors that outperform its current average capability. In the training of Large Language Models (LLMs) using Actor-Critic methods (e.g., PPO), this baseline is typically estimated by a Value Model (Critic) often as large as the policy model itself. However, as the policy continuously evolves, the value model requires expensive, synchronous incremental training to accurately track the shifting capabilities of the policy. To avoid this overhead, Group Relative Policy Optimization (GRPO) eliminates the coupled value model by using the average reward of a group of rollouts as the baseline; yet, this approach necessitates extensive sampling to maintain estimation stability. In this paper, we propose $V_0$, a Generalist Value Model capable of estimating the expected performance of any model on unseen prompts without requiring parameter updates. We reframe value estimation by treating the policy's dynamic capability as an explicit context input; specifically, we leverage a history of instruction-performance pairs to dynamically profile the model, departing from the traditional paradigm that relies on parameter fitting to perceive capability shifts. Focusing on value estimation at State Zero (i.e., the initial prompt, hence $V_0$), our model serves as a critical resource scheduler. During GRPO training, $V_0$ predicts success rates prior to rollout, allowing for efficient sampling budget allocation; during deployment, it functions as a router, dispatching instructions to the most cost-effective and suitable model. Empirical results demonstrate that $V_0$ significantly outperforms heuristic budget allocation and achieves a Pareto-optimal trade-off between performance and cost in LLM routing tasks.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑