RLVP：惩罚路径，奖励结果
RLVP: Penalize the Path, Reward the Outcome
AI总结：
研究在现实世界中智能体在线学习面临的挑战，提出“惩罚路径，奖励结果”方法，能在几乎零违规情况下实现高任务成功率，并给出有效惩罚的设计规则。
摘要：
在现实世界中代表我们行事的智能体（如拨打电话）必须从代价高昂且往往不可逆转的交互中进行在线学习，而非从廉价的模拟器步骤中学习。这带来两个问题：一是可部署性取决于路径而非仅结果，基于结果的奖励无法表达某些结果中性约束；二是由于每次交互成本高，智能体必须从极少示例中高效学习。基于可验证奖励的强化学习（RLVR）无法应对这两个挑战。相比之下，真实的智能体环境能廉价检测不良动作。“惩罚路径，奖励结果”的方法能在几乎零违规的情况下实现高任务成功率，而仅基于结果的训练几乎每一集都会违反约束。我们还提供了四条有效惩罚的设计规则，包括避免单独使用惩罚时出现的不作为陷阱。
英文摘要：
Agents acting on our behalf in the real world (e.g. placing phone calls) must learn online from costly, often irreversible interactions rather than cheap simulator steps. Two things follow. First, deployability depends on the path, not only the outcome. An agent must respect outcome-neutral constraints such as not repeatedly calling an unresponsive user, respecting business hours, or completing required authentication constraints that outcome-based rewards cannot express, since violating them frequently improves apparent success. Second, because each interaction is expensive, the agent must learn efficiently from very few examples. Reinforcement learning from verifiable rewards (RLVR) is blind to both challenges: it optimizes solely on the outcome and wastes expensive rollouts on all-fail groups where group-relative advantage collapses to zero. Attempts to densify supervision by rewarding progress target the hard-to-verify direction. In contrast, real agentic environments can cheaply detect bad moves. Since group-relative advantage is equivalent to within-group variance, a dense signal helps only when it supplies variance the outcome lacks. A verifiable penalty on the path meets this condition reliably, while a progress potential helps only where partial progress is reachable. The resulting recipe "penalize the path, reward the outcome" achieves high task success with near-zero violations, where outcome-only training violates constraints on nearly every episode. We provide four design rules for effective penalties, including avoidance of the inaction trap that arises when a penalty is used in isolation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
