发表机构
AI Robotics; School of Computer Science, Peking University（AI Robotics; 北京大学计算机学院）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对灵巧操作中探索噪声干扰精确控制的问题，DexPolicy将探索规模作为训练步数的显式函数进行退火，在多种策略优化设置下显著提升目标成功率。
AI 中文摘要
强化学习（RL）用于灵巧操作必须发现手指-物体接触，然后精确控制物体；服务于第一个目标的动作噪声可能会干扰第二个目标。在轨迹引导设置（如ViViDex）中，RL从人类视频中细化手-物体轨迹，我们的基线PPO在5M步后接近其初始动作噪声运行，这促使对探索规模进行显式控制。DexPolicy使该规模成为训练步数的显式函数，从广泛探索退火到狭窄探索，同时保持损失、架构、奖励和优化器设置固定。我们研究了三种策略优化设置：PPO、无评论家GRPO延续和流参数化PPO变体（FPO）。在五个YCB物体和三个训练种子上，平均确定性目标成功率从49.4%提高到68.1%（FPO），从14.1%提高到45.4%（GRPO），从32.0%提高到35.7%（PPO）。在带有Inspire/RH56手的RealMan RM75手臂上，三个物体上的360次试验将平均目标成功率从25.0%提高到85.0%（FPO），从10.0%提高到63.3%（GRPO），从8.3%提高到43.3%（PPO），每个物体-方法条件训练一个模型。PPO组件筛选偏向于噪声控制而非测试的优化器收缩；所选PPO调度在三个测试物体上产生比相同端点的线性衰减更高的平均目标成功率。训练回报、确定性目标成功率和执行噪声容忍度分离；因此，应根据预期执行条件下的终端任务成功率，针对每个任务和策略优化设置来评判调度。代码：此https URL。网站：此https URL。
英文摘要
Reinforcement learning (RL) for dexterous manipulation must discover finger-object contacts and then control the object precisely; the action noise that serves the first goal can interfere with the second. In trajectory-guided settings such as ViViDex, where RL refine hand-object trajectories from human video, our baseline PPO runs end near their initial action noise after 5M steps, motivating explicit control of exploration scale. DexPolicy makes that scale an explicit function of training steps, annealing from broad to narrow exploration while holding loss, architecture, reward, and optimizer settings fixed. We study three policy-optimization settings: PPO, critic-free GRPO continuation, and a flow-parameterized PPO variant (FPO). Across five YCB objects and three training seeds, mean deterministic Target success rises from 49.4% to 68.1% (FPO), 14.1% to 45.4% (GRPO), and 32.0% to 35.7% (PPO). On a RealMan RM75 arm with an Inspire/RH56 hand, 360 trials over three objects raise mean Target success from 25.0% to 85.0% (FPO), 10.0% to 63.3% (GRPO), and 8.3% to 43.3% (PPO), with one trained model per object-method condition. PPO component screening favors noise control over the tested optimizer contraction; the selected PPO schedule yields higher mean Target success than linear decay with the same endpoints on three tested objects. Training return, deterministic Target success, and tolerance to execution noise dissociate; schedules should therefore be judged by terminal task success under the intended execution conditions, per task and policy-optimization setting. Code: https://github.com/AIGeeksGroup/DexPolicy. Website: https://aigeeksgroup.github.io/DexPolicy.
