OGPO：面向可恢复视觉-语言-动作模型的离线目标条件策略优化
OGPO: Offline Goal-conditioned Policy Optimization for Recoverable Vision-Language-Action Models
AI总结：
该研究提出OGPO框架，将VLA后续训练从轨迹级动作模仿转向关键状态可达性学习，在LIBERO等基准上提升了VLA策略性能与分布偏移下的恢复能力。
摘要：
视觉-语言-动作（VLA）模型在机器人操纵领域展现出强大潜力，但其后续训练仍主要依赖基于机器人演示的轨迹级监督模仿。尽管该范式对学习可行动作有效，却会监督策略逐步复现专家动作，使策略偏向演示特定的动作路径，且在执行偏离演示轨迹时，仅能提供有限的引导以恢复至任务进展状态。为解决这一局限，我们提出OGPO，一种面向可恢复VLA后续训练的离线目标条件策略优化框架。OGPO并非仅将离线演示视为逐步动作标签，而是自适应地用语义关键状态重新标记轨迹，并构建目标条件过程奖励以优化策略，使其能够到达这些任务进展状态。通过这种方式，OGPO将VLA后续训练从轨迹级动作模仿转向关键状态可达性学习，使策略能够从可行的非演示状态完成目标状态，且无需额外在线交互。在LIBERO和MetaWorld上的实验表明，OGPO相比标准监督微调可持续提升VLA策略性能。此外，在LIBERO-Plus及我们提出的执行中扰动基准LIBERO-ReAct上的评估进一步证明，OGPO在分布偏移和可恢复非演示状态下，具备更强的鲁棒性和目标导向恢复能力。
英文摘要：
Vision-language-action (VLA) models have shown strong promise in robotic manipulation, yet their post-training still predominantly relies on trajectory-level supervised imitation over robot demonstrations. While effective for learning feasible actions, this paradigm supervises policies to reproduce expert actions at each step, biasing them toward demonstration-specific action paths and providing limited guidance for recovering toward task-progress states once execution deviates from demonstrated trajectories. To address this limitation, we propose OGPO, an Offline Goal-conditioned Policy Optimization framework for recoverable VLA post-training. Instead of treating offline demonstrations merely as step-wise action labels, OGPO adaptively relabels trajectories with semantic key states and constructs goal-conditioned process rewards to optimize the policy for reaching these task-progress states. In this way, OGPO shifts VLA post-training from trajectory-level action imitation to key-state reachability learning, enabling policies to acquire a more robust ability to complete goal states from feasible off-demonstration states without requiring additional online interaction. Experiments on LIBERO and MetaWorld show that OGPO consistently improves VLA policy performance over standard supervised fine-tuning. Moreover, evaluations on LIBERO-Plus and our proposed LIBERO-ReAct, a mid-execution perturbation benchmark, further demonstrate that OGPO achieves stronger robustness and goal-directed recoverability under distribution shifts and recoverable off-demonstration states.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
