基于动作偏好优化的人类辅助机器人策略精调
Human-assisted Robotic Policy Refinement via Action Preference Optimization
AI总结：
针对VLA模型依赖离线演示、部署后难优化的问题，提出动作偏好优化方法APO，通过人机协作收集交互轨迹，结合自适应重加权算法与二元期望信号实现从失败中学习，在仿真与真实操作任务中展现更优泛化性与鲁棒性。
摘要：
构建可靠且可迭代优化的机器人系统是落地真实世界应用的核心前提。尽管视觉-语言-动作（Vision-Language-Action, VLA）模型已被公认为这类机器人部署的基础模型，但其对离线专家演示的依赖严重限制了部署后的优化能力。为缓解这一局限，我们提出动作偏好优化（Action Preference Optimization, APO）方法，通过与环境交互收集的人类辅助偏好对齐来精调VLA模型。该方法首先搭建人机协作框架，用于可靠的故障纠正以及通过人类干预收集交互轨迹。但由于机器人动作的不可逆性和token分布不匹配的挑战，直接利用这些交互轨迹进行偏好优化并非易事。为解决该问题，APO提出一种自适应重加权算法，结合从交互中提取的二元期望信号，使VLA模型能有效抑制易失败动作，同时增强纠正动作的适配性。最终，APO赋予VLA模型从失败中学习的关键能力，为其在动态环境中的迭代优化和可靠部署铺平道路。在仿真和真实场景中开展的实验证明，我们的人类辅助框架在各类操作任务上具备更优的泛化性和鲁棒性。我们认为这项工作能为通过人机协作高效、稳定地优化VLA模型带来启发。代码和数据集已发布于https://github.com/GeWu-Lab/Action-Preference-Optimization
英文摘要：
Establishing a reliable and iteratively refined robotic system is essential for deploying real-world applications. While Vision-Language-Action (VLA) models are widely recognized as the foundation model for such robotic deployment, their reliance on offline expert demonstrations critically limits their capacity for post-deployment refinement. To mitigate this limitation, we introduce Action Preference Optimization (APO), a method designed to refine VLA models by human-assisted preference alignment gathered through interaction with environments. This method begins with a human-robot collaboration framework for reliable failure correction and interaction trajectory collection through human intervention. However, directly leveraging these interaction trajectories for preference optimization is non-trivial due to the challenges of irreversible robotic actions and token distribution mismatch. To solve this, APO proposes an adaptive reweighting algorithm with binary desirability signals derived from interaction, empowering VLA models effectively suppress failure-prone actions while enhancing corrective action adaptation. Ultimately, APO equips VLA models with the crucial capability to learn from failure, paving the way for their iterative refinement and reliable deployment in dynamic environments. The experiments conducted in simulation and real-world scenarios prove superior generalization and robustness of our human-assisted framework across a variety of manipulation tasks. We believe this work could bring insights for efficient and stable optimization of VLA models through human-robot collaboration. The code and dataset are released at https://github.com/GeWu-Lab/Action-Preference-Optimization
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
