arXiv 2605.302262026-10-02cs.ROcs.AI

BORA: 弥合离线强化学习与在线残差适应以实现真实世界灵巧VLA模型

BORA: Bridging Offline Reinforcement Learning and Online Residual Adaptation for Real-World Dexterous VLA Models

Shanghai Jiao Tong University（上海交通大学）；CASIA（中国科学院自动化研究所）；Shanghai AI Laboratory（上海人工智能实验室）；USTC（中国科学技术大学）

Zhongxi Chen, Yifan Han, Bin Qiu, Zhangliang Gao, Yanming Shao, Huanming Liu, Congsheng Xu, Xiaoyu Chen, Xingyu Ye, Yao Mu, Wenzhao Lian

备注：: 9 pages,7 figures

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出BORA框架，通过离线构建动作条件价值引导的评论家，并结合在线冻结VLA基础、引入人类在环的分块残差适应机制，解决灵巧操作中高维探索导致的时间不一致、样本低效和硬件风险问题，在五个真实灵巧任务上平均成功率提升33%。

摘要：

视觉-语言-动作（VLA）模型已成为将视觉-语言理解融入真实世界机器人操作的一种有前景的范式。然而，由于高维手部控制和复合执行误差，灵巧操作对VLA策略仍然具有挑战性，这使得真实世界的强化学习后训练对于弥合视觉基础动作生成与物理可靠灵巧执行之间的差距至关重要。然而，高维灵巧探索常常引发真实世界中的时间不一致性、样本低效和硬件风险。为应对这些挑战，我们提出BORA，一种为真实世界灵巧VLA模型设计的离线到在线强化学习后训练框架。在离线阶段，BORA构建一个以VLM的认知令牌和动作块作为输入的评论家。这种设计实现了动作条件价值引导，使评论家能够评估超越视觉上下文的灵巧手部运动。在随后的在线阶段，BORA冻结VLA基础，并引入一种轻量级、人类在环（HiL）的分块残差适应机制，以减轻真实世界执行误差并进一步在真实物理环境中纠正离线学习到的意图。通过继承离线评论家并采用干预驱动奖励，BORA有效纠正执行差异并适应真实世界物理变化，同时将预训练策略作为稳定先验。在五个复杂真实世界灵巧任务上的广泛评估表明，BORA显著优于纯模仿学习和传统解耦强化学习基线，在标准设置下平均成功率绝对提升33%，在未见物体泛化中提升高达43%。

英文摘要：

Vision-Language-Action (VLA) policies provide strong behavioral priors for dexterous manipulation, yet adapting them on real robots remains challenging because high-DoF contact failures are difficult for humans to correct and online interaction is expensive. We present BORA, an offline-to-online reinforcement learning system that integrates an action-conditioned critic into a consistency-policy VLA and reuses the learned critic for frozen-base residual adaptation. To obtain executable corrective data, BORA combines wearable arm--hand teleoperation with a demonstration-guided local policy that translates coarse human intent into coordinated, embodiment-specific finger motions for contact-rich skills. Online robot rollouts and human corrections are mixed with offline data to update only a lightweight residual actor, avoiding full-model fine-tuning. We evaluate BORA on six real-world tasks using single-arm and bimanual platforms equipped with two dexterous-hand models. With only 20 online trajectories per task, BORA improves average success from 60.8% to 82.5% on standard objects and from 52% to 70% on held-out objects, while policy assistance substantially improves intervention reliability in bimanual twisting. These results demonstrate a practical route from executable human correction to efficient real-robot VLA adaptation.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。