arXiv 2602.207152026-02-24cs.RO

IG-RFT: 一种基于交互引导的强化学习框架用于长时域机器人操作中的VLA模型

IG-RFT: An Interaction-Guided RL Framework for VLA Models in Long-Horizon Robotic Manipulation

Grasp Laboratory, Mechanical Engineering Department, Zhejiang University（浙江大学机械工程学院抓取实验室）；Torch Kernel Co., Ltd.（火炬核科技有限公司）

Zhian Su, Weijie Kong, Haonan Dong, Huixu Dong

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

IG-RFT通过交互引导强化学习框架提升VLA模型在长时域机器人任务中的性能，实现85%的成功率，优于传统方法。

摘要：

视觉-语言-动作（VLA）模型在通用机器人策略中展现出显著潜力；然而，由于分布偏移和高质量演示的稀缺，它们在新现实领域中难以泛化到长时域复杂任务。尽管强化学习（RL）为策略改进提供了有前途的途径，但将其应用于现实中的VLA微调面临探索效率、训练稳定性和样本成本等方面的挑战。为了解决这些问题，我们提出了IG-RFT，一种针对流式VLA模型设计的交互引导强化微调系统。首先，为了促进有效的策略优化，我们引入了交互引导优势加权回归（IG-AWR），一种RL算法，能够根据机器人的交互状态动态调节探索强度。此外，为了解决稀疏或任务特定奖励的限制，我们设计了一种新的混合密集奖励函数，该函数整合了轨迹级奖励和子任务级奖励。最后，我们构建了一个包含SFT、离线RL和人机协同RL的三阶段RL系统，用于微调VLA模型。在四个具有挑战性的长时域任务上的广泛现实实验表明，IG-RFT实现了平均成功率为85.0%，显著优于SFT（18.8%）和标准离线RL基线（40.0%）。消融研究确认了IG-AWR和混合奖励塑造的关键贡献。总而言之，我们的工作建立并验证了VLA模型在现实机器人操作中的新型强化微调系统。

英文摘要：

Vision-Language-Action (VLA) models have demonstrated significant potential for generalist robotic policies; however, they struggle to generalize to long-horizon complex tasks in novel real-world domains due to distribution shifts and the scarcity of high-quality demonstrations. Although reinforcement learning (RL) offers a promising avenue for policy improvement, applying it to real-world VLA fine-tuning faces challenges regarding exploration efficiency, training stability, and sample cost. To address these issues, we propose IG-RFT, a novel Interaction-Guided Reinforced Fine-Tuning system designed for flow-based VLA models. Firstly, to facilitate effective policy optimization, we introduce Interaction-Guided Advantage Weighted Regression (IG-AWR), an RL algorithm that dynamically modulates exploration intensity based on the robot's interaction status. Furthermore, to address the limitations of sparse or task-specific rewards, we design a novel hybrid dense reward function that integrates the trajectory-level reward and the subtask-level reward. Finally, we construct a three-stage RL system comprising SFT, Offline RL, and Human-in-the-Loop RL for fine-tuning VLA models. Extensive real-world experiments on four challenging long-horizon tasks demonstrate that IG-RFT achieves an average success rate of 85.0%, significantly outperforming SFT (18.8%) and standard Offline RL baselines (40.0%). Ablation studies confirm the critical contributions of IG-AWR and hybrid reward shaping. In summary, our work establishes and validates a novel reinforced fine-tuning system for VLA models in real-world robotic manipulation.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

AI总结：

摘要：

英文摘要：

相关论文