arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.30795cs.RO
收藏

Feat2Go: 面向具身强化学习的视觉特征基础价值估计

Feat2Go: Visual Feature-Grounded Value Estimation for Embodied Reinforcement Learning

Wangxuan Institute of Computer Technology, Peking University, China(北京大学计算机科学技术研究院)

Junyang Shu, Zhiwei Lin, Bingqing Wei, Yongtao Wang

AI总结:

提出Feat2Go框架,通过预训练视觉世界模型提取补丁级子目标相似度并聚类语义阶段,训练具身价值模型预测结构进度以重塑终端奖励,显著提升VLA模型在单臂和双臂操作任务中的强化学习性能。

摘要:

强化学习是提升视觉-语言-动作(VLA)模型能力的一种有前景的方法,同时避免了模仿学习对大量数据的需求。然而,其对VLA模型的有效性常受限于稀疏监督以及为长程操作设计信息丰富的奖励信号的困难。在这项工作中,我们提出了Feat2Go,一种用于具身强化学习的细粒度价值估计框架。具体来说,Feat2Go首先通过测量与子目标状态的补丁级相似性,并利用基于趋势的聚类将回合划分为语义阶段,从预训练的视觉世界模型中导出一个连续的进度目标。然后,我们训练一个具身价值模型,根据当前观测和任务指令预测这一结构进度,并在策略优化过程中使用预测值重塑终端奖励。所提出的框架与现有的VLA策略强化学习流程(包括PPO和GRPO)兼容,且不依赖手动奖励工程。在ManiSkill3和RoboTwin 2.0上的大量实验表明,Feat2Go在单臂和双臂操作设置下均能持续提升现有VLA模型的性能。更具体地说,在ManiSkill3上,Feat2Go将OpenVLAOFT的平均分布外成功率从17.5%提升至82.9%,同时保留了96.9%的分布内性能。在RoboTwin 2.0上,Feat2Go在域随机化任务设置中实现了88.8%的平均成功率,优于先前的强化学习方法。

英文摘要:

Reinforcement learning is a promising approach for improving the capabilities of vision-language-action (VLA) models while avoiding the heavy data requirements of imitation learning. However, its effectiveness for VLA models is often constrained by sparse supervision and the difficulty of designing informative reward signals for long-horizon manipulation. In this work, we present Feat2Go, a fine-grained value estimation framework for embodied reinforcement learning. Specifically, Feat2Go first derives a continuous progress target from a pretrained visual world model by measuring patch-level similarity to subgoal states and partitioning episodes into semantic stages with trend-based clustering. We then train an embodied value model to predict this structural progress from the current observation and task instruction, and use the predicted value to reshape terminal rewards during policy optimization. The proposed framework is compatible with existing VLA policy reinforcement learning pipelines, including PPO and GRPO, and does not rely on manual reward engineering. Extensive experiments on ManiSkill3 and RoboTwin 2.0 demonstrate that Feat2Go consistently improves the performance of existing VLA models under both single-arm and bimanual manipulation settings. More specifically, on ManiSkill3, Feat2Go improves OpenVLAOFT from 17.5% to 82.9% average out-of-distribution success while retaining 96.9% in-distribution performance. On RoboTwin 2.0, Feat2Go achieves an average success rate of 88.8% in domain-randomized task settings, outperforming prior reinforcement learning methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑