通过解锁ViT的完整表示构建视觉动作变换器
VAT: Vision Action Transformer by Unlocking Full Representation of ViT
AI总结:
VAT通过解锁ViT的完整表示,提出了一种新的视觉动作变换器架构,实现了在模拟操作任务中98.15%的成功率,优于现有方法。
摘要:
在机器人学习中,视觉变换器(ViTs)是视觉感知的标准方法,但大多数方法通过仅使用最终层的特征丢弃了有价值的信息。我们认为这提供了不足的表示,并提出视觉动作变换器(VAT),这是一种扩展自ViT的新架构,能够解锁ViT的完整特征层次。VAT在所有变换器层中处理专门的动作标记与视觉特征,使感知与动作生成的深度和渐进融合成为可能。在一系列模拟操作任务上,VAT在四个LIBERO基准测试中实现了98.15%的平均成功率,建立了新的最先进的水平,优于先前的方法如OpenVLA-OFT。我们的工作不仅提出了一个强大的模仿学习模型,还展示了利用视觉模型完整''表示轨迹''以推进机器人策略的关键重要性。该项目的GitHub链接是https://github.com/sellerbubble/VAT。
英文摘要:
In robot learning, Vision Transformers (ViTs) are standard for visual perception, yet most methods discard valuable information by using only the final layer's features. We argue this provides an insufficient representation and propose the Vision Action Transformer (VAT), a novel architecture that is extended from ViT and unlocks the full feature hierarchy of ViT. VAT processes specialized action tokens with visual features across all transformer layers, enabling a deep and progressive fusion of perception and action generation. On a suite of simulated manipulation tasks, VAT achieves a 98.15\% average success rate across four LIBERO benchmarks, establishing a new state-of-the-art by outperforming prior methods like OpenVLA-OFT. Our work presents not only a powerful model for imitation learning but also demonstrates the critical importance of leveraging the complete ''representation trajectory'' of vision models to advance robotic policy. The GitHub URL for the project code is https://github.com/sellerbubble/VAT.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。