补丁策略：通过密集视觉表示实现高效具身控制

中文摘要 AI 辅助

来自视觉Transformer（ViT）的预训练密集视觉特征强大但在机器人学习中未得到充分利用。现代机器人策略要么将每个观察压缩为单个全局令牌，要么依赖从头训练的视觉主干，牺牲了细粒度空间细节和大规模视觉预训练的好处。虽然存在像大型视觉语言动作模型（VLA）那样对密集补丁特征进行操作的策略，但它们往往繁重且缓慢。我们用补丁策略弥合了这一差距，它是一种最小的架构扩展，使基于Transformer的策略能够直接使用密集预训练补丁令牌，而无需完整VLM的计算开销。其核心是一个块因果注意力掩码，在保留标准策略时间因果关系的同时，让模型在每次观察时关注多个补丁令牌以及其他状态信息。补丁策略轻量级、快速且高效。在四个模拟和三个真实世界环境套件中，我们的方法比使用最先进全局池化表示的策略实现了40%的相对改进。此外，它在使用约0.7%参数的情况下，比微调后的OpenVLA - OFT高出18%。我们相信补丁策略为机器人社区提供了一个管道，以便在不牺牲高频反应控制所需的训练效率或推理速度的情况下，轻松利用视觉表示学习的持续进展。视频可在此https URL查看

英文摘要

Pretrained dense visual features from Vision Transformers (ViTs) are powerful yet have been underutilized in robot learning. Modern robot policies either compress each observation into a single global token, or rely on visual backbones trained from scratch, sacrificing both fine-grained spatial detail and the benefits of large-scale visual pre-training. While there exist policies that do operate on dense patch features like large vision-language-action models (VLAs), they tend to be heavy and slow, inheriting the full cost of a billion-parameter vision-language model (VLM) backbone. We close this gap with Patch Policy, a minimal architectural extension that enables transformer-based policies to consume dense pre-trained patch tokens directly without the computational overhead of a full VLM. At its core is a block-causal attention mask that preserves the temporal causality of standard policies while letting the model attend over many patch tokens per observation, alongside other state information. Patch Policy is lightweight, fast, and highly effective. Across four simulated and three real-world environment suites, our method achieves a 40% relative improvement over policies using state-of-the-art global-pooled representations. Furthermore, it surpasses fine-tuned OpenVLA-OFT by 18% while using roughly 0.7% of the parameters. We believe Patch Policy provides a pipeline for the robotics community to readily leverage continuing progress in visual representation learning, without sacrificing the training efficiency or inference speed required for high-frequency, reactive control. Videos can be viewed at https://patch-policy.github.io

发表机构

• Courant Institute, New York University（纽约大学柯朗数学科学研究所）
• Meta-FAIR（Meta FAIR）
• AMI Labs（AMI 实验室）

机构由 AI 辅助整理，请以论文原文为准。