Interleave-VLA：利用交错图像-文本指令增强机器人操作
Interleave-VLA: Enhancing Robot Manipulation with Interleaved Image-Text Instructions
AI总结：
Interleave-VLA提出首个能理解交错图像-文本指令并直接生成连续动作的机器人学习范式，通过自动转换指令构建21万集数据集，实现零样本泛化，域外性能提升2倍。
摘要：
基础模型的兴起为物理世界中的通用机器人策略铺平了道路。现有方法依赖纯文本指令，往往难以泛化到未见过的场景。我们认为，交错图像-文本输入提供了更丰富、偏差更小的上下文，使机器人能够通过更多样的人机交互更好地处理未见任务。基于这一见解，我们提出了Interleave-VLA，这是首个能够理解交错图像-文本指令并直接在物理世界中生成连续动作序列的机器人学习范式。它提供了一种自然、灵活且与模型无关的范式，能以最小的修改扩展最先进的视觉-语言-动作（VLA）模型，同时实现强大的零样本泛化。Interleave-VLA还包含一个自动流水线，将Open X-Embodiment中的文本指令转换为交错图像-文本指令，从而构建了一个包含21万集的大规模真实世界交错具身数据集。在仿真和真实世界中的综合评估表明，Interleave-VLA具有两大优势：（1）与文本输入基线相比，对未见物体的域外泛化能力提升了2倍；（2）以零样本方式支持灵活的任务接口和多样化的指令，例如手绘草图。我们将Interleave-VLA强大的零样本能力归因于指令图像的使用，它有效缓解了幻觉，以及包含来自互联网图像的异构多模态数据集，这为可扩展性提供了潜力。更多信息请访问 https://interleave-vla.github.io/Interleave-VLA-Anonymous/
英文摘要：
The rise of foundation models paves the way for generalist robot policies in the physical world. Existing methods relying on text-only instructions often struggle to generalize to unseen scenarios. We argue that interleaved image-text inputs offer richer and less biased context and enable robots to better handle unseen tasks with more versatile human-robot interaction. Building on this insight, Interleave-VLA, the first robot learning paradigm capable of comprehending interleaved image-text instructions and directly generating continuous action sequences in the physical world, is introduced. It offers a natural, flexible, and model-agnostic paradigm that extends state-of-the-art vision-language-action (VLA) models with minimal modifications while achieving strong zero-shot generalization. Interleave-VLA also includes an automatic pipeline that converts text instructions from Open X-Embodiment into interleaved image-text instructions, resulting in a large-scale real-world interleaved embodied dataset with 210k episodes. Comprehensive evaluation in simulation and the real world shows that Interleave-VLA offers two major benefits: (1) improves out-of-domain generalization to unseen objects by 2x compared to text input baselines, (2) supports flexible task interfaces and diverse instructions in a zero-shot manner, such as hand-drawn sketches. We attribute Interleave-VLA's strong zero-shot capability to the use of instruction images, which effectively mitigate hallucinations, and the inclusion of heterogeneous multimodal datasets, enriched with Internet-sourced images, offering potential for scalability. More information is available at https://interleave-vla.github.io/Interleave-VLA-Anonymous/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
