EmbodiedMidtrain: 通过中训练弥合视觉语言模型与视觉语言动作模型之间的差距
EmbodiedMidtrain: Bridging the Gap between Vision-Language Models and Vision-Language-Action Models via Mid-training
AI总结：
本文提出EmbodiedMidtrain方法，通过中训练弥合视觉语言模型与视觉语言动作模型之间的差距，实验表明中训练能提升不同VLM骨干网络的性能，且在初始化阶段对VLA微调有显著帮助。
摘要：
视觉语言动作模型（VLAs）继承了视觉语言模型（VLMs）的视觉和语言能力，但大多数VLAs是基于未适应于具身域的现成VLMs构建的，限制了其下游性能。本文提出EmbodiedMidtrain，通过分析VLMs与VLAs之间的数据分布差距，构建了一个轻量级可学习接近性估计器，从大规模VLM池中选择最符合VLA的候选者，并在下游VLA微调前中训练VLM。实验表明，中训练在不同VLM骨干网络上均能提升性能，结果可与专家VLAs和更大规模、更高预算训练的现成VLMs竞争。进一步分析显示，中训练为VLA微调提供了更强的初始化，且收益在训练早期逐步显现。此外，数据引擎捕捉了数据集级和样本级对齐信号，在空间推理任务上优于文本中心任务，同时保持VLM数据的多样性。所有代码、数据和模型将向未来研究开放。
英文摘要：
Vision-Language-Action Models (VLAs) inherit their visual and linguistic capabilities from Vision-Language Models (VLMs), yet most VLAs are built from off-the-shelf VLMs that are not adapted to the embodied domain, limiting their downstream performance. In this work, we propose EmbodiedMidtrain to bridge the gap between VLMs and VLAs. We first characterize the data distribution gap between them, showing that VLA data occupy compact regions that are largely separated from the broader VLM distribution, while the degree of alignment varies substantially both across and within VLM data sources. Then, we build a mid-training data engine that leverages a lightweight learnable proximity estimator to select the most VLA-aligned candidates from a large VLM pool, and mid-trains the VLM on this curated mixture before downstream VLA fine-tuning. Experiments on three robot manipulation benchmarks show that mid-training consistently improves performance across different VLM backbones, achieving results competitive with expert VLAs and off-the-shelf VLMs trained with larger model scale and training budgets. Further analysis reveals that mid-training provides a stronger initialization for VLA fine-tuning, with gains emerging from the earliest steps and widening throughout training. Moreover, the data engine captures both dataset-level and sample-level alignment signals, favoring spatial reasoning over text-centric tasks while preserving the diversity of the VLM data. We will release all code, data and models for future research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
