arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2503.10322cs.CV
收藏

迈向快速、基于记忆且数据高效的视觉-语言策略

Towards Fast, Memory-based and Data-Efficient Vision-Language Policy

School of Electronic Information and Communications, Huazhong University of Science and Technology(华中科技大学电子信息与通信学院)

Haoxuan Li, Sixu Yan, Yuhan Li, Xinggang Wang

AI总结:

针对现有视觉语言策略推理昂贵、模态失配且经验处理能力有限的问题,本文提出基于10亿参数VLM微调的轻量记忆型策略模型LiteVLP,在VIMA-Bench和长时程任务中实现更高效率与性能。

摘要:

在互联网规模视觉-语言数据上预训练的视觉语言模型(VLM)已展现出将其知识迁移到机器人学习的潜力。然而,现有范式面临三个关键挑战:(1)大规模模型参数导致昂贵的推理成本;(2)数据模态不匹配引起频繁的领域偏移;(3)处理过去或未来经验的能力有限。在本工作中,我们提出 LiteVLP,这是一种轻量级、基于记忆且通用的视觉-语言策略生成模型。LiteVLP 构建于一个预训练的 10 亿参数 VLM 之上,并在一个小规模、对话风格的机器人数据集上进行微调。通过大量实验,我们证明 LiteVLP 在 VIMA-Bench 上以最少的训练时间优于最先进的视觉-语言策略。此外,LiteVLP 在保持极高准确率的同时展现出卓越的推理速度。在长时程操作任务中,LiteVLP 还表现出显著的记忆能力,比表现最佳的基线模型高出 18.8%。这些结果凸显 LiteVLP 是将 VLM 智能融入机器人学习的一种有前景的模型。

英文摘要:

Vision Language Models (VLMs) pretrained on Internet-scale vision-language data have demonstrated the potential to transfer their knowledge to robotic learning. However, the existing paradigm encounters three critical challenges: (1) expensive inference cost resulting from large-scale model parameters, (2) frequent domain shifts caused by mismatched data modalities, and (3) limited capacity to handle past or future experiences. In this work, we propose LiteVLP, a lightweight, memory-based, and general-purpose vision-language policy generation model. LiteVLP is built upon a pre-trained 1B-parameter VLM and fine-tuned on a tiny-scale and conversation-style robotic dataset. Through extensive experiments, we demonstrate that LiteVLP outperforms state-of-the-art vision-language policy on VIMA-Bench, with minimal training time. Furthermore, LiteVLP exhibits superior inference speed while maintaining exceptional high accuracy. In long-horizon manipulation tasks, LiteVLP also shows remarkable memory ability, outperforming the best-performing baseline model by 18.8%. These results highlight LiteVLP as a promising model to integrating the intelligence of VLMs into robotic learning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑