arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.00592cs.ROcs.AIcs.CLcs.CVcs.LG
收藏

LangGap:诊断和缩小视觉-语言-动作模型中的语言差距

LangGap: Diagnosing and Closing the Language Gap in Vision-Language-Action Models

Department of Electrical and Computer Engineering, National University of Singapore, Singapore(电子与计算机工程系,新加坡国立大学)

Yuchen Hou, Lin Zhao

AI总结:

LangGap通过系统性语义扰动诊断和多样化任务设计,揭示并缩小VLA模型在理解多样语言指令方面的差距。

摘要:

视觉-语言-动作(VLA)模型在标准基准上实现了超过95%的成功率。然而,通过系统性实验,我们发现当前最先进的VLA模型大多忽略了语言指令。先前的工作缺乏:(1)系统性的语义扰动诊断;(2)一个迫使语言理解的基准;(3)语言多样性的训练数据。本文构建了LangGap基准,基于一种四维语义扰动方法——在保持桌面布局固定的情况下改变指令语义,揭示了π0.5中的语言理解缺陷。现有基准如LIBERO只给定一个任务每种布局,未能充分利用可用的物体和目标位置;LangGap在相同布局下完全多样化了拾取和放置任务,迫使模型真正理解语言。实验显示,针对性的数据增强可以部分缩小语言差距——单任务训练的成功率从0%提升到90%,多任务训练从0%提升到28%。然而,随着扩展任务的语义多样性增加,模型的学习能力证明严重不足;即使训练的任务也表现不佳。这揭示了VLA模型在理解多样语言指令方面的根本挑战——正是LangGap的长期价值所在。

英文摘要:

Vision-Language-Action (VLA) models achieve over 95% success on standard benchmarks. However, through systematic experiments, we find that current state-of-the-art VLA models largely ignore language instructions. Prior work lacks: (1) systematic semantic perturbation diagnostics, (2) a benchmark that forces language understanding by design, and (3) linguistically diverse training data. This paper constructs the LangGap benchmark, based on a four-dimensional semantic perturbation method -- varying instruction semantics while keeping the tabletop layout fixed -- revealing language understanding deficits in π0.5. Existing benchmarks like LIBERO assign only one task per layout, underutilizing available objects and target locations; LangGap fully diversifies pick-and-place tasks under identical layouts, forcing models to truly understand language. Experiments show that targeted data augmentation can partially close the language gap -- success rate improves from 0% to 90% with single-task training, and 0% to 28% with multi-task training. However, as semantic diversity of extended tasks increases, model learning capacity proves severely insufficient; even trained tasks perform poorly. This reveals a fundamental challenge for VLA models in understanding diverse language instructions -- precisely the long-term value of LangGap.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑