arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.01863cs.CL
收藏

VSP:评估视觉语言模型在空间规划任务中感知与推理的双重挑战

VSP: Assessing the dual challenges of perception and reasoning in spatial planning tasks for VLMs

UC Santa Barbara(加州大学圣巴巴拉分校);Adobe Research(奥多比研究院);MIT-IBM Watson AI Lab(麻省理工学院-IBM沃森人工智能实验室)

Qiucheng Wu, Handong Zhao, Michael Saxon, Trung Bui, William Yang Wang, Yang Zhang, Shiyu Chang

AI总结:

本文提出VSP基准,通过总体空间规划及感知、推理细粒度子任务评估VLM,发现开源与私有模型均存在感知缺陷和推理瓶颈。

摘要:

视觉语言模型(VLM)是一类令人兴奋的新兴语言模型(LM),它们将经典语言模型的能力与图像处理系统的能力融合在一起。然而,这些能力结合的方式并不总是直观的,因此有必要进行直接研究。VLM中一项研究不足的能力是视觉空间规划,即理解物体的空间排列,并在视觉场景中制定行动计划以实现预期结果的能力。在我们的研究中,我们提出了VSP这一基准,它:1)总体评估这些模型的空间规划能力;2)将视觉规划任务分解为更细粒度的子任务,包括感知和推理,并衡量语言模型在这些子任务中的能力。我们的评估表明,开源和私有VLM甚至无法为简单的空间规划任务生成有效的计划。对细粒度分析任务的评估进一步揭示了模型在视觉感知方面的根本缺陷以及推理能力中的瓶颈,这解释了它们在一般空间规划任务中表现较差的原因。我们的工作阐明了提升VLM空间规划能力的未来方向。我们的基准已公开发布于 https://github.com/UCSB-NLP-Chang/Visual-Spatial-Planning。

英文摘要:

Vision language models (VLMs) are an exciting emerging class of language models (LMs) that have merged classic LM capabilities with those of image processing systems. However, the ways that these capabilities combine are not always intuitive and warrant direct investigation. One understudied capability in VLMs is visual spatial planning -- the ability to comprehend the spatial arrangements of objects and devise action plans to achieve desired outcomes in visual scenes. In our study, we introduce VSP, a benchmark that 1) evaluates the spatial planning capability in these models in general, and 2) breaks down the visual planning task into finer-grained sub-tasks, including perception and reasoning, and measure the LMs capabilities in these sub-tasks. Our evaluation shows that both open-source and private VLMs fail to generate effective plans for even simple spatial planning tasks. Evaluations on the fine-grained analytical tasks further reveal fundamental deficiencies in the models' visual perception and bottlenecks in reasoning abilities, explaining their worse performance in the general spatial planning tasks. Our work illuminates future directions for improving VLMs' abilities in spatial planning. Our benchmark is publicly available at https://github.com/UCSB-NLP-Chang/Visual-Spatial-Planning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑