Vision language models (VLMs) are an exciting emerging class of language models (LMs) that have merged classic LM capabilities with those of image processing systems. However, the ways that these capabilities combine are not always intuitive and warrant direct investigation. One understudied capability in VLMs is visual spatial planning -- the ability to comprehend the spatial arrangements of objects and devise action plans to achieve desired outcomes in visual scenes. In our study, we introduce VSP, a benchmark that 1) evaluates the spatial planning capability in these models in general, and 2) breaks down the visual planning task into finer-grained sub-tasks, including perception and reasoning, and measure the LMs capabilities in these sub-tasks. Our evaluation shows that both open-source and private VLMs fail to generate effective plans for even simple spatial planning tasks. Evaluations on the fine-grained analytical tasks further reveal fundamental deficiencies in the models' visual perception and bottlenecks in reasoning abilities, explaining their worse performance in the general spatial planning tasks. Our work illuminates future directions for improving VLMs' abilities in spatial planning. Our benchmark is publicly available at https://github.com/UCSB-NLP-Chang/Visual-Spatial-Planning.
Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs
Tongji University(同济大学);Nanjing University of Posts and Telecommunications(南京邮电大学);University of Science and Technology of China(中国科学技术大学);Shanghai AI Laboratory(上海人工智能实验室);The Chinese University of Hong Kong(香港中文大学);Nanjing University(南京大学)
Learning to Reason About Implicit Multimodal Harm: From Diagnosis to Generalization
Max Planck Institute for Informatics(马克斯·普朗克院信息研究所);Saarland Informatics Campus(萨尔兰州信息校园);Saarland University(萨尔兰州大学);The University of Edinburgh(爱丁堡大学);Samsung AI Center, Cambridge(三星AI中心,剑桥)
MedHallTune: An Instruction-Tuning Benchmark for Mitigating Medical Hallucination in Vision-Language Models
The Chinese University of Hong Kong(香港中文大学);Institute of Medical Intelligence and XR, The Chinese University of Hong Kong(香港中文大学医学智能与XR研究所);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
How effective are VLMs in assisting humans in inferring the quality of mental models from Multimodal short answers?
Department of Computer Science and Engineering, IIT Bombay(印度理工学院班加罗尔分校计算机科学与工程系);Center for Educational Technology, IIT Bombay(印度理工学院班加罗尔分校教育技术中心);School of Management, Mahindra University(马恒达大学管理学院)
On the Risk of Misleading Reports: Diagnosing Textual Biases in Multimodal Clinical AI
MICS, CentraleSupélec - Université Paris-Saclay, France(MICS,中央圣艾尔布兰大学-巴黎萨克雷大学,法国);Google DeepMind, London, UK(谷歌DeepMind,伦敦,英国);CONICET, Universidad de Buenos Aires, Argentina(CONICET,布宜诺斯艾利斯大学,阿根廷)
MMBench: Is Your Multi-modal Model an All-around Player?
Shanghai AI Laboratory(上海人工智能实验室);Nanyang Technological University(南洋理工大学);National University of Singapore(新加坡国立大学);Zhejiang University(浙江大学);The Chinese University of Hong Kong(香港中文大学)
From Scientific Crop Imagery to Agronomic Reasoning: A Multimodal LLM Benchmark for Plant Phenotyping
University of Memphis(孟菲斯大学);University of Missouri(密苏里大学);University of Arkansas Division of Agriculture(阿肯色大学农业分部);Mississippi State University(密西西比州立大学)