Rapid advancements in 3D vision-language (3D-VL) tasks have opened up new avenues for human interaction with embodied agents or robots using natural language. Despite this progress, we find a notable limitation: existing 3D-VL models exhibit sensitivity to the styles of language input, struggling to understand sentences with the same semantic meaning but written in different variants. This observation raises a critical question: Can 3D vision-language models truly understand natural language? To test the language understandability of 3D-VL models, we first propose a language robustness task for systematically assessing 3D-VL models across various tasks, benchmarking their performance when presented with different language style variants. Importantly, these variants are commonly encountered in applications requiring direct interaction with humans, such as embodied robotics, given the diversity and unpredictability of human language. We propose a 3D Language Robustness Dataset, designed based on the characteristics of human language, to facilitate the systematic study of robustness. Our comprehensive evaluation uncovers a significant drop in the performance of all existing models across various 3D-VL tasks. Even the state-of-the-art 3D-LLM fails to understand some variants of the same sentences. Further in-depth analysis suggests that the existing models have a fragile and biased fusion module, which stems from the low diversity of the existing dataset. Finally, we propose a training-free module driven by LLM, which improves language robustness. Datasets and code will be available at github.
GLIMPSE: Do Large Vision-Language Models Truly Think With Videos or Just Glimpse at Them?
UNC Chapel-Hill(北卡罗来纳大学教堂山分校);Microsoft(微软);NUS(新加坡国立大学);University of Minnesota(明尼苏达大学);Case Western Reserve University(凯斯西储大学);Rutgers University(罗格斯大学);UIUC(伊利诺伊大学厄巴纳-香槟分校)
Do Vision-Language Models Truly Perform Vision Reasoning? A Rigorous Study of the Modality Gap
College of Computing and Data Science, Nanyang Technological University, Singapore(南洋理工大学计算机与数据科学学院,新加坡);Alibaba-NTU Global e-Sustainability CorpLab (ANGEL)(阿里巴巴-国立大学全球可持续发展公司实验室(ANGEL));Tongyi Lab, Alibaba Group, China(阿里云实验室,阿里巴巴集团,中国)
Can Multimodal Large Language Models Truly Understand Small Objects?
Shanghai AI Laboratory(上海人工智能实验室);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));University of Science and Technology of China(中国科学技术大学);Fudan University(复旦大学);MMLab, The Chinese University of Hong Kong(香港中文大学MMLab)
From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Models
School of Computer Science, Sichuan University(四川大学计算机学院);School of Electronic and Computer Engineering, Peking University Shenzhen Graduate School(北京大学深圳研究生院电子与计算机工程学院);Institute of Artificial Intelligence (TeleAI), China Telecom and Northwestern Polytechnical University(中国电信与西北工业大学人工智能研究院(TeleAI))
VisionGPT: Vision-Language Understanding Agent Using Generalized Multimodal Framework
Stanford University(斯坦福大学);Peking University(北京大学);Harvard University(哈佛大学);University of Washington, Seattle(华盛顿大学西雅图分校);University of California, Los Angeles(加利福尼亚大学洛杉矶分校)
SpatialBot: Precise Spatial Understanding with Vision Language Models
School of Artificial Intelligence, Shanghai Jiao Tong University(上海交通大学人工智能学院);Stanford University(斯坦福大学);BAAI(北京智源人工智能研究院);Peking University(北京大学);University of Oxford(牛津大学);Southeast University(东南大学)
Vision-Language Pre-training with Object Contrastive Learning for 3D Scene Understanding
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);College of Computer Science and Software Engineering, Shenzhen University(深圳大学计算机与软件学院);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Research Center of Artificial Intelligence, Peng Cheng Laboratory(鹏城实验室人工智能研究中心)
Contextualized Visual Personalization in Vision-Language Models
Department of Electrical and Computer Engineering, Seoul National University, Seoul, South Korea(电气电子工程系,首尔国立大学,首尔,韩国);Interdisciplinary Program in Artificial Intelligence, Seoul National University, Seoul, Korea(人工智能交叉学科项目,首尔国立大学,首尔,韩国)