Recent vision-language (VL) models are powerful, but can they reliably distinguish "right" from "left"? We curate three new corpora to quantify model comprehension of such basic spatial relations. These tests isolate spatial reasoning more precisely than existing datasets like VQAv2, e.g., our What'sUp benchmark contains sets of photographs varying only the spatial relations of objects, keeping their identity fixed (see Figure 1: models must comprehend not only the usual case of a dog under a table, but also, the same dog on top of the same table). We evaluate 18 VL models, finding that all perform poorly, e.g., BLIP finetuned on VQAv2, which nears human parity on VQAv2, achieves 56% accuracy on our benchmarks vs. humans at 99%. We conclude by studying causes of this surprising behavior, finding: 1) that popular vision-language pretraining corpora like LAION-2B contain little reliable data for learning spatial relationships; and 2) that basic modeling interventions like up-weighting preposition-containing instances or fine-tuning on our corpora are not sufficient to address the challenges our benchmarks pose. We are hopeful that these corpora will facilitate further research, and we release our data and code at https://github.com/amitakamath/whatsup_vlms.
Investigating Faithfulness in Large Audio Language Models
Concordia University（康科迪亚大学）；Mila - Quebec AI Institute（魁北克人工智能研究院）；Université Laval（拉瓦尔大学）；Birla Institute of Technology and Science, Pilani（比拉理工学院和科学学院，皮兰尼）
Investigating Cross-Modal Skill Injection: Scenarios, Methods, and Hyperparameters
State Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University（多媒体信息处理国家重点实验室，计算机科学学院，北京大学）；WeChat AI, Tencent Inc., China（腾讯公司，中国）；The University of Hong Kong（香港大学）
Investigating the Effectiveness of ChatGPT in Mathematical Reasoning and Problem Solving: Evidence from the Vietnamese National High School Graduation Examination
Eastern International University（东方国际大学）；International University, VNUHCM（越南国立大学胡志明市国际大学）
Why Do MLLMs Struggle with Spatial Understanding? A Systematic Analysis from Data to Architecture
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Tsinghua University（清华大学）；Harbin Institute of Technology（哈尔滨工业大学）；Wuhan AI Research（武汉人工智能研究院）；Institute of Microelectronics of the Chinese Academy of Sciences（中国科学院微电子研究所）
An Enigma of Artificial Reason: Investigating the Production-Evaluation Gap in Large Reasoning Models
NUS Department of Computer Science（国立新加坡大学计算机科学系）；MIT EECS（麻省理工学院电子工程与计算机科学系）；A*STAR（新加坡科技研究局）；Singapore-MIT Alliance for Research and Technology (SMART)（新加坡-麻省理工联合研究技术机构（SMART））
Stands to Reason: Investigating the Effect of Reasoning on Idiomaticity Detection
School of Computer Science, University of Sheffield, UK（谢菲尔德大学计算机科学学院）；Healthy Lifespan Institute, University of Sheffield（谢菲尔德大学健康寿命研究所）；Institute of Data Science and Artificial Intelligence, University of Exeter, UK（埃克塞特大学数据科学与人工智能研究所）
RealMath-Eval: Why SOTA Judges Struggle with Real Human Reasoning
University of Wisconsin–Madison（威斯康星大学麦迪逊分校）；East China Normal University（华东师范大学）；New York University（纽约大学）；Tongji University（同济大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
Shiny Stories, Hidden Struggles: Investigating the Representation of Disability Through the Lens of LLMs
Department of Foreign Languages and Literature, University of Verona（外国语言文学系，威尼斯大学）；Department of Information Engineering and Computer Science, University of Trento（信息工程与计算机科学系，特伦托大学）；Data and Web Science Group, University of Mannheim（数据与网络科学小组，曼海姆大学）
Investigating CoT Monitorability in Large Reasoning Models
Provable Responsible AI and Data Analytics (PRADA) Lab（可证明负责任的人工智能与数据分析师实验室）；King Abdullah University of Science and Technology（卡布斯大学）；University of Georgia（佐治亚大学）；University of Macau（澳门大学）
From $P(y|x)$ to $P(y)$: Investigating Reinforcement Learning in Pre-train Space
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；National University of Singapore（新加坡国立大学）；Tencent AI Lab（腾讯AI实验室）
Investigating the interaction of linguistic and mathematical reasoning in language models using multilingual number puzzles
School of Engineering and Applied Sciences, Harvard University（哈佛大学工程与应用科学学院）；Department of Linguistics, Harvard University（哈佛大学语言学系）；Kempner Institute for the Study of Natural and Artificial Intelligence at Harvard University（哈佛大学凯普纳自然与人工智能研究 institute）
