Compositional Reasoning (CR) entails grasping the significance of attributes, relations, and word order. Recent Vision-Language Models (VLMs), comprising a visual encoder and a Large Language Model (LLM) decoder, have demonstrated remarkable proficiency in such reasoning tasks. This prompts a crucial question: have VLMs effectively tackled the CR challenge? We conjecture that existing CR benchmarks may not adequately push the boundaries of modern VLMs due to the reliance on an LLM-only negative text generation pipeline. Consequently, the negatives produced either appear as outliers from the natural language distribution learned by VLMs' LLM decoders or as improbable within the corresponding image context. To address these limitations, we introduce ConMe -- a compositional reasoning benchmark and a novel data generation pipeline leveraging VLMs to produce `hard CR Q&A'. Through a new concept of VLMs conversing with each other to collaboratively expose their weaknesses, our pipeline autonomously generates, evaluates, and selects challenging compositional reasoning questions, establishing a robust CR benchmark, also subsequently validated manually. Our benchmark provokes a noteworthy, up to 33%, decrease in CR performance compared to preceding benchmarks, reinstating the CR challenge even for state-of-the-art VLMs.
WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所多模态人工智能系统国家重点实验室 (MAIS)）
Rethinking Video-Language Model from the Language Input Perspective
School of Software Engineering, Huazhong University of Science and Technology（华中科技大学软件学院）；Nanyang Technological University, Singapore（新加坡南洋理工大学）；University College London（伦敦大学学院）；Huazhong University of Science and Technology（华中科技大学）；Wuhan University（武汉大学）
GeoBench: Rethinking Multimodal Geometric Problem-Solving via Hierarchical Evaluation
Shanghai Jiao Tong University（上海交通大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Fudan University（复旦大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
Rethinking Concept Bottleneck Models: From Pitfalls to Solutions
Middle East Technical University (METU)（中东技术大学）；Helmholtz Munich（亥姆霍兹慕尼黑中心）；Munich Center for Machine Learning（慕尼黑机器学习中心）；Munich Data Science Institute（慕尼黑数据科学研究所）；Technical University of Munich (TUM)（慕尼黑工业大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；RTG Neuroexplicit Models（神经显式模型研究训练组）；Robotics & AI Center (ROMER), METU（中东技术大学机器人与人工智能中心（ROMER））
Seeing the Trees for the Forest: Rethinking Weakly-Supervised Medical Visual Grounding
Australian Institute for Machine Learning, University of Adelaide（澳大利亚机器学习研究所，阿德莱德大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Macquarie University（麦考瑞大学）；Hanoi University of Science and Technology（河内科学技术大学）；University of Wollongong（沃林根大学）；Flinders University（弗林德斯大学）
