Modern vision-language models struggle with basic compositional reasoning, failing to bind attributes to objects or relations to their referents. Existing benchmarks either rely on noisy real images that conflate confounding visual variables with the reasoning failure, or use simplistic synthetic scenes lacking the realism modern VLMs are tuned for. We introduce \textbf{Auto-Comp}, a fully automated, concept-driven pipeline that bridges this gap by generating photorealistic compositional benchmarks at scale. Its core innovation is a \textit{parallel A/B construction}: for each concept, the pipeline emits a \textit{Minimal} sample (template caption, isolated objects on a white background) and a \textit{Contextual} sample (LLM-rewritten caption, objects embedded in a realistic scene), isolating core binding ability from visio-linguistic complexity. We instantiate \textit{four} task families spanning the two canonical axes of compositional binding: \textit{Color} and \textit{Shape-Color} (attribute binding), and \textit{Position} and \textit{Relative Size} (relational binding). We evaluate over 25 VLMs spanning CLIP, SigLIP, hard-negative-trained, and frontier generative models. The findings are consistent across architectures and scales: every model exhibits a large Swap-vs-Confusion gap, with low-entropy distractors (e.g., repeated objects or colors) exposing failures \textit{beyond} the known bag-of-words limitations. We further uncover a task-dependent trade-off: visio-linguistic context aids relational reasoning but hinders attribute binding through visual clutter. We publicly release the pipeline and benchmarks.
Despite the success of large vision and language models (VLMs) in many downstream applications, it is unclear how well they encode compositional information. Here, we create the Attribution, Relation, and Order (ARO) benchmark to systematically evaluate the ability of VLMs to understand different types of relationships, attributes, and order. ARO consists of Visual Genome Attribution, to test the understanding of objects' properties; Visual Genome Relation, to test for relational understanding; and COCO & Flickr30k-Order, to test for order sensitivity. ARO is orders of magnitude larger than pr…
Cross-Modal Masked Compositional Concept Modeling for Enhancing Visio-Linguistic Compositionality
MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China（中国科学技术大学，教育部脑启发智能感知与认知重点实验室）；Independent Researcher（独立研究员）
TokenSwap: Backdoor Attack on the Compositional Understanding of Large Vision-Language Models
School of Computer Science and Engineering, Southeast University, China（东南大学计算机科学与工程学院）；School of Electrical Engineering and Computer Science, University of Queensland, Australia（昆士兰大学电气工程与计算机科学学院）；Design Pillar, Singapore University of Technology and Design（新加坡科技设计大学设计学院）；Centre for Frontier AI Research (CFAR), Agency for Science, Technology and Research (A*STAR), Singapore（前沿人工智能研究中心（CFAR））；Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR), Singapore（高性能计算研究所（IHPC））
SSMNBench: Diagnosing Image-based Cross-View Human-Object Understanding via Single-View Sufficiency and Multi-View Necessity
The University of Queensland（昆士兰大学）；Australian Institute for Machine Learning, Adelaide University（阿德莱德大学澳大利亚机器学习研究所）；University of Technology Sydney（悉尼科技大学）；Follow Me AI Pty LTD（Follow Me AI有限公司）
Cognitive Science Program, University of California, Berkeley（加州大学伯克利分校认知科学项目）；Department of Electrical and Computer Engineering, University of California San Diego（加州大学圣地亚哥分校电气与计算机工程系）；School of Computer Science, Georgia Institute of Technology & Emory University（佐治亚理工学院计算机科学学院及埃默里大学）；Department of Computer Science, Johns Hopkins University（约翰霍普金斯大学计算机科学系）；Department of Cognitive Science, University of California San Diego（加州大学圣地亚哥分校认知科学系）；Equal Advising Department of Computer Science & Wilmer Eye Institute, Johns Hopkins University（约翰霍普金斯大学计算机科学系及威尔默眼科研究所）；Language Technologies Institute, Carnegie Mellon University（卡内基梅隆大学语言技术研究所）；Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）；Weinberg Institute for Cognitive Science, University of Michigan（密歇根大学韦恩伯格认知科学研究所）
CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decoding
UMass Amherst（马萨诸塞大学阿默斯特分校）；Wuhan University（武汉大学）；University of California, Los Angeles（加州大学洛杉矶分校）；MIT-IBM Watson AI Lab（MIT-IBM沃森人工智能实验室）；South China University of Technology（华南理工大学）
Nanyang Technological University（南洋理工大学）；The University of Hong Kong（香港大学）；SenseTime Research and Tetras.AI（商汤研究院与Tetras.AI）；Shanghai AI Laboratory（上海人工智能实验室）
Pre-trained vision-language models (VLMs) learn to align vision and language representations on large-scale datasets, where each image-text pair usually contains a bag of semantic concepts. However, existing open-vocabulary object detectors only align region embeddings individually with the corresponding features extracted from the VLMs. Such a design leaves the compositional structure of semantic concepts in a scene under-exploited, although the structure may be implicitly learned by the VLMs. In this work, we propose to align the embedding of bag of regions beyond individual regions. The pro…
