多模态大语言模型（MLLMs）在视觉识别和语义理解方面取得了显著进展。然而，其进行精确组合空间推理的能力仍 largely 未被探索。现有基准测试通常涉及相对简单的任务，依赖语义近似或粗略的相对位置，而其评估指标通常有限，缺乏严谨的数学公式。为弥合这一差距，我们引入TangramPuzzle，一个以几何为基础的基准测试，通过经典Tangram游戏的视角评估组合空间推理。我们提出Tangram Construction Expression（TCE），一个符号几何框架，将Tangram组装 grounded 在精确、可由机器验证的坐标规范中，以缓解视觉近似的模糊性。我们设计了两个互补的任务：轮廓预测，要求从局部组件推断出整体形状，以及端到端代码生成，要求解决逆向几何组装问题。我们对先进的开源和专有模型进行了广泛的评估实验，揭示了一个有趣的见解：MLLMs倾向于优先匹配目标轮廓，而忽视几何约束，导致碎片的扭曲或变形。
英文摘要：
Multimodal Large Language Models (MLLMs) have achieved remarkable progress in visual recognition and semantic understanding, yet precise compositional spatial reasoning under geometric constraints remains underexplored. Existing benchmarks mainly assess coarse spatial relations and rarely support rigorous geometric verification or multiple valid solutions in constructive tasks. To address these limitations, we introduce TangramPuzzle, a benchmark comprising 668 validated configurations and 1,336 instances for evaluating compositional spatial reasoning under geometric constraints. We formulate the Tangram Construction Expression (TCE) to encode tangram configurations with machine-verifiable geometric specifications, together with a multi-solution-aware verifier that separately evaluates geometric validity and silhouette fidelity. TangramPuzzle covers two complementary evaluation tasks: part-to-whole Outline Prediction and whole-to-part End-to-End Assembly Generation. We conduct extensive evaluation experiments on advanced open-source and proprietary models, revealing an interesting insight: MLLMs tend to prioritize matching the target silhouette while neglecting geometric constraints, leading to distortions or deformations of the pieces.
Evaluating Morphological Compositional Generalization in Large Language Models
EPFL（瑞士联邦理工学院）；Idiap Research Institute（Idiap研究机构）；Duke University（杜克大学）；Brandeis University（布兰迪斯大学）；LMU Munich（慕尼黑大学）；University of Cambridge（剑桥大学）；Johns Hopkins University（约翰霍普金斯大学）；Università della Svizzera Italiana（瑞士意大利大学）；New York University（纽约大学）
Evaluating Compositional Scene Understanding in Multimodal Generative Models
Department of Psychology, University of California, Los Angeles（加州大学洛杉矶分校心理学系）；Microsoft Research, NYC（微软研究院纽约分部）；Air Force Research Laboratory（美国空军研究实验室）；Department of Statistics, University of California, Los Angeles（加州大学洛杉矶分校统计学系）
Massachusetts Institute of Technology（麻省理工学院）；Google Research（谷歌研究院）
Despite their strong performance on many tasks, pre-trained language models have been shown to struggle on out-of-distribution compositional generalization. Meanwhile, recent work has shown considerable improvements on many NLP tasks from model scaling. Can scaling up model size also improve compositional generalization in semantic parsing? We evaluate encoder-decoder models up to 11B parameters and decoder-only models up to 540B parameters, and compare model scaling curves for three different methods for applying a pre-trained language model to a new task: fine-tuning all parameters, prompt t…
An Examination of the Compositionality of Large Generative Vision-Language Models
AI Thrust, The Hong University of Science and Technology (Guangzhou)（香港科技大学（广州）AI方向）；Department of Computer Science and Engineering, The Hong Kong University of Science and Technology（香港科技大学计算机科学与工程系）
ObjectCompose: Evaluating Resilience of Vision-Based Models on Object-to-Background Compositional Changes
Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）；Center of Secure Cyber-Physical Security Systems（安全网络物理系统中心）；Australian National University（澳大利亚国立大学）；Linköping University（林雪平大学）
ChatGPT as Data Augmentation for Compositional Generalization: A Case Study in Open Intent Detection
Ingenuity Labs Research Institute（创新实验室研究所）；Queen’s University（女王大学）；Smith School of Business（史密斯商学院）；Department of Electrical and Computer Engineering（电气与计算机工程系）
Despite the success of large vision and language models (VLMs) in many downstream applications, it is unclear how well they encode compositional information. Here, we create the Attribution, Relation, and Order (ARO) benchmark to systematically evaluate the ability of VLMs to understand different types of relationships, attributes, and order. ARO consists of Visual Genome Attribution, to test the understanding of objects' properties; Visual Genome Relation, to test for relational understanding; and COCO & Flickr30k-Order, to test for order sensitivity. ARO is orders of magnitude larger than pr…
Sri Sathya Sai Institute of Higher Learning（斯里萨提亚赛高等教育学院）；Samsung Research（三星研究院）
Visual Question Answering (VQA) needs a means of evaluating the strengths and weaknesses of models. One aspect of such an evaluation is the evaluation of compositional generalisation, or the ability of a model to answer well on scenes whose scene-setups are different from the training set. Therefore, for this purpose, we need datasets whose train and test sets differ significantly in composition. In this work, we present several quantitative measures of compositional separation and find that popular datasets for VQA are not good evaluators. To solve this, we present Uncommon Objects in Unseen …
Compositional Reasoning with Transformers, RNNs, and Chain of Thought
Courant Institute of Mathematical Sciences（Courant数学科学研究所）；New York University（纽约大学）；Center for Data Science（数据科学中心）；Center for Computational Mathematics（计算数学中心）；Flatiron Institute（Flatiron研究所）
Language models (LMs) can perform complex reasoning either end-to-end, with hidden latent state, or compositionally, with transparent intermediate state. Composition offers benefits for interpretability and safety, but may need workflow support and infrastructure to remain competitive. We describe iterated decomposition, a human-in-the-loop workflow for developing and refining compositional LM programs. We improve the performance of compositions by zooming in on failing components and refining them through decomposition, additional context, chain of thought, etc. To support this workflow, we d…
Funny or Persuasive, but Not Both: Evaluating Fine-Grained Multi-Concept Control in LLMs
University of Cambridge（剑桥大学）；CISPA Helmholtz Center for Information Security（CISPA亥姆霍兹信息安全中心）；Apta（Apta公司）；Royal Holloway, University of London（伦敦大学皇家霍洛威学院）
The University of Hong Kong（香港大学）；University of Wollongong（伍伦贡大学）；Tencent AI Lab（腾讯人工智能实验室）；Australian Centre for Robotic Vision, University of Adelaide（阿德莱德大学澳大利亚机器人视觉中心）
Referring expression comprehension (REF) aims at identifying a particular object in a scene by a natural language expression. It requires joint reasoning over the textual and visual domains to solve the problem. Some popular referring expression datasets, however, fail to provide an ideal test bed for evaluating the reasoning ability of the models, mainly because 1) their expressions typically describe only some simple distinctive properties of the object and 2) their images contain limited distracting information. To bridge the gap, we propose a new dataset for visual reasoning in context of …
MedKIT: Evaluating Knowledge Integration and Generalization in Large Language Models
University of Tübingen（蒂宾根大学）；Tübingen AI Center（蒂宾根人工智能中心）；Helmholtz Munich（亥姆霍兹慕尼黑中心）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；University of Virginia（弗吉尼亚大学）；University of British Columbia（不列颠哥伦比亚大学）；Harvard Medical School（哈佛医学院）；Technical University of Munich（慕尼黑工业大学）
Scene Graph-Guided Generative AI Framework for Synthesizing and Evaluating Industrial Hazard Scenarios
Ph.D. Student, Dept. of Civil Eng., University of Texas at Arlington. E-mail；Assistant Professor, Dept. of Computer Sci. \& Eng., University of Texas at Arlington. E-mail；Assistant Professor, Dept. of Civil Eng., University of Texas at Arlington. E-mail
FramingQA: Does the Question Shape the Answer? Measuring the Compositional Framing Effect
University of Oxford（牛津大学）；Oxford Internet Institute（牛津互联网研究院）；Thomson Reuters（汤森路透）；University of California San Diego（加州大学圣迭戈分校）；University of Cambridge（剑桥大学）；Institute of Biomedical Engineering, University of Oxford（牛津大学生物医学工程研究所）；LMU Munich（慕尼黑大学）
MUSE: Benchmarking Large Vision-Language Models on Multi-Modal Understanding in Situated Education
AI Singapore, National University of Singapore（新加坡国立大学新加坡人工智能）；School of Computing, National University of Singapore（新加坡国立大学计算学院）；Institute of Advanced Intelligence and Computing, A*STAR（新加坡科技研究局先进智能与计算研究所）
