Humans appear to represent objects when reasoning about physics with coarse, volumetric "bodies" that smooth concavities, trading fine visual detail for efficient physical predictions. Yet, the structure of these representations remains largely unknown. Segmentation models, in contrast, are trained for pixel-accurate masks that may misalign with such bodies. We ask whether and when these models nonetheless acquire human-like object representations. Using a time-to-collision (TTC) and change detection (CD) behavioral task with data from 178 and 50 human participants, respectively, we introduce a pipeline and an alignment metric to compare the visual representations of segmentation models to those of humans. We do this systematically on multiple architectures (DINOv2, SegFormer, DeepLabV3+, and UPerNet), varying their size and training time. We find that briefly trained models segment objects too coarsely, aligning poorly with humans, while fully trained models segment objects too finely. For each model, there is an intermediate training regime that best matches the coarse bodies observed in human behaviour, and larger models tend to reach it earlier. We show these bodies emerge under resource constraints in general-purpose vision models, providing computational support to resource-rational accounts of human cognition. This work provides a foundational framework for testing alignment between vision models and humans and shows there is a growing gap between the state-of-the-art in artificial intelligence and human cognition, driven by scaling model size and training.
Modeling instance-level context and object-object relationships is extremely challenging. It requires reasoning about bounding boxes of different classes, locations \etc. Above all, instance-level spatial reasoning inherently requires modeling conditional distributions on previous detections. Unfortunately, our current object detection systems do not have any {\bf memory} to remember what to condition on! The state-of-the-art object detectors still detect all object in parallel followed by non-maximal suppression (NMS). While memory has been used for tasks such as captioning, they mostly use i…
Artificial and Natural Intelligence Toulouse Institute（图卢兹人工智能与自然智能研究所）；Université de Toulouse（图卢兹大学）；Centre de Recherche Cerveau & Cognition (CerCo), CNRS（法国国家科学研究中心大脑与认知研究中心）
Visual understanding requires comprehending complex visual relations between objects within a scene. Here, we seek to characterize the computational demands for abstract visual reasoning. We do this by systematically assessing the ability of modern deep convolutional neural networks (CNNs) to learn to solve the "Synthetic Visual Reasoning Test" (SVRT) challenge, a collection of twenty-three visual reasoning problems. Our analysis reveals a novel taxonomy of visual reasoning tasks, which can be primarily explained by both the type of relations (same-different vs. spatial-relation judgments) and…
National University of Singapore（新加坡国立大学）；Institute for Infocomm Research, A*STAR（新加坡科技研究局信息通信研究院）；New York University（纽约大学）；University of California Santa Cruz（加利福尼亚大学圣克鲁兹分校）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Recent work in computer vision and cognitive reasoning has given rise to an increasing adoption of the Violation-of-Expectation (VoE) paradigm in synthetic datasets. Inspired by infant psychology, researchers are now evaluating a model's ability to label scenes as either expected or surprising with knowledge of only expected scenes. However, existing VoE-based 3D datasets in physical reasoning provide mainly vision data with little to no heuristics or inductive biases. Cognitive models of physical reasoning reveal infants create high-level abstract representations of objects and interactions. …
While attention has been an increasingly popular component in deep neural networks to both interpret and boost the performance of models, little work has examined how attention progresses to accomplish a task and whether it is reasonable. In this work, we propose an Attention with Reasoning capability (AiR) framework that uses attention to understand and improve the process leading to task outcomes. We first define an evaluation metric based on a sequence of atomic reasoning operations, enabling a quantitative measurement of attention that considers the reasoning process. We then collect human…
Solving the Clustering Reasoning Problems by Modeling a Deep-Learning-Based Probabilistic Model
Beijing Engineering Research Center of Industrial Spectrum Imaging, School of Automation and Electrical Engineering, University of Science and Technology Beijing（工业光谱成像北京工程研究中心，自动化与电气工程学院，北京科技大学）
Temporal relational modeling in video is essential for human action understanding, such as action recognition and action segmentation. Although Graph Convolution Networks (GCNs) have shown promising advantages in relation reasoning on many tasks, it is still a challenge to apply graph convolution networks on long video sequences effectively. The main reason is that large number of nodes (i.e., video frames) makes GCNs hard to capture and model temporal relations in videos. To tackle this problem, in this paper, we introduce an effective GCN module, Dilated Temporal Graph Reasoning Module (DTGR…
SurGo-R1: Benchmarking and Modeling Contextual Reasoning for Operative Zone in Surgical Video
National University of Singapore, Singapore（新加坡国立大学）；Southern Medical University, China（南方医科大学）；Guangzhou Research Translation and Innovation Institute, National University of Singapore, China（广州研究翻译与创新研究院，新加坡国立大学，中国）
Interleaved Latent Visual Reasoning with Selective Perceptual Modeling
China University of Geosciences, Wuhan（中国地质大学（武汉））；Shanghai Innovation Institute（上海创新研究院）；University of Southern California（南加州大学）；Fudan University（复旦大学）；Zhejiang University（浙江大学）；Shanghai Jiao Tong University（上海交通大学）
Unified Modeling of Lane and Lane Topology for Driving Scene Reasoning
School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）；Zhongguancun Academy, Beijing, China（中关村学院，北京，中国）；Hangzhou International Innovation Institute, Beihang University（杭州国际创新院，北京航空航天大学）；Meituan, Beijing, China（美团，北京，中国）
University of California, Los Angeles（加州大学洛杉矶分校）；Inception Institute of Artificial Intelligence（人工智能初创研究院）；International Center for AI and Robot Autonomy (CARA)（国际人工智能与机器人自主中心（CARA））
We propose a novel model to address the task of Visual Dialog which exhibits complex dialog structures. To obtain a reasonable answer based on the current question and the dialog history, the underlying semantic dependencies between dialog entities are essential. In this paper, we explicitly formalize this task as inference in a graphical model with partially observed nodes and unknown graph structures (relations in dialog). The given dialog entities are viewed as the observed nodes. The answer to a given question is represented by a node with missing value. We first introduce an Expectation M…
Psychologists recognize Raven's Progressive Matrices as a very effective test of general human intelligence. While many computational models have been developed by the AI community to investigate different forms of top-down, deliberative reasoning on the test, there has been less research on bottom-up perceptual processes, like Gestalt image completion, that are also critical in human test performance. In this work, we investigate how Gestalt visual reasoning on the Raven's test can be modeled using generative image inpainting techniques from computer vision. We demonstrate that a self-supervi…
Think, then Score: Decoupled Reasoning and Scoring for Video Reward Modeling
University of Science and Technology of China（中国科学技术大学）；Kling Team, Kuaishou Technology（快手科技 Kling 团队）；Institute of Software Chinese Academy of Sciences（中国科学院软件研究所）
Modeling Beyond MOS: Quality Assessment Models Must Integrate Context, Reasoning, and Multimodality
R&D Lab F-Initiatives（F-Initiatives研发实验室）；Université d’Orleans（奥尔良大学）；Université Sorbonne Paris Nord（巴黎-索邦大学）；University of Dubai（迪拜大学）；IULM University（IULM大学）
Understanding actions and gestures in video streams requires temporal reasoning of the spatial content from different time instants, i.e., spatiotemporal (ST) modeling. In this survey paper, we have made a comparative analysis of different ST modeling techniques for action and gecture recognition tasks. Since Convolutional Neural Networks (CNNs) are proved to be an effective tool as a feature extractor for static images, we apply ST modeling techniques on the features of static images from different time instants extracted by CNNs. All techniques are trained end-to-end together with a CNN feat…
REMEMBER: Retrieval-based Explainable Multimodal Evidence-guided Modeling for Brain Evaluation and Reasoning in Zero- and Few-shot Neurodegenerative Diagnosis
Lausanne University Hospital (CHUV)（洛桑大学医院（CHUV））；University of Lausanne (UNIL)（洛桑大学（UNIL））；Vietnam National University（越南国立大学）；University of Science, Vietnam National University Ho Chi Minh City（越南国立大学胡志明市科学大学）；International University, Vietnam National University Ho Chi Minh City（越南国立大学胡志明市国际大学）
