Existing Large Vision-Language Models (LVLMs) excel at matching concepts across multi-modal inputs but struggle with compositional concepts and high-level relationships between entities. This paper introduces Progressive multi-granular Vision-Language alignments (PromViL), a novel framework to enhance LVLMs' ability in performing grounded compositional visual reasoning tasks. Our approach constructs a hierarchical structure of multi-modal alignments, ranging from simple to complex concepts. By progressively aligning textual descriptions with corresponding visual regions, our model learns to leverage contextual information from lower levels to inform higher-level reasoning. To facilitate this learning process, we introduce a data generation process that creates a novel dataset derived from Visual Genome, providing a wide range of nested compositional vision-language pairs. Experimental results demonstrate that our PromViL framework significantly outperforms baselines on various visual grounding and compositional question answering tasks. The code is available at: https://github.com/lqh52/PromViL.
University of the Chinese Academy of Sciences（中国科学院大学）；Huazhong University of Science and Technology（华中科技大学）；National University of Singapore（新加坡国立大学）
Abstract visual reasoning connects mental abilities to the physical world, which is a crucial factor in cognitive development. Most toddlers display sensitivity to this skill, but it is not easy for machines. Aimed at it, we focus on the Raven Progressive Matrices Test, designed to measure cognitive reasoning. Recent work designed some black-boxes to solve it in an end-to-end fashion, but they are incredibly complicated and difficult to explain. Inspired by cognitive studies, we propose a Multi-Granularity Modularized Network (MMoN) to bridge the gap between the processing of raw sensory infor…
Beijing University of Posts and Telecommunications（北京邮电大学）；University of Surrey（萨里大学）
Fine-grained visual classification (FGVC) is much more challenging than traditional classification tasks due to the inherently subtle intra-class object variations. Recent works mainly tackle this problem by focusing on how to locate the most discriminative parts, more complementary parts, and parts of various granularities. However, less effort has been placed to which granularities are the most discriminative and how to fuse information cross multi-granularity. In this work, we propose a novel framework for fine-grained visual classification to tackle these problems. In particular, we propos…
Conan: Progressive Learning to Reason Like a Detective over Multi-Scale Visual Evidence
State Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University（多媒体信息处理国家重点实验室，计算机学院，北京大学）；WeChat AI, Tencent Inc., China（微信AI，腾讯公司，中国）
Most of the existing semantic segmentation approaches with image-level class labels as supervision, highly rely on the initial class activation map (CAM) generated from the standard classification network. In this paper, a novel "Progressive Patch Learning" approach is proposed to improve the local details extraction of the classification, producing the CAM better covering the whole object rather than only the most discriminative regions as in CAMs obtained in conventional classification models. "Patch Learning" destructs the feature maps into patches and independently processes each local pat…
Perceive, Reflect and Understand Long Video: Progressive Multi-Granular Clue Exploration with Interactive Agents
School of Electronic Engineering, Xidian University（西安电子科技大学电子工程学院）；Department of Computer Science and Engineering, Hong Kong University of Science and Technology（香港理工大学计算机科学与工程系）；College of Computer and Information, Hohai University（河海大学计算机与信息学院）
Chain-of-Glimpse: Search-Guided Progressive Object-Grounded Reasoning for Video Understanding
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, China.（网络与交换技术国家重点实验室，北京邮电大学，北京，中国）；Institute of Big Data, College of Computer Science and Artificial Intelligence, Fudan University, China.（大数据研究院，复旦大学计算机科学与人工智能学院，中国）；ARC Lab, Tencent PCG, Shenzhen, China.（腾讯PCG深圳实验室，深圳，中国）；School of Artificial Intelligence, Beijing University of Technology, Beijing, China.（北京理工大学人工智能学院，北京，中国）
Zhejiang Gongshang University（浙江工商大学）；Huazhong University of Science and Technology（华中科技大学）；University of Science and Technology of China（中国科学技术大学）
This paper targets the task of language-based video moment localization. The language-based setting of this task allows for an open set of target activities, resulting in a large variation of the temporal lengths of video moments. Most existing methods prefer to first sample sufficient candidate moments with various temporal lengths, and then match them with the given query to determine the target moment. However, candidate moments generated with a fixed temporal granularity may be suboptimal to handle the large variation in moment lengths. To this end, we propose a novel multi-stage Progressi…
Artemis: Geometry-Grounded Multi-Agent Driving World Models with Shared 3D State and Progressive Memory Update
University of Oxford（牛津大学）；University of Cambridge（剑桥大学）；University of Twente（特文特大学）；National University of Singapore（新加坡国立大学）；University of Bath（巴斯大学）
Fine-grained Visual Classification (FGVC) aims to identify objects from subcategories. It is a very challenging task because of the subtle inter-class differences. Existing research applies large-scale convolutional neural networks or visual transformers as the feature extractor, which is extremely computationally expensive. In fact, real-world scenarios of fine-grained recognition often require a more lightweight mobile network that can be utilized offline. However, the fundamental mobile network feature extraction capability is weaker than large-scale models. In this paper, based on the ligh…
University of East Anglia（东英吉利大学）；Newcastle University（纽卡斯尔大学）；Inception Institute of Artificial Intelligence（起源人工智能研究院）
Recognising remote sensing scene images remains challenging due to large visual-semantic discrepancies. These mainly arise due to the lack of detailed annotations that can be employed to align pixel-level representations with high-level semantic labels. As the tagging process is labour-intensive and subjective, we hereby propose a novel Multi-Granularity Canonical Appearance Pooling (MG-CAP) to automatically capture the latent ontological structure of remote sensing datasets. We design a granular framework that allows progressively cropping the input image to learn multi-grained features. For …
Renmin University of China（中国人民大学）；Microsoft Research Asia（微软亚洲研究院）
The neural machine translation model has suffered from the lack of large-scale parallel corpora. In contrast, we humans can learn multi-lingual translations even without parallel texts by referring our languages to the external world. To mimic such human learning behavior, we employ images as pivots to enable zero-resource translation learning. However, a picture tells a thousand words, which makes multi-lingual sentences pivoted by the same image noisy as mutual translations and thus hinders the translation model learning. In this work, we propose a progressive learning approach for image-piv…
VDialogUE: A Unified Evaluation Benchmark for Visually-grounded Dialogue
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）；University of Science and Technology of China（中国科学技术大学）
PSVMA+: Exploring Multi-granularity Semantic-visual Adaption for Generalized Zero-shot Learning
Institute of Information Science, Beijing Jiaotong University（北京交通大学信息科学研究所）；Hefei University of Technology（合肥工业大学）；National University of Singapore（新加坡国立大学）
MotionMERGE: A Multi-granular Framework for Human Motion Editing, Reasoning, Generation, and Explanation
Computer Vision Institute, School of Computer Science and Software Engineering, Shenzhen University（计算机视觉研究院，计算机科学与软件工程学院，深圳大学）；Guangdong Key Laboratory of Intelligent Information Processing, Shenzhen University（广东省智能信息处理重点实验室，深圳大学）；School of Computer Science, University of Nottingham Ningbo China（Nottingham Ningbo 中国计算机科学学院）；Department of Electrical and Computer Engineering, National University of Singapore（电子与计算机工程系，新加坡国立大学）；Department of Radiation Oncology, Stanford University（放射肿瘤科，斯坦福大学）；Sun Yat-sen University（中山大学）；School of Computer Science, University of Nottingham（计算机科学学院，Nottingham大学）
GBSK: Skeleton Clustering via Granular-ball Computing and Multi-Sampling for Large-Scale Data
College of Computer Science and Technology, Huaqiao University（华侨大学计算机科学与技术学院）；Fujian Provincial Key Laboratory of Computer Vision and Machine Learning, Huaqiao University（华侨大学计算机视觉与机器学习省级重点实验室）；Key Laboratory of Cyberspace Big Data Intelligent Security, Ministry of Education（教育部网络空间大数据安全重点实验室）；Chongqing Key Laboratory of Computational Intelligence, Chongqing University of Posts and Telecommunications（重庆邮电大学计算智能市级重点实验室）；Chongqing Key Laboratory of Big Data Intelligent Computing, Chongqing University of Posts and Telecommunications（重庆邮电大学大数据智能计算市级重点实验室）；College of Computer and Information Science, Chongqing Normal University（重庆师范大学计算机与信息科学学院）；College of Big Data and Intelligent Engine（大数据与智能工程学院）
Shape of Thought: Progressive Object Assembly via Visual Chain-of-Thought
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）科学与工程学院）；School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）；Sun Yat-sen University（中山大学）；The Hong Kong University of Science and Technology, Guangzhou（香港科学与技术大学（广州））；Shenzhen Future Network of Intelligence Institute (FNii-Shenzhen)（深圳未来网络智能研究所（FNii-Shenzhen））；Guangdong Provincial Key Laboratory of Future Networks of Intelligence, CUHK(SZ)（广东省未来网络智能重点实验室，CUHK(SZ)）
Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Models
University of California, Davis（加州大学戴维斯分校）；Virginia Tech（弗吉尼亚理工大学）；The Chinese University of Hong Kong（香港中文大学）；NVIDIA；Adobe Research（Adobe研究）；Fudan University（复旦大学）；Meta AI
ReasonCLIP-58M: Visually Grounded Commonsense Reasoning Supervision for CLIP
Khalifa University（卡利法大学）；University of Western Australia（西澳大学）；The Chinese University of Hong Kong（香港中文大学）；University of Melbourne（墨尔本大学）；University of Central Florida（佛罗里达中央大学）
GRAVA: Grounded Reasoning-to-Action Representation and Learning for Autonomous Driving
National Engineering Research Center of Electric Vehicles, Beijing Institute of Technology（北京理工大学电动车辆国家工程研究中心）；Shenzhen Automotive Research Institute, Beijing Institute of Technology（北京理工大学深圳汽车研究院）；Shenzhen Jiguangzhijie Technology Co., Ltd.（深圳极光智界科技有限公司）；Nanyang Technological University（南洋理工大学）
