Multi-modal large language models (MLLMs) have shown impressive generalization across tasks using images and text modalities. While their extension to video has enabled tasks such as video question answering and video captioning, their pixel-level visual grounding abilities are less studied. In this work, we raise the pertinent question of whether motion is used in pixel-level visual grounding and whether video MLLMs can segment objects based on natural language expressions describing their motion patterns. We identify the shortcomings in the current benchmarks, where we show that a single frame can often suffice for capturing the motion referring expression without any temporal reasoning. To address this, we introduce four motion-centric probing techniques, particularly designed for the visual grounding task, to study video MLLMs' ability to identify true motion from a fake one and their ability to grasp the motion order. Consequently, we provide a motion-centric benchmark, MoCentric-Bench. It ensures that video MLLMs are evaluated towards leveraging the interaction between motion and language rather than being dominated by static appearance cues emphasized in existing visual grounding datasets. We further establish strong single-image baselines that are on par with or outperform prior methods. Finally, we explore simple motion-centric adaptation techniques that provide state-of-the-art performance on our MoCentric-Bench. Our motion-centric benchmark, evaluation and findings challenge future models to improve dense spatiotemporal grounding and pixel-level understanding within videos. Code and datasets will be made publicly available at https://github.com/MSiam/PixFoundation-2.0.git.
Visual grounding is a ubiquitous building block in many vision-language tasks and yet remains challenging due to large variations in visual and linguistic features of grounding entities, strong context effect and the resulting semantic ambiguities. Prior works typically focus on learning representations of individual phrases with limited context information. To address their limitations, this paper proposes a language-guided graph representation to capture the global context of grounding entities and their relations, and develop a cross-modal graph matching strategy for the multiple-phrase vis…
Referring expression grounding is an important and challenging task in computer vision. To avoid the laborious annotation in conventional referring grounding, unpaired referring grounding is introduced, where the training data only contains a number of images and queries without correspondences. The few existing solutions to unpaired referring grounding are still preliminary, due to the challenges of learning image-text matching and lack of the top-down guidance with unpaired data. In this paper, we propose a novel bidirectional cross-modal matching (BiCM) framework to address these challenges…
LSVG: Language-Guided Scene Graphs with 2D-Assisted Multi-Modal Encoding for 3D Visual Grounding
School of Automation Science and Engineering, South China University of Technology（自动化科学与工程学院，华南理工大学）；School of Future Technology, South China University of Technology（未来技术学院，华南理工大学）
RGBX-R1: Visual Modality Chain-of-Thought Guided Reinforcement Learning for Multimodal Grounding
School of Artificial Intelligence, Tianjin University（天津大学人工智能学院）；Low-Altitude Intelligence Lab, Xiong’an National Innovation Center（雄安国家创新中心低空智能实验室）；Xiong’an Guochuang Lantian Technology Co., Ltd.（雄安国创莲田科技有限公司）；College of Electronic Science and Technology, National University of Defense Technology（国防科技大学电子科学学院）
Visual question answering (VQA) is the multi-modal task of answering natural language questions about an input image. Through cross-dataset adaptation methods, it is possible to transfer knowledge from a source dataset with larger train samples to a target dataset where training set is limited. Suppose a VQA model trained on one dataset train set fails in adapting to another, it is hard to identify the underlying cause of domain mismatch as there could exists a multitude of reasons such as image distribution mismatch and question distribution mismatch. At UCLA, we are working on a VQG module t…
SeCG: Semantic-Enhanced 3D Visual Grounding via Cross-modal Graph Attention
School of Automation Science and Engineering, South China University of Technology（华南理工大学自动化科学与工程学院）；School of Software Engineering, South China University of Technology（华南理工大学软件学院）
AugRefer: Advancing 3D Visual Grounding via Cross-Modal Augmentation and Spatial Relation-based Referring
University of Science and Technology of China（中国科学技术大学）；Singapore University of Technology and Design（新加坡科技设计大学）；Hefei University of Technology（合肥工业大学）
HiVG: Hierarchical Multimodal Fine-grained Modulation for Visual Grounding
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Pengcheng Laboratory（鹏城实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））
KG-ViP: Bridging Knowledge Grounding and Visual Perception in Multi-modal LLMs for Visual Question Answering
University of Science and Technology of China（中国科学技术大学）；Data Darkness Lab, MIRACLE Center, USTC（数据黑暗实验室，MIRACLE中心，中国科学技术大学）；School of Computer Science and Technology, Xidian University（西安电子科技大学计算机科学与技术学院）
University of Science and Technology of China（中国科学技术大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）；Microsoft（微软）；JD Explore Academy（京东探索研究院）；Amazon（亚马逊）；University of Sydney（悉尼大学）；Shanghai Artificial Intelligent Laboratory（上海人工智能实验室）
In this work, we explore neat yet effective Transformer-based frameworks for visual grounding. The previous methods generally address the core problem of visual grounding, i.e., multi-modal fusion and reasoning, with manually-designed mechanisms. Such heuristic designs are not only complicated but also make models easily overfit specific data distributions. To avoid this, we first propose TransVG, which establishes multi-modal correspondences by Transformers and localizes referred regions by directly regressing box coordinates. We empirically show that complicated fusion modules can be replace…
Hierarchical Alignment-enhanced Adaptive Grounding Network for Generalized Referring Expression Comprehension
School of Computer Science and Technology, Xi’an Jiaotong University（西安交通大学计算机科学与技术学院）；Institute of Big Data, Fudan University（复旦大学大数据学院）；Shanghai University of Finance and Economics（上海财经大学）；Nanyang Technological University（南洋理工大学）
University of Electronic Science and Technology of China（电子科技大学）
Crowd understanding has aroused the widespread interest in vision domain due to its important practical significance. Unfortunately, there is no effort to explore crowd understanding in multi-modal domain that bridges natural language and computer vision. Referring expression comprehension (REF) is such a representative multi-modal task. Current REF studies focus more on grounding the target object from multiple distinctive categories in general scenarios. It is difficult to applied to complex real-world crowd understanding. To fill this gap, we propose a new challenging dataset, called RefCro…
CAS Key Laboratory of GIPAS, University of Science and Technology of China（中国科学技术大学中国科学院GIPAS重点实验室）；University of Rochester（罗切斯特大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）
In this paper, we present a neat yet effective transformer-based framework for visual grounding, namely TransVG, to address the task of grounding a language query to the corresponding region onto an image. The state-of-the-art methods, including two-stage or one-stage ones, rely on a complex module with manually-designed mechanisms to perform the query reasoning and multi-modal fusion. However, the involvement of certain mechanisms in fusion module design, such as query decomposition and image scene graph, makes the models easily overfit to datasets with specific scenarios, and limits the plen…
Unified Representation Space for 3D Visual Grounding
School of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics（计算机科学与技术学院，南京航空航天大学）；Urban Data Science Section, Delft University of Technology（数据科学部，代尔夫特理工大学）
School of Data Science（数据科学学院）；Engineering East China Normal University Shanghai, China（工程 东华师范大学 上海中国）；OpenScience Lab Shanghai AI Laboratory Shanghai, China（OpenScience Lab 上海AI实验室 上海中国）；School of Life Science（生命科学学院）；Technology Xi'an Jiaotong University Xi'an, China（技术 西安交通大学 西安中国）；Medical Artificial Intelligence Laboratory Westlake University Hangzhou, China（医学人工智能实验室 西湖大学 杭州中国）
The Chinese University of Hong Kong（香港中文大学）；SenseTime Research（商汤科技研究院）
Referring expression grounding aims at locating certain objects or persons in an image with a referring expression, where the key challenge is to comprehend and align various types of information from visual and textual domain, such as visual attributes, location and interactions with surrounding regions. Although the attention mechanism has been successfully applied for cross-modal alignments, previous attention models focus on only the most dominant features of both modalities, and neglect the fact that there could be multiple comprehensive textual-visual correspondences between images and r…
BARE: Towards Bias-Aware and Reasoning-Enhanced One-Tower Visual Grounding
School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；Pengcheng Laboratory（鹏城实验室）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
3D-SPS: Single-Stage 3D Visual Grounding via Referred Point Progressive Selection
Institute of Artificial Intelligence, Beihang University（北京航空航天大学人工智能研究院）；Hangzhou Innovation Institute, Beihang University（北京航空航天大学杭州创新研究院）；Meituan Inc.（美团公司）
In this paper, we study the problem of visual grounding by considering both phrase extraction and grounding (PEG). In contrast to the previous phrase-known-at-test setting, PEG requires a model to extract phrases from text and locate objects from images simultaneously, which is a more practical setting in real applications. As phrase extraction can be regarded as a $1$D text segmentation problem, we formulate PEG as a dual detection problem and propose a novel DQ-DETR model, which introduces dual queries to probe different features from image and text for object prediction and phrase mask pred…
MedSG-Bench: A Benchmark for Medical Image Sequences Grounding
Beijing University of Posts and Telecommunications（北京邮电大学）；Tianjin University（天津大学）；South China Hospital, Medical School, Shenzhen University（深圳大学医学院南方医院）
ASAP: Advancing Semantic Alignment Promotes Multi-Modal Manipulation Detecting and Grounding
School of Computer Science and Information Engineering, Hefei University of Technology, China（计算机科学与信息工程学院，合肥工业大学，中国）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, China（人工智能研究所，合肥综合性国家科学中心，中国）
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）；The Chinese University of Hong Kong（香港中文大学）
Visual grounding is a task to locate the target indicated by a natural language expression. Existing methods extend the generic object detection framework to this problem. They base the visual grounding on the features from pre-generated proposals or anchors, and fuse these features with the text embeddings to locate the target mentioned by the text. However, modeling the visual features from these predefined locations may fail to fully exploit the visual context and attribute information in the text query, which limits their performance. In this paper, we propose a transformer-based framework…
Visual Grounding with Multi-modal Conditional Adaptation
Wuhan University of Technology（武汉理工大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Sanya Science and Education Innovation Park of Wuhan University of Technology（武汉理工大学三亚科教创新园）
Applied Artificial Intelligence Institute, Deakin University（迪肯大学应用人工智能研究所）
The current success of modern visual reasoning systems is arguably attributed to cross-modality attention mechanisms. However, in deliberative reasoning such as in VQA, attention is unconstrained at each step, and thus may serve as a statistical pooling mechanism rather than a semantic operation intended to select information relevant to inference. This is because at training time, attention is only guided by a very sparse signal (i.e. the answer label) at the end of the inference chain. This causes the cross-modality attention weights to deviate from the desired visual-language bindings. To r…
Talk2Sensors: 3D Visual Grounding in Autonomous Driving via Sensor-Adaptive Physical Cue Matching
Thrust of Artificial Intelligence, The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州）人工智能学域）；MMLab, CUHK（香港中文大学MMLab）；School of Transportation Science and Engineering, Harbin Institute of Technology（哈尔滨工业大学交通科学与工程学院）；School of Advanced Technology, Xi’an Jiaotong-Liverpool University（西交利物浦大学先进技术学院）；School of Electronics and Computer Science, University of Southampton（南安普顿大学电子与计算机科学学院）；School of Intelligent Manufacturing and Smart Transportation, Suzhou City University（苏州城市学院智能制造与智能交通学院）；Qingdao University of Science and Technology（青岛科技大学）；Institute for Math & AI, Wuhan University（武汉大学数学与人工智能研究院）；Institute of Big Data, Fudan University（复旦大学大数据研究院）
Carnegie Mellon University（卡内基梅隆大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Visual grounding is a task that aims to locate a target object according to a natural language expression. As a multi-modal task, feature interaction between textual and visual inputs is vital. However, previous solutions mainly handle each modality independently before fusing them together, which does not take full advantage of relevant textual information while extracting visual features. To better leverage the textual-visual relationship in visual grounding, we propose a Query-conditioned Convolution Module (QCM) that extracts query-aware visual features by incorporating query information i…
Hangzhou Dianzi University（杭州电子科技大学）；Zhejiang University（浙江大学）；University of Texas at San Antonio（德克萨斯大学圣安东尼奥分校）；University of Sydney（悉尼大学）
Visual grounding aims to localize an object in an image referred to by a textual query phrase. Various visual grounding approaches have been proposed, and the problem can be modularized into a general framework: proposal generation, multi-modal feature representation, and proposal ranking. Of these three modules, most existing approaches focus on the latter two, with the importance of proposal generation generally neglected. In this paper, we rethink the problem of what properties make a good proposal generator. We introduce the diversity and discrimination simultaneously when generating propo…
Charles University（查理大学）；Nara Institute of Science and Technology（奈良先端科学技术大学院大学）；RIKEN AIP（理化学研究所先进智能项目中心）
We present our work in progress exploring the possibilities of a shared embedding space between textual and visual modality. Leveraging the textual nature of object detection labels and the hypothetical expressiveness of extracted visual object representations, we propose an approach opposite to the current trend, grounding of the representations in the word embedding space of the captioning system instead of grounding words or sentences in their associated images. Based on the previous work, we apply additional grounding losses to the image captioning training objective aiming to force visual…
The University of Hong Kong（香港大学）；Deepwise AI Lab（深智云人工智能实验室）；Sun Yat-sen University（中山大学）
Grounding referring expressions in images aims to locate the object instance in an image described by a referring expression. It involves a joint understanding of natural language and image content, and is essential for a range of visual tasks related to human-computer interaction. As a language-to-vision matching task, the core of this problem is to not only extract all the necessary information (i.e., objects and the relationships among them) in both the image and referring expression, but also make full use of context information to align cross-modal semantic concepts in the extracted infor…
Focus When Necessary: Adaptive Routing and Collaborative Grounding for Training-Free Visual Grounding
East China University of Science and Technology（华东理工大学）；Tsinghua University（清华大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；University of Science and Technology of China（中国科学技术大学）
University of Padua（帕多瓦大学）；Bruno Kessler Foundation（布鲁诺·凯斯勒基金会）
Given a textual phrase and an image, the visual grounding problem is the task of locating the content of the image referenced by the sentence. It is a challenging task that has several real-world applications in human-computer interaction, image-text reference resolution, and video-text reference resolution. In the last years, several works have addressed this problem by proposing more and more large and complex models that try to capture visual-textual dependencies better than before. These models are typically constituted by two main components that focus on how to learn useful multi-modal f…
