We introduce ED-VTG, a method for fine-grained video temporal grounding utilizing multi-modal large language models. Our approach harnesses the capabilities of multimodal LLMs to jointly process text and video, in order to effectively localize natural language queries in videos through a two-stage process. Rather than being directly grounded, language queries are initially transformed into enriched sentences that incorporate missing details and cues to aid in grounding. In the second stage, these enriched queries are grounded, using a lightweight decoder, which specializes at predicting accurate boundaries conditioned on contextualized representations of the enriched queries. To mitigate noise and reduce the impact of hallucinations, our model is trained with a multiple-instance-learning objective that dynamically selects the optimal version of the query for each training sample. We demonstrate state-of-the-art results across various benchmarks in temporal video grounding and paragraph grounding settings. Experiments reveal that our method significantly outperforms all previously proposed LLM-based temporal grounding approaches and is either superior or comparable to specialized models, while maintaining a clear advantage against them in zero-shot evaluation scenarios.
Efficient Temporal Extrapolation of Multimodal Large Language Models with Temporal Grounding Bridge
Beijing Institute for General Artificial Intelligence (BIGAI)（北京通用人工智能研究院）；State Key Laboratory of General Artificial Intelligence（通用人工智能国家重点实验室）；Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机技术研究院）
A Survey on Video Temporal Grounding with Multimodal Large Language Model
School of Computer Science and Technology, Harbin Institute of Technology（哈尔滨工业大学计算机科学与技术学院）；School of Computer Science and Technology, Shandong Jianzhu University（山东建筑大学计算机科学与技术学院）；School of Intelligence Science and Technology, Peking University（北京大学智能科学与技术学院）；Department of Computing, The Hong Kong Polytechnic University（香港理工大学计算机系）
Vector Institute for AI（向量人工智能研究所）；Meta AI；University of British Columbia（不列颠哥伦比亚大学）
Video understanding tasks take many forms, from action detection to visual query localization and spatio-temporal grounding of sentences. These tasks differ in the type of inputs (only video, or video-query pair where query is an image region or sentence) and outputs (temporal segments or spatio-temporal tubes). However, at their core they require the same fundamental understanding of the video, i.e., the actors and objects in it, their actions and interactions. So far these tasks have been tackled in isolation with individual, highly specialized architectures, which do not exploit the interpl…
TimeLens2: Generalist Video Temporal Grounding with Multimodal LLMs
Nanjing University（南京大学）；Shanghai AI Laboratory（上海人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）；Zhejiang University（浙江大学）；University of Science and Technology of China（中国科学技术大学）；Fudan University（复旦大学）
Visual dialogue is a challenging task since it needs to answer a series of coherent questions on the basis of understanding the visual environment. Previous studies focus on the implicit exploration of multimodal co-reference by implicitly attending to spatial image features or object-level image features but neglect the importance of locating the objects explicitly in the visual content, which is associated with entities in the textual content. Therefore, in this paper we propose a {\bf M}ultimodal {\bf I}ncremental {\bf T}ransformer with {\bf V}isual {\bf G}rounding, named MITVG, which consi…
HiVG: Hierarchical Multimodal Fine-grained Modulation for Visual Grounding
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Pengcheng Laboratory（鹏城实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））
MIMO: A medical vision language model with visual referring multimodal input and pixel grounding multimodal output
School of Software & Microelectronics, Peking University（软件与微电子学院，北京大学）；School of Computer Science, Peking University（计算机学院，北京大学）；National Engineering Research Center for Software Engineering, Peking University（软件工程国家工程研究中心，北京大学）；Peking University Sixth Hospital（北京大学第六医院）；Augusta University（奥古斯塔大学）；Peking University First Hospital（北京大学第一医院）
Mixture-of-Thought-Tokens: Unifying Perception and Reasoning for Free-form Multimodal Grounding
Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（西安交通大学人工智能与机器人研究所）；Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd（星辰通用人工智能实验室，中国电信人工智能技术（北京）有限公司）；University of Science and Technology Beijing（北京科技大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Shanghai Jiao Tong University（上海交通大学）
VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in Videos
Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）；Tianjin University（天津大学）；Carnegie Mellon University（卡内基梅隆大学）；Linköping University（林雪平大学）；Australian National University（澳大利亚国立大学）
Dreaming in Flow: Bidirectional Self-Refinement for Unified Multimodal Models
Shanghai Jiao Tong University（上海交通大学）；Institute of Artificial Intelligence, China Telecom (TeleAI)（中国电信人工智能研究院（TeleAI））；University of Science and Technology of China（中国科学技术大学）
College of Intelligence and Computing, Tianjin University（天津大学智能与计算学部）
Visual context provides grounding information for multimodal machine translation (MMT). However, previous MMT models and probing studies on visual features suggest that visual information is less explored in MMT as it is often redundant to textual information. In this paper, we propose an object-level visual context modeling framework (OVC) to efficiently capture and explore visual information for multimodal machine translation. With detected objects, the proposed OVC encourages MMT to ground translation on desirable visual objects by masking irrelevant objects in the visual modality. We equip…
We propose a new spatial memory module and a spatial reasoner for the Visual Grounding (VG) task. The goal of this task is to find a certain object in an image based on a given textual query. Our work focuses on integrating the regions of a Region Proposal Network (RPN) into a new multi-step reasoning model which we have named a Multimodal Spatial Region Reasoner (MSRR). The introduced model uses the object regions from an RPN as initialization of a 2D spatial memory and then implements a multi-step reasoning process scoring each region according to the query, hence why we call it a multimodal…
IoU-PD: IoU-Aware Privileged Distillation for Visual Grounding with Multimodal Large Language Models
University of Chinese Academy of Sciences（中国科学院大学）；State Key Laboratory of Communication Content Cognition（通信内容认知技术国家重点实验室）；Peng Cheng Laboratory（鹏城实验室）
Trifuse: Enhancing Attention-Based GUI Grounding via Multimodal Fusion
College of Computer Science and Technology, National University of Defense Technology（计算机科学与技术学院，国防科技大学）；Intelligent Game and Decision Lab, Academy of Military Sciences（智能游戏与决策实验室，军事科学院）
East China Normal University（华东师范大学）；Alibaba Group（阿里巴巴集团）；NLPR, CASIA（中国科学院自动化研究所模式识别国家重点实验室）；Fudan University（复旦大学）
Visual grounding focuses on establishing fine-grained alignment between vision and natural language, which has essential applications in multimodal reasoning systems. Existing methods use pre-trained query-agnostic visual backbones to extract visual feature maps independently without considering the query information. We argue that the visual features extracted from the visual backbones and the features really needed for multimodal reasoning are inconsistent. One reason is that there are differences between pre-training tasks and visual grounding. Moreover, since the backbones are query-agnost…
Beijing Institute of Technology（北京理工大学）；JD AI Research（京东人工智能研究院）；Australian National University（澳大利亚国立大学）
Multimodal named entity recognition (MNER) is a critical step in information extraction, which aims to detect entity spans and classify them to corresponding entity types given a sentence-image pair. Existing methods either (1) obtain named entities with coarse-grained visual clues from attention mechanisms, or (2) first detect fine-grained visual regions with toolkits and then recognize named entities. However, they suffer from improper alignment between entity types and visual regions or error propagation in the two-stage manner, which finally imports irrelevant visual information into texts…
Grounding Everything in Tokens for Multimodal Large Language Models
MoE Key Lab of Artificial Intelligence（人工智能混合专家模型关键实验室）；AI Institute, Shanghai Jiao Tong University（上海交通大学人工智能研究院）；Central Research Institute, Huawei（华为中央研究院）
UC Berkeley EECS（加州大学伯克利分校电子工程与计算机科学系）；Sony Corp.（索尼公司）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）
Modeling textual or visual information with vector representations trained from large language or visual datasets has been successfully explored in recent years. However, tasks such as visual question answering require combining these vector representations with each other. Approaches to multimodal pooling include element-wise product or sum, as well as concatenation of the visual and textual representations. We hypothesize that these methods are not as expressive as an outer product of the visual and textual vectors. As the outer product is typically infeasible due to its high dimensionality,…
RGBX-R1: Visual Modality Chain-of-Thought Guided Reinforcement Learning for Multimodal Grounding
School of Artificial Intelligence, Tianjin University（天津大学人工智能学院）；Low-Altitude Intelligence Lab, Xiong’an National Innovation Center（雄安国家创新中心低空智能实验室）；Xiong’an Guochuang Lantian Technology Co., Ltd.（雄安国创莲田科技有限公司）；College of Electronic Science and Technology, National University of Defense Technology（国防科技大学电子科学学院）
