In this work, we introduce a novel task - Humancentric Spatio-Temporal Video Grounding (HC-STVG). Unlike the existing referring expression tasks in images or videos, by focusing on humans, HC-STVG aims to localize a spatiotemporal tube of the target person from an untrimmed video based on a given textural description. This task is useful, especially for healthcare and security-related applications, where the surveillance videos can be extremely long but only a specific person during a specific period of time is concerned. HC-STVG is a video grounding task that requires both spatial (where) and temporal (when) localization. Unfortunately, the existing grounding methods cannot handle this task well. We tackle this task by proposing an effective baseline method named Spatio-Temporal Grounding with Visual Transformers (STGVT), which utilizes Visual Transformers to extract cross-modal representations for video-sentence matching and temporal localization. To facilitate this task, we also contribute an HC-STVG dataset consisting of 5,660 video-sentence pairs on complex multi-person scenes. Specifically, each video lasts for 20 seconds, pairing with a natural query sentence with an average of 17.25 words. Extensive experiments are conducted on this dataset, demonstrating the newly-proposed method outperforms the existing baseline methods.
Georgia Institute of Technology（佐治亚理工学院）；MIT（麻省理工学院）；Stanford University（斯坦福大学）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
When answering questions about an image, it not only needs knowing what -- understanding the fine-grained contents (e.g., objects, relationships) in the image, but also telling why -- reasoning over grounding visual cues to derive the answer for a question. Over the last few years, we have seen significant progress on visual question answering. Though impressive as the accuracy grows, it still lags behind to get knowing whether these models are undertaking grounding visual reasoning or just leveraging spurious correlations in the training data. Recently, a number of works have attempted to ans…
EPIC-Bench: A Perception-Centric Benchmark for Fine-Grained Embodied Visual Grounding in Vision-Language Models
X-Humanoid；Fudan University（复旦大学）；University of Science and Technology of China（中国科学技术大学）；University of Manchester（曼彻斯特大学）；Monash University（墨尔本大学）；Celonis AI；University of New South Wales（新南威尔士大学）
RSTGCN: Railway-centric Spatio-Temporal Graph Convolutional Network for Train Delay Prediction
Department of Computer Science and Engineering, Indian Institute of Technology Kharagpur（计算机科学与工程系，印度理工学院Kharagpur）；Machine Intelligence Unit, Indian Statistical Institute（机器智能单位，印度统计研究所）
Motion-Grounded Video Reasoning: Understanding and Perceiving Motion at Pixel Level
University of Central Florida（中佛罗里达大学）；University of Western Australia（西澳大学）；Amazon Web Services（亚马逊网络服务）；The University of Texas at Dallas（德克萨斯大学达拉斯分校）
Natural Human-Robot Interaction (HRI) is one of the key components for service robots to be able to work in human-centric environments. In such dynamic environments, the robot needs to understand the intention of the user to accomplish a task successfully. Towards addressing this point, we propose a software architecture that segments a target object from a crowded scene, indicated verbally by a human user. At the core of our system, we employ a multi-modal deep neural network for visual grounding. Unlike most grounding methods that tackle the challenge using pre-trained object detectors via a…
We present a new method, PARsing And visual GrOuNding (ParaGon), for grounding natural language in object placement tasks. Natural language generally describes objects and spatial relations with compositionality and ambiguity, two major obstacles to effective language grounding. For compositionality, ParaGon parses a language instruction into an object-centric graph representation to ground objects individually. For ambiguity, ParaGon uses a novel particle-based graph neural network to reason about object placements with uncertainty. Essentially, ParaGon integrates a parsing algorithm into a p…
RGBX-R1: Visual Modality Chain-of-Thought Guided Reinforcement Learning for Multimodal Grounding
School of Artificial Intelligence, Tianjin University（天津大学人工智能学院）；Low-Altitude Intelligence Lab, Xiong’an National Innovation Center（雄安国家创新中心低空智能实验室）；Xiong’an Guochuang Lantian Technology Co., Ltd.（雄安国创莲田科技有限公司）；College of Electronic Science and Technology, National University of Defense Technology（国防科技大学电子科学学院）
The University of Hong Kong（香港大学）；MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；Stanford University（斯坦福大学）；MIT BCS, CBMM, CSAIL（麻省理工学院脑与认知科学系、大脑、心智与机器中心、计算机科学与人工智能实验室）；MIT-IBM Watson AI Lab（MIT-IBM沃森人工智能实验室）
We study the problem of dynamic visual reasoning on raw videos. This is a challenging problem; currently, state-of-the-art models often require dense supervision on physical object properties and events from simulation, which are impractical to obtain in real life. In this paper, we present the Dynamic Concept Learner (DCL), a unified framework that grounds physical objects and events from video and language. DCL first adopts a trajectory extractor to track each object over time and to represent it as a latent, object-centric feature vector. Building upon this object-centric representation, DC…
GeoSearcher: Anchor-Guided Progressive Reasoning for Remote Sensing Visual Grounding with Process Supervision
Key Laboratory of Target Cognition and Application Technology (TCAT), Chinese Academy of Sciences（中国科学院目标认知与应用技术重点实验室）；School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences（中国科学院大学电子电气与通信工程学院）
From Region Arrival to Instance-Level Grounding in Vision-and-Language Navigation
Australian Institute for Machine Learning, Adelaide University（澳大利亚机器学习研究所，阿德莱德大学）；Ruyi Dynamics（如意动力）；Fudan University（复旦大学）；Swiss Federal Institute of Technology Lausanne (EPFL)（瑞士洛桑联邦理工学院）
GeoWeaver: Grounding Visual Tokens with Geometric Evidence before Scene Reasoning
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Pengcheng Laboratory（鹏城实验室）；Pazhou Lab (Huangpu)（琶洲实验室（黄埔））；Hainan University（海南大学）；University of California at Merced（加州大学默塞德分校）
College of Computer Science, Zhejiang University（浙江大学计算机科学与技术学院）；Huawei Cloud & AI（华为云与人工智能）
Spatio-temporal video grounding aims to retrieve the spatio-temporal tube of a queried object according to the given sentence. Currently, most existing grounding methods are restricted to well-aligned segment-sentence pairs. In this paper, we explore spatio-temporal video grounding on unaligned data and multi-form sentences. This challenging task requires to capture critical object relations to identify the queried target. However, existing approaches cannot distinguish notable objects and remain in ineffective relation modeling between unnecessary objects. Thus, we propose a novel object-awar…
University of Washington（华盛顿大学）；Facebook AI Research（Facebook 人工智能研究院）；Princeton University（普林斯顿大学）
Existing work in language grounding typically study single environments. How do we build unified models that apply across multiple environments? We propose the multi-environment Symbolic Interactive Language Grounding benchmark (SILG), which unifies a collection of diverse grounded language learning environments under a common interface. SILG consists of grid-world environments that require generalization to new dynamics, entities, and partially observed worlds (RTFM, Messenger, NetHack), as well as symbolic counterparts of visual worlds that require interpreting rich natural language with res…
SNOW: Spatio-Temporal Scene Understanding with World Knowledge for Open-World Embodied Reasoning
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Esslingen University of Applied Sciences（埃斯林根应用科学大学）；Dr. Ing. h.c. F. Porsche AG（德意志联邦汽车工业联合会）；University of Michigan（密歇根大学）；Voxel51 Inc.（Voxel51公司）
VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Grounding
The Chinese University of Hong Kong（香港中文大学）；Shanghai AI Laboratory（上海人工智能实验室）；Zhejiang University（浙江大学）；Centre for Perceptual and Interactive Intelligence（感知与交互智能中心）
Video-GroundingDINO: Towards Open-Vocabulary Spatio-Temporal Video Grounding
Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）；Australian National University（澳大利亚国立大学）；University of California, Merced（加州大学默塞德分校）；Google Research（谷歌研究院）；Linköping University（林雪平大学）
