Temporal grounding aims to localize a video moment which is semantically aligned with a given natural language query. Existing methods typically apply a detection or regression pipeline on the fused representation with the research focus on designing complicated prediction heads or fusion strategies. Instead, from a perspective on temporal grounding as a metric-learning problem, we present a Mutual Matching Network (MMN), to directly model the similarity between language queries and video moments in a joint embedding space. This new metric-learning framework enables fully exploiting negative samples from two new aspects: constructing negative cross-modal pairs in a mutual matching scheme and mining negative pairs across different videos. These new negative samples could enhance the joint representation learning of two modalities via cross-modal mutual matching to maximize their mutual information. Experiments show that our MMN achieves highly competitive performance compared with the state-of-the-art methods on four video grounding benchmarks. Based on MMN, we present a winner solution for the HC-STVG challenge of the 3rd PIC workshop. This suggests that metric learning is still a promising method for temporal grounding via capturing the essential cross-modal correlation in a joint embedding space. Code is available at https://github.com/MCG-NJU/MMN.
Query-based video grounding is an important yet challenging task in video understanding, which aims to localize the target segment in an untrimmed video according to a sentence query. Most previous works achieve significant progress by addressing this task in a fully-supervised manner with segment-level labels, which require high labeling cost. Although some recent efforts develop weakly-supervised methods that only need the video-level knowledge, they generally match multiple pre-defined segment proposals with query and select the best one, which lacks fine-grained frame-level details for dis…
Temporal grounding aims to retrieve moments of the described event within an untrimmed video by a language query. Typically, existing methods assume annotations are precise and unique, yet one query may describe multiple moments in many cases. Hence, simply taking it as a one-vs-one mapping task and striving to match single-label annotations will inevitably introduce false negatives during optimization. In this study, we reformulate this task as a one-vs-many optimization problem under the condition of single positive labels. The unlabeled moments are considered unobserved rather than negative…
Grounding is All You Need? Dual Temporal Grounding for Video Dialog
National University of Singapore（新加坡国立大学）；University of Science and Technology of China（中国科学技术大学）；Hefei University of Technology（合肥工业大学）；Singapore Management University（新加坡管理大学）
National University of Singapore（新加坡国立大学）；University of Science and Technology of China（中国科学技术大学）
Video Question Answering (VideoQA) is the task of answering questions about a video. At its core is understanding the alignments between visual scenes in video and linguistic semantics in question to yield the answer. In leading VideoQA models, the typical learning objective, empirical risk minimization (ERM), latches on superficial correlations between video-question pairs and answers as the alignments. However, ERM can be problematic, because it tends to over-exploit the spurious correlations between question-irrelevant scenes and answers, instead of inspecting the causal effect of question-…
Multi-Scale Contrastive Learning for Video Temporal Grounding
Institute of Data Science (IDS), National University of Singapore（数据科学研究所（IDS），新加坡国立大学）；Tongji University（同济大学）；Nanyang Technological University (NTU)（南洋理工大学）
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Amazon Alexa AI（亚马逊Alexa人工智能部门）
To solve video-and-language grounding tasks, the key is for the network to understand the connection between the two modalities. For a pair of video and language description, their semantic relation is reflected by their encodings' similarity. A good multi-modality encoder should be able to well capture both inputs' semantics and encode them in the shared feature space where embedding distance gets properly translated into their semantic similarity. In this work, we focused on this semantic connection between video and language, and developed a multi-level alignment training scheme to directly…
ID-VTG: Image-Disambiguated Video Temporal Grounding
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；State Key Laboratory of General Artificial Intelligence, Peking University（北京大学通用人工智能国家重点实验室）
Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs
The University of Texas MD Anderson Cancer Center（德克萨斯大学MD安德森癌症中心）；Eisai Inc.（卫材株式会社）；IISc, Bangalore（印度科学研究所班加罗尔分校）；Cohere Labs Community（Cohere实验室社区）
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；NVIDIA（英伟达）；Bar Ilan University（巴伊兰大学）
Phrase grounding, the problem of associating image regions to caption words, is a crucial component of vision-language tasks. We show that phrase grounding can be learned by optimizing word-region attention to maximize a lower bound on mutual information between images and caption words. Given pairs of images and captions, we maximize compatibility of the attention-weighted regions and the words in the corresponding caption, compared to non-corresponding pairs of images and captions. A key idea is to construct effective negative captions for learning through language model guided word substitu…
Concept-Guided Noisy Negative Suppression for Zero-Shot Classification and Grounding of Chest X-Ray Findings
The Center for Smart Health, School of Nursing, the Hong Kong Polytechnic University, Hong Kong, China（香港理工大学智能健康中心，护理学院，中国香港）；Research Institute for Smart Ageing, the Hong Kong Polytechnic University, Hong Kong, China（香港理工大学智能老龄化研究 institute，中国香港）；School of Biomedical Engineering, Tsinghua Medicine, Tsinghua University, Beijing, China（清华大学生物医学工程学院，清华大学，北京，中国）；Queen Mary Hospital, LKS Faculty of Medicine, The University of Hong Kong, Hong Kong, China（香港大学李嘉诚医学院Queen Mary医院，中国香港）
Temporal sentence grounding (TSG) is an important yet challenging task in multimedia information retrieval. Although previous TSG methods have achieved decent performance, they tend to capture the selection biases of frequently appeared video-query pairs in the dataset rather than present robust multimodal reasoning abilities, especially for the rarely appeared pairs. In this paper, we study the above issue of selection biases and accordingly propose a Debiasing-TSG (D-TSG) model to filter and remove the negative biases in both vision and language modalities for enhancing the model generalizat…
Humans excel at learning from expert demonstrations and solving their own problems. To equip intelligent robots and assistants, such as AR glasses, with this ability, it is essential to ground human hand interactions (i.e., affordances) from demonstration videos and apply them to a target image like a user's AR glass view. The video-to-image affordance grounding task is challenging due to (1) the need to predict fine-grained affordances, and (2) the limited training data, which inadequately covers video-image discrepancies and negatively impacts grounding. To tackle them, we propose Affordance…
Weakly Supervised Temporal Sentence Grounding via Positive Sample Mining
University of Science and Technology of China（中国科学技术大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Beihang University（北京航空航天大学）；State Key Laboratory for Novel Software Technology（国家软件创新科技重点实验室）；Nanjing University（南京大学）；Shenzhen Key Laboratory of Computer Vision and Pattern Recognition（深圳计算机视觉与模式识别重点实验室）；Shenzhen Institute of Advanced Technology（深圳先进技术研究院）；Chinese Academy of Sciences（中国科学院）
National Tsing Hua University（台湾清华大学）；Microsoft Research（微软研究院）
Narrated 360° videos are typically provided in many touring scenarios to mimic real-world experience. However, previous work has shown that smart assistance (i.e., providing visual guidance) can significantly help users to follow the Normal Field of View (NFoV) corresponding to the narrative. In this project, we aim at automatically grounding the NFoVs of a 360° video given subtitles of the narrative (referred to as "NFoV-grounding"). We propose a novel Visual Grounding Model (VGM) to implicitly and efficiently predict the NFoVs given the video content and subtitles. Specifically, at each fram…
HawkEye: Training Video-Text LLMs for Grounding Text in Videos
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；Beijing Institute for General Artificial Intelligence（北京通用人工智能研究院）
Concentrate After Imagination: Text-Conditioned Evidence Grounding for Partially Relevant Video Retrieval
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；University of Electronic Science and Technology of China（电子科技大学）；The Hong Kong University of Science and Technology（香港科技大学）
