Joint video-language learning has received increasing attention in recent years. However, existing works mainly focus on single or multiple trimmed video clips (events), which makes human-annotated event boundaries necessary during inference. To break away from the ties, we propose a grounded vision-language learning framework for untrimmed videos, which automatically detects informative events and effectively excavates the alignments between multi-sentence descriptions and corresponding event segments. Instead of coarse-level video-language alignments, we present two dual pretext tasks to encourage fine-grained segment-level alignments, i.e., text-to-event grounding (TEG) and event-to-text generation (ETG). TEG learns to adaptively ground the possible event proposals given a set of sentences by estimating the cross-modal distance in a joint semantic space. Meanwhile, ETG aims to reconstruct (generate) the matched texts given event proposals, encouraging the event representation to retain meaningful semantic information. To encourage accurate label assignment between the event set and the text set, we propose a novel semantic-aware cost to mitigate the sub-optimal matching results caused by ambiguous boundary annotations. Our framework is easily extensible to tasks covering visually-grounded language understanding and generation. We achieve state-of-the-art dense video captioning performance on ActivityNet Captions, YouCook2 and YouMakeup, and competitive performance on several other language generation and understanding tasks. Our method also achieved 1st place in both the MTVG and MDVC tasks of the PIC 4th Challenge. Our code is publicly available at https://github.com/zjr2000/GVL.
LongEgoRefer: A Benchmark for Long-Form Egocentric Video Referring Expression Comprehension
Woven by Toyota, Japan（丰田公司，日本）；The University of Tokyo, Japan（东京大学，日本）；National Institute of Informatics, Japan（日本信息机构国家研究所）；Institute of Science Tokyo, Japan（东京科学研究所，日本）；NII LLMC, Japan（日本信息机构LLMC）；Kyoto University, Japan（京都大学，日本）
GeoChat: Grounded Large Vision-Language Model for Remote Sensing
Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）；Birla Institute of Technology & Science, Hyderabad（海得拉巴贝拉理工学院）；Australian National University（澳大利亚国立大学）；Linköping University（林雪平大学）
NAVER AI Lab（NAVER人工智能实验室）；KAIST AI（韩国科学技术院人工智能学院）；SNU AIIS（首尔国立大学人工智能研究所）
Efficient video-language modeling should consider the computational cost because of a large, sometimes intractable, number of video frames. Parametric approaches such as the attention mechanism may not be ideal since its computational cost quadratically increases as the video length increases. Rather, previous studies have relied on offline feature extraction or frame sampling to represent the video efficiently, focusing on cross-modal modeling in short video clips. In this paper, we propose a semi-parametric video-grounded text generation model, SeViT, a novel perspective on scalable video-la…
Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Models
University of California, Davis（加州大学戴维斯分校）；Virginia Tech（弗吉尼亚理工大学）；The Chinese University of Hong Kong（香港中文大学）；NVIDIA；Adobe Research（Adobe研究）；Fudan University（复旦大学）；Meta AI
Shanghai Jiao Tong University（上海交通大学）；Beijing Institute of Technology（北京理工大学）；Johns Hopkins University（约翰霍普金斯大学）；University of California Merced（加州大学默塞德分校）
Columbia University（哥伦比亚大学）；Virginia Tech（弗吉尼亚理工大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Given a long untrimmed video and natural language queries, video grounding (VG) aims to temporally localize the semantically-aligned video segments. Almost all existing VG work holds two simple but unrealistic assumptions: 1) All query sentences can be grounded in the corresponding video. 2) All query sentences for the same video are always at the same semantic scale. Unfortunately, both assumptions make today's VG models fail to work in practice. For example, in real-world multimodal assets (eg, news articles), most of the sentences in the article can not be grounded in their affiliated video…
Polaffini: A feature-based approach for robust affine and polyaffine image registration
Hawkes Institute & Department of Computer Science, University College London, London, UK（霍克斯研究所及大学学院伦敦计算机科学系，伦敦，英国）；Institut Pasteur, Université Paris Cité, Unité de Neuroanatomie Appliquée et Théorique（巴斯德研究所，巴黎城市大学，应用与理论神经解剖学单元）；AINOSTICS ltd., Manchester, UK（AINOSTICS有限公司，曼彻斯特，英国）
OralGPT-Omni: A Versatile Dental Multimodal Large Language Model
Faculty of Dentistry, The University of Hong Kong（香港大学牙科学院）；College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机科学与软件工程学院）；The Hong Kong University of Science and Technology (GZ)（香港科学与技术大学）；School of Biomedical Engineering, Southern Medical University（南方医科大学生物医学工程学院）；Singapore University of Technology and Design（新加坡科技与设计大学）；University of Auckland（奥克兰大学）；University of Science and Technology of China（中国科学技术大学）；School of Computer Science, Peking University（北京大学计算机学院）；College of Artificial Intelligence, Shenzhen University（深圳大学人工智能学院）
