CONE: An Efficient COarse-to-fiNE Alignment Framework for Long Video Temporal Grounding
City University of Hong Kong(香港城市大学);Microsoft Research Asia(微软亚洲研究院);National University of Singapore(新加坡国立大学);Beihang University(北京航空航天大学);Singapore Management University(新加坡管理大学);Sun Yat-sen University(中山大学)
Zhijian Hou, Wanjun Zhong, Lei Ji, Difei Gao, Kun Yan, Wing-Kwong Chan, Chong-Wah Ngo, Zheng Shou, Nan Duan
备注:
ACL 2023 Camera Ready. 14 pages, 7 figures, 4 tables
This paper tackles an emerging and challenging problem of long video temporal grounding~(VTG) that localizes video moments related to a natural language (NL) query. Compared with short videos, long videos are also highly demanded but less explored, which brings new challenges in higher inference computation cost and weaker multi-modal alignment. To address these challenges, we propose CONE, an efficient COarse-to-fiNE alignment framework. CONE is a plug-and-play framework on top of existing VTG models to handle long videos through a sliding window mechanism. Specifically, CONE (1) introduces a query-guided window selection strategy to speed up inference, and (2) proposes a coarse-to-fine mechanism via a novel incorporation of contrastive learning to enhance multi-modal alignment for long videos. Extensive experiments on two large-scale long VTG benchmarks consistently show both substantial performance gains (e.g., from 3.13% to 6.87% on MAD) and state-of-the-art results. Analyses also reveal higher efficiency as the query-guided window selection mechanism accelerates inference time by 2x on Ego4D-NLQ and 15x on MAD while keeping SOTA results. Codes have been released at https://github.com/houzhijian/CONE.
ChatVTG: Video Temporal Grounding via Chat with Video Dialogue Large Language Models
Institute of Information Science, Beijing Jiaotong University(北京交通大学信息科学研究所);JD Explore Academy(京东探索研究院);University of Science and Technology of China(中国科学技术大学);Horace Mann School(霍勒斯·曼学校)
VAGU & GtS: LLM-Based Benchmark and Framework for Joint Video Anomaly Grounding and Understanding
Beijing Jiaotong University(北京交通大学);State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences(多模态人工智能系统国家重点实验室,自动化研究所,中国科学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
MedScope: Incentivizing "Think with Videos" for Clinical Reasoning via Coarse-to-Fine Tool Calling
College of Health Science and Technology, Shanghai Jiao Tong University School of Medicine, Shanghai, China(上海交通大学医学院健康科学与技术学院);Fudan University, Shanghai, China(复旦大学);Shanghai Innovation Institute, Shanghai, China(上海创新研究院);Tsinghua University, Beijing, China(清华大学);Shanghai Artificial Intelligence Laboratory, Shanghai, China(上海人工智能实验室);The Chinese University of Hong Kong, Hong Kong, China(香港中文大学);Ruijin Hospital, Shanghai Jiaotong University, Shanghai, China(上海交通大学瑞金医院)
Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Models
University of California, Davis(加州大学戴维斯分校);Virginia Tech(弗吉尼亚理工大学);The Chinese University of Hong Kong(香港中文大学);NVIDIA;Adobe Research(Adobe研究);Fudan University(复旦大学);Meta AI
TimeScope: Towards Task-Oriented Temporal Grounding In Long Videos
Beijing Academy of Artificial Intelligence(北京人工智能研究院);School of AI, Shanghai Jiao Tong University(上海交通大学人工智能学院);University of Trento(特伦多大学);Singapore Management University(新加坡管理大学)
Generalist Scanner Meets Specialist Locator: A Synergistic Coarse-to-Fine Framework for Robust GUI Grounding
University of California, San Diego(加州大学圣地亚哥分校);ByteDance(字节跳动);University of California, Merced(加州大学默塞德分校);University of Southern California(南加州大学);University at Buffalo(布法罗大学);The University of Queensland(昆士兰大学)
Rethinking Weakly-supervised Video Temporal Grounding From a Game Perspective
Hubei Key Laboratory of Distributed System Security(湖北分布式系统安全重点实验室);Hubei Engineering Research Center on Big Data Security(大数据安全工程研究中心);School of Cyber Science and Engineering(网络安全科学与工程学院);Huazhong University of Science and Technology(华中科技大学);University of Central Florida(佛罗里达中央大学);Zhejiang Gongshang University(浙江工商大学);Guangzhou University(广州大学);The Chinese University of Hong Kong(香港中文大学);Peking University(北京大学)
An Efficient COarse-to-fiNE Alignment Framework @ Ego4D Natural Language Queries Challenge 2022
City University of Hong Kong(香港城市大学);Microsoft Research Asia(微软亚洲研究院);National University of Singapore(新加坡国立大学);Beihang University(北京航空航天大学);Singapore Management University(新加坡管理大学);Sun Yat-sen University(中山大学)
ReTool-Video: Recursive Tool-Using Video Agents with Meta-Augmented Tool Grounding
Chongqing University(重庆大学);Tianjin University(天津大学);MAIS, Institute of Automation, Chinese Academy of Sciences(自动化研究所,中国科学院MAIS);Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR), Singapore(新加坡科技研究局高性能计算研究所);Chongqing National Data AI Research Institute, AI Research Lab(重庆国家数据AI研究院,AI研究实验室)
Decoupling Semantics and Logic: A Training-Free Coarse-to-Fine Pipeline for Video Retrieval-Augmented Generation
School of Computer Science & Tech, Huazhong University of Science and Technology(华中科技大学计算机科学与技术学院);School of AI and Automation, Huazhong University of Science and Technology(华中科技大学人工智能与自动化学院)
Multi-task Visual Grounding with Coarse-to-Fine Consistency Constraints
School of Automation, Southeast University(东南大学自动化学院);Youtu Lab, Tencent(腾讯优图实验室);College of Information Science and Electronic Engineering, Zhejiang University(浙江大学信息科学与电子工程学院)
EARL: Towards a Unified Analysis-Guided Reinforcement Learning Framework for Egocentric Interaction Reasoning and Pixel Grounding
Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong SAR(香港理工大学电子与电气工程系);Division of Emerging Interdisciplinary Areas (EMIA), The Hong Kong University of Science and Technology, Hong Kong SAR(香港理工大学新兴跨学科领域研究中心)
HawkEye: Training Video-Text LLMs for Grounding Text in Videos
Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);Huawei Noah’s Ark Lab(华为诺亚方舟实验室);Beijing Institute for General Artificial Intelligence(北京通用人工智能研究院)
CoF: Coarse to Fine-Grained Image Understanding for Multi-modal Large Language Models
School of Automation, Northwestern Polytechnical University(西北工业大学自动化学院);School of Cybersecurity, Northwestern Polytechnical University(西北工业大学网络空间安全学院);Alibaba Group(阿里巴巴集团);Southeast University(东南大学);Advanced Ocean Institute of Southeast University(东南大学先进海洋研究院);Zhejiang University of Technology(浙江工业大学);Binjiang Institute of Artificial Intelligence(滨江人工智能研究院)
Exploiting Feature Diversity for Make-up Temporal Video Grounding
Tencent Youtu Lab(腾讯优图实验室)
本技术报告介绍了MTVG的第三个获奖解决方案,这是在ACM MM 2022的第四届情景中人(PIC)挑战赛中引入的一项新任务。MTVG的目的是在一个未修剪的视频的基础上的文本描述的时间边界的步骤。这项任务的最大挑战是化妆步骤的细粒度视频-文本语义。然而,目前的方法主要是使用基于动作的预训练模型提取视频特征。由于动作比化妆步骤更粗粒度,因此基于动作的特征不足以提供细粒度的线索。为了解决这个问题,我们建议通过利用功能diffusion实现细粒度表示。具体来说,我们提出了一系列的方法,从特征提取,网络优化,模型集成。因此,我们在MTVG比赛中获得了第三名。
Dynamic Prompting of Frozen Text-to-Image Diffusion Models for Panoptic Narrative Grounding
School of Artificial Intelligence, Beihang University(北京航空航天大学人工智能学院);School of Computer Science and Information Engineering, Hefei University of Technology(合肥工业大学计算机与信息工程学院);School of Software, Beihang University(北京航空航天大学软件学院);Meituan(美团);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程学院)