Recent progress in multimodal large language models has markedly enhanced the understanding of short videos (typically under one minute), and several evaluation datasets have emerged accordingly. However, these advancements fall short of meeting the demands of real-world applications such as embodied intelligence for long-term decision-making, in-depth movie reviews and discussions, and live sports commentary, all of which require comprehension of long videos spanning several hours. To address this gap, we introduce LVBench, a benchmark specifically designed for long video understanding. Our dataset comprises publicly sourced videos and encompasses a diverse set of tasks aimed at long video comprehension and information extraction. LVBench is designed to challenge multimodal models to demonstrate long-term memory and extended comprehension capabilities. Our extensive evaluations reveal that current multimodal models still underperform on these demanding long video understanding tasks. Through LVBench, we aim to spur the development of more advanced models capable of tackling the complexities of long video comprehension. Our data and code are publicly available at: https://lvbench.github.io.
LvBench: A Benchmark for Long-form Video Understanding with Versatile Multi-modal Question Answering
OpenGVLab, Shanghai AI Laboratory, China（OpenGVLab，上海人工智能实验室）；University of Science and Technology of China（中国科学技术大学）；Honor Device Co.,Ltd（荣耀设备有限公司）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究所，中国科学院）；Nanjing University（南京大学）
Symphony: A Cognitively-Inspired Multi-Agent System for Long-Video Understanding
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Kuaishou Technology（快手科技）；School of Future Technology, University of Chinese Academy of Sciences（中国科学院大学未来技术学院）
VideoTapestry: Query-Adaptive Memory Refinement for Multi-Agent Long-Video Understanding
University of Science and Technology of China（中国科学技术大学）；Jilin University（吉林大学）；Hangzhou Dianzi University（杭州电子科技大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）
LensWalk: Agentic Video Understanding by Planning How You See in Videos
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Peng Cheng Laboratory（鹏城实验室）；College of Computer Science and Electronic Engineering, Hunan University（湖南大学计算机科学与电子工程学院）；University of Chinese Academy of Sciences（中国科学院大学）
Native Active Perception as Reasoning for Omni-Modal Understanding
The Chinese University of Hong Kong（香港中文大学）；Shanghai Jiao Tong University（上海交通大学）；Nanyang Technological University（南洋理工大学）；Qwen Team, Alibaba Group（阿里巴巴集团Qwen团队）
Reasoning with Memory: A Temporal Granularity-Adaptive Framework for Training-Free Long Video Understanding
National University of Singapore（新加坡国立大学）；Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR)（高性能计算研究所，科学、技术与研究机构（A*STAR））；Nanyang Technological University（南洋理工大学）；University of Science and Technology of China（中国科学技术大学）；Jilin University（吉林大学）
University of Science and Technology of China（中国科学技术大学）；Microsoft Research Asia（微软亚洲研究院）；Microsoft Azure AI（微软Azure人工智能）
In this paper, we propose NUWA-XL, a novel Diffusion over Diffusion architecture for eXtremely Long video generation. Most current work generates long videos segment by segment sequentially, which normally leads to the gap between training on short videos and inferring long videos, and the sequential generation is inefficient. Instead, our approach adopts a ``coarse-to-fine'' process, in which the video can be generated in parallel at the same granularity. A global diffusion model is applied to generate the keyframes across the entire time range, and then local diffusion models recursively fil…
Memory-enhanced Retrieval Augmentation for Long Video Understanding
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学光华学院人工智能学院）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；University of Trento（特伦托大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Recurrent Neural Networks (RNNs) have been widely used in sequence analysis and modeling. However, when processing high-dimensional data, RNNs typically require very large model sizes, thereby bringing a series of deployment challenges. Although various prior works have been proposed to reduce the RNN model sizes, executing RNN models in resource-restricted environments is still a very challenging problem. In this paper, we propose to develop extremely compact RNN models with fully decomposed hierarchical Tucker (FDHT) structure. The HT decomposition does not only provide much higher storage c…
NARU: A Benchmark for NARrative Evolution and Cultural Nuance Understanding in Japanese Extreme Long Video
The University of Tokyo（东京大学）；Kyushu University（九州大学）；Macau University of Science and Technology（澳门科技大学）；Infinimind Japan Inc.（Infinimind日本公司）；University of Alberta（阿尔伯塔大学）
CoVStream: Edge-Cloud Collaboration for Understanding of Long Video Streams
The State Key Lab of Brain-Machine Intelligence, Zhejiang University（浙江大学脑机智能国家重点实验室）；vivo Mobile Communication Co., Ltd., Shenzhen, China（深圳 vivo 通信有限公司）
Deep convolutional neural networks (ConvNets) have been recently shown to attain state-of-the-art performance for action recognition on standard-resolution videos. However, less attention has been paid to recognition performance at extremely low resolutions (eLR) (e.g., 16 x 12 pixels). Reliable action recognition using eLR cameras would address privacy concerns in various application environments such as private homes, hospitals, nursing/rehabilitation facilities, etc. In this paper, we propose a semi-coupled filter-sharing network that leverages high resolution (HR) videos during training in…
Indiana University, Bloomington（印第安纳大学伯明顿分校）；University of Central Florida（中佛罗里达大学）；Midea Corporate Research Center（美的企业研究中心）
A major emerging challenge is how to protect people's privacy as cameras and computer vision are increasingly integrated into our daily lives, including in smart devices inside homes. A potential solution is to capture and record just the minimum amount of information needed to perform a task of interest. In this paper, we propose a fully-coupled two-stream spatiotemporal architecture for reliable human action recognition on extremely low resolution (e.g., 12x16 pixel) videos. We provide an efficient method to extract spatial and temporal features and to aggregate them into a robust feature re…
