Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);School of Information, Renmin University of China(中国人民大学信息学院);Baichuan Inc.(百川公司);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
Yifan Du, Kun Zhou, Yuqi Huo, Yifan Li, Wayne Xin Zhao, Haoyu Lu, Zijia Zhao, Bingning Wang, Weipeng Chen, Ji-Rong Wen
With the rapid development of video Multimodal Large Language Models (MLLMs), numerous benchmarks have been proposed to assess their video understanding capability. However, due to the lack of rich events in the videos, these datasets may suffer from the short-cut bias that the answers can be deduced from a few frames, without the need to watch the entire video. To address this issue, we introduce Event-Bench, an event-oriented long video understanding benchmark built on existing datasets and human annotations. Event-Bench includes six event-related tasks and 2,190 test instances to comprehensively evaluate video event understanding ability. Additionally, we propose Video Instruction Merging~(VIM), a cost-effective method that enhances video MLLMs using merged, event-intensive video instructions, addressing the scarcity of human-annotated, event-intensive data. Extensive experiments show that the best-performing model, GPT-4o, achieves an overall accuracy of 53.33, significantly outperforming the best open-source model by 41.42%. Leveraging an effective instruction synthesis method and an adaptive model architecture, VIM surpasses both state-of-the-art open-source models and GPT-4V on the Event-Bench. All code, data, and models are publicly available at https://github.com/RUCAIBox/Event-Bench.
Two-Stream Action Recognition-Oriented Video Super-Resolution
CAS Key Laboratory of Technology in Geo-Spatial Information Processing and Application System(中国科学院地理空间信息处理与应用系统技术重点实验室);University of Science and Technology of China(中国科学技术大学)
LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache and Retrieval
School of Computer Science, Shanghai Jiao Tong University(上海交通大学计算机科学学院);College of Intelligent Robotics and Advanced Manufacturing, Fudan University(复旦大学智能机器人与先进制造学院);Guizhou Provincial Laboratory of Big Data, College of Computer Science and Technology, Guizhou University(贵州大数据省实验室,贵州大学计算机科学与技术学院)
LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videos
Southern University of Science and Technology(南方科技大学);University of Birmingham(伯明翰大学);University of Electronic Science and Technology of China(电子科技大学);The University of Hong Kong(香港大学);University of Manchester(曼彻斯特大学)
Video-EM: Event-Centric Episodic Memory for Long-Form Video Understanding
City University of Hong Kong(香港城市大学);University of Science and Technology of China(中国科学技术大学);Tianjin University(天津大学);Nanjing University(南京大学);Zhejiang University(浙江大学);The Institute of Artificial Intelligence (TeleAI), China Telecom(中国电信人工智能研究所(TeleAI))
TimeScope: Towards Task-Oriented Temporal Grounding In Long Videos
Beijing Academy of Artificial Intelligence(北京人工智能研究院);School of AI, Shanghai Jiao Tong University(上海交通大学人工智能学院);University of Trento(特伦多大学);Singapore Management University(新加坡管理大学)
EVIT: Event-Oriented Instruction Tuning for Event Reasoning
School of Computer Science, Peking University(北京大学计算机学院);Key Laboratory of High Confidence Software Technologies (PKU), MOE, China(高可信软件技术重点实验室(北京大学));Advanced Institute of Big Data(大数据先进研究院)
TRACE: Temporal Grounding Video LLM via Causal Event Modeling
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳)理工学院);Tencent PCG(腾讯平台与内容事业群);Shenzhen Institute of Artificial Intelligence and Robotics for Society (AIRS)(深圳市人工智能与机器人研究院);Guangdong Provincial Key Laboratory of Future Networks of Intelligence(广东省未来智能网络重点实验室)
EventMemAgent: Hierarchical Event-Centric Memory for Online Video Understanding with Adaptive Tool Use
Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, School of Artificial Intelligence, Beihang University(北京未来区块链与隐私计算先进创新中心,人工智能学院,北京航空航天大学);Hangzhou International Innovation Institute, Beihang University(杭州国际创新研究院,北京航空航天大学);Paradigm Inc.(4Paradigm公司)
AI Oriented Large-Scale Video Management for Smart City: Technologies, Standards and Beyond
Institute of Digital Media, Peking University(北京大学数字媒体研究所);SECE of Shenzhen Graduate School, Peking University(北京大学深圳研究生院信息工程学院);Department of Computer Science, City University of Hong Kong(香港城市大学计算机科学系);Lenovo Research(联想研究院)
OVC-Net: Object-Oriented Video Captioning with Temporal Graph and Detail Enhancement
School of Artificial Intelligence Beijing Univerisity of Posts\ Telecommunications Beijing, China Email;Department of Electrical;School of Artificial Intelligence Beijing Univerisity of Posts
IntentVCNet: Bridging Spatio-Temporal Gaps for Intention-Oriented Controllable Video Captioning
University of Science and Technology of China(中国科学技术大学);Hefei Institutes of Physical Science,Chinese Academy of Sciences(合肥物理研究所,中国科学院);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(人工智能研究所,合肥综合性国家科学中心);State Key Lab. for Novel Software Technology, Nanjing University(新型软件技术国家重点实验室,南京大学);University of Chicago(芝加哥大学);National University of Defense Technology(国防科技大学);Harbin Institute of Technology(哈尔滨工业大学)
GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understanding
Huazhong University of Science and Technology(华中科技大学);University of Notre Dame(圣母大学);Imperial College London(伦敦帝国学院);Peking University(北京大学);Microsoft Research(微软研究院);Lehigh University(理海大学)