Long-form video understanding represents a significant challenge within computer vision, demanding a model capable of reasoning over long multi-modal sequences. Motivated by the human cognitive process for long-form video understanding, we emphasize interactive reasoning and planning over the ability to process lengthy visual inputs. We introduce a novel agent-based system, VideoAgent, that employs a large language model as a central agent to iteratively identify and compile crucial information to answer a question, with vision-language foundation models serving as tools to translate and retrieve visual information. Evaluated on the challenging EgoSchema and NExT-QA benchmarks, VideoAgent achieves 54.1% and 71.3% zero-shot accuracy with only 8.4 and 8.2 frames used on average. These results demonstrate superior effectiveness and efficiency of our method over the current state-of-the-art methods, highlighting the potential of agent-based approaches in advancing long-form video understanding.
Embodied VideoAgent: Persistent Memory from Egocentric Videos and Embodied Sensors Enables Dynamic Scene Understanding
State Key Laboratory of General Artificial Intelligence, BIGAI(通用人工智能国家重点实验室);University of Science and Technology of China(中国科学技术大学);Tsinghua University(清华大学);Peking University(北京大学)
Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining
National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University(国家多媒体信息处理重点实验室,计算机科学学院,北京大学);The University of Hong Kong(香港大学);Renmin University of China(中国人民大学)
PreMind: Multi-Agent Video Understanding for Advanced Indexing of Presentation-style Videos
Bosch Research North America(博世北美研究院);Bosch Center for Artificial Intelligence (BCAI)(博世人工智能中心);Department of Computer Science and Engineering, Texas A&M University(德克萨斯农工大学计算机科学与工程学院)
LensWalk: Agentic Video Understanding by Planning How You See in Videos
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);Peng Cheng Laboratory(鹏城实验室);College of Computer Science and Electronic Engineering, Hunan University(湖南大学计算机科学与电子工程学院);University of Chinese Academy of Sciences(中国科学院大学)
VideoGAIA: A Benchmark for General AI Assistants on Agentic Video Understanding
The Chinese University of Hong Kong(香港中文大学);Ant Group(蚂蚁集团);Tsinghua University(清华大学);Tongji University(同济大学);The Hong Kong University of Science and Technology(香港科技大学)
VideoTemp-o3: Harmonizing Temporal Grounding and Video Understanding in Agentic Thinking-with-Videos
Shandong University(山东大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Beihang University(北航);Southern University of Science and Technology(南方科技大学)
VideoTapestry: Query-Adaptive Memory Refinement for Multi-Agent Long-Video Understanding
University of Science and Technology of China(中国科学技术大学);Jilin University(吉林大学);Hangzhou Dianzi University(杭州电子科技大学);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院)
MUPA: Towards Multi-Path Agentic Reasoning for Grounded Video Question Answering
Lanzhou University(兰州大学);National University of Singapore(新加坡国立大学);Sun Yat-sen University(中山大学);Jinan University(暨南大学);Nanyang Technological University(南洋理工大学);Peking University(北京大学);University of Science and Technology of China(中国科学技术大学);Hefei University of Technology(合肥工业大学)
ReTool-Video: Recursive Tool-Using Video Agents with Meta-Augmented Tool Grounding
Chongqing University(重庆大学);Tianjin University(天津大学);MAIS, Institute of Automation, Chinese Academy of Sciences(自动化研究所,中国科学院MAIS);Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR), Singapore(新加坡科技研究局高性能计算研究所);Chongqing National Data AI Research Institute, AI Research Lab(重庆国家数据AI研究院,AI研究实验室)
GUIDE: Resolving Domain Bias in GUI Agents through Real-Time Web Video Retrieval and Plug-and-Play Annotation
Shanghai Jiao Tong University(上海交通大学);State Key Laboratory for General Artificial Intelligence, BIGAI(通用人工智能国家重点实验室,北京通用人工智能研究院);Beijing Institute of Technology(北京理工大学)
Auto-US: An Ultrasound Video Diagnosis Agent Using Video Classification Framework and LLMs
The Anhui Provincial International Joint Research Center for Advanced Technology in Medical Imaging(安徽省级国际联合先进技术医学影像研究中心);Anhui University(安徽大学);School of First Clinical Medicine(第一临床医学院);Anhui University of Science and Technology(安徽理工大学);School of Medicine(医学院);The First Hospital(第一医院)
UniVA: Universal Video Agent towards Open-Source Next-Generation Video Generalist
Singapore Management University(新加坡管理大学);University of Rochester(罗切斯特大学);University College London(伦敦大学学院);National University of Singapore(新加坡国立大学);The Chinese University of Hong Kong(香港中文大学);Stanford University(斯坦福大学)
Think, Then Verify: A Hypothesis-Verification Multi-Agent Framework for Long Video Understanding
College of Computer Science, Zhejiang University of Technology(浙江工业大学计算机科学学院);Zhejiang Key Laboratory of Visual Information Intelligent Processing(浙江视觉信息智能处理重点实验室);UC Berkeley(伯克利大学);College of Computer Science and Technology, East China Normal University(华东师范大学计算机科学与技术学院)
Video-Browser: Towards Open-web Agentic Video Browsing
Singapore Management University(新加坡管理大学);University of Trento(特伦托大学);Beijing Academy of Artificial Intelligence(北京智源人工智能研究院);Hong Kong Polytechnic University(香港理工大学)