With the rapid advancement of video understanding, existing benchmarks are becoming increasingly saturated, exposing a critical discrepancy between inflated leaderboard scores and real-world model capabilities. To address this widening gap, we introduce Video-MME-v2, a comprehensive benchmark designed to rigorously evaluate the robustness and faithfulness of video understanding. To systematically evaluate model capabilities, we design a \textbf{progressive tri-level hierarchy} that incrementally increases the complexity of video comprehension, ranging from multi-point visual information aggregation, to temporal dynamics modeling, and ultimately to complex multimodal reasoning. Besides, in contrast to conventional per-question accuracy, we propose a \textbf{group-based non-linear evaluation} strategy that enforces both consistency across related queries and coherence in multi-step reasoning. It penalizes fragmented or guess-based correctness and assigns credit only to answers supported by valid reasoning. To guarantee data quality, Video-MME-v2 is constructed through a rigorously controlled human annotation pipeline, involving 12 annotators and 50 independent reviewers. Backed by \textbf{3,300 human-hours} and up to \textbf{5 rounds} of quality assurance, Video-MME-v2 aims to serve as one of the most authoritative video benchmarks. Extensive experiments reveal a substantial gap between current best model Gemini-3-Pro and human experts, and uncover a clear hierarchical bottleneck where errors in visual information aggregation and temporal modeling propagate to limit high-level reasoning. We further find that thinking-based reasoning is highly dependent on textual cues, improving performance with subtitles but sometimes degrading it in purely visual settings. By exposing these limitations, Video-MME-v2 establishes a demanding new testbed for the development of next-generation video MLLMs.
MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models
State Key Laboratory for Novel Software Technology, Nanjing University(南京大学新型软件技术国家重点实验室);School of Intelligence Science and Technology, Nanjing University(南京大学智能科学与技术学院);Tencent Youtu Lab(腾讯优图实验室);Xiamen University(厦门大学);CASIA(中国科学院自动化研究所)
MME-Reasoning: A Comprehensive Benchmark for Logical Reasoning in MLLMs
Fudan University(复旦大学);MMLab, The Chinese University of Hong Kong(中大香港人工智能实验室);Shanghai AI Laboratory(上海人工智能实验室);University of Science and Technology of China(中国科学技术大学);Nanjing University(南京大学)
GCAgent: Long-Video Understanding via Schematic and Narrative Episodic Memory
Integrated Vision and Language Lab., School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST)(整合视觉与语言实验室,电气工程学院,韩国科学技术院(KAIST));Visual Intelligence Research Section, Superintelligence Creative Research Laboratory, Electronics and Telecommunications Research Institute (ETRI)(视觉智能研究部,超智能创意研究实验室,电子电信研究院)
MME-CoF-Pro: Evaluating Reasoning Coherence in Video Generative Models with Text and Visual Hints
Northeastern University(东北大学);The Chinese University of Hong Kong(香港中文大学);Westlake University(西湖大学);ByteDance Seed(字节跳动种子);Peking University(北京大学);NVIDIA(英伟达)
Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis
State Key Laboratory for Novel Software Technology(新型软件技术国家重点实验室);School of Intelligence Science and Technology(智能科学与技术学院);State Key Laboratory of Cognitive Intelligence(认知智能国家重点实验室);XMU(厦门大学);HKU(香港大学);PKU(北京大学);CUHK(香港中文大学);ECNU(华东师范大学);CASIA(中国科学院自动化研究所)
Perceive, Reflect and Understand Long Video: Progressive Multi-Granular Clue Exploration with Interactive Agents
School of Electronic Engineering, Xidian University(西安电子科技大学电子工程学院);Department of Computer Science and Engineering, Hong Kong University of Science and Technology(香港理工大学计算机科学与工程系);College of Computer and Information, Hohai University(河海大学计算机与信息学院)
MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs
Nanjing University(南京大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Science and Technology of China(中国科学技术大学);Nanyang Technological University(南洋理工大学)
VideoGAIA: A Benchmark for General AI Assistants on Agentic Video Understanding
The Chinese University of Hong Kong(香港中文大学);Ant Group(蚂蚁集团);Tsinghua University(清华大学);Tongji University(同济大学);The Hong Kong University of Science and Technology(香港科技大学)
What Makes for Good Visual Instructions? Synthesizing Complex Visual Reasoning Instructions for Visual Instruction Tuning
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);School of Information, Renmin University of China(中国人民大学信息学院);Meituan Group(美团集团)
Uncovering What, Why and How: A Comprehensive Benchmark for Causation Understanding of Video Anomaly
Beijing University of Posts and Telecommunications(北京邮电大学);Queen Mary University of London(伦敦玛丽女王大学);Nanyang Technological University(南洋理工大学);Yunnan University(云南大学);Zhejiang University(浙江大学);China Telecom Co., Ltd. Sichuan Branch(中国电信股份有限公司四川分公司)
OmniHD-Scenes: A Next-Generation Multimodal Dataset for Autonomous Driving
School of Automotive Studies, Tongji University(同济大学汽车学院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);AI Foundation(2077AI基金会);Momoni AI;College of Information Science and Electronic Engineering, Zhejiang University(浙江大学信息科学与电子工程学院);School of Information and Electrical Engineering, Hangzhou City University(杭州城市学院信息与电气工程学院)
VKnowU: Evaluating Visual Knowledge Understanding in Multimodal LLMs
University of Science and Technology of China(中国科学技术大学);Shanghai AI Laboratory(上海人工智能实验室);Nanjing University(南京大学);Shanghai Innovation Institute(上海创新研究院);City University of Hong Kong(香港城市大学)
Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey
Peking University(北京大学);Beihang Univerisy(北京航空航天大学);University of Hongkong(香港大学);Shenzhen Institute of Advanced Technology, China Academy of Sciences(中国科学院深圳先进技术研究院);Tsinghua University(清华大学);M-A-P;The Chinese University of Hong Kong(香港中文大学);Alibaba Group(阿里巴巴集团);University of Cambridge(剑桥大学);Microsoft Research(微软研究院);UIUC(伊利诺伊大学厄巴纳-香槟分校);Humanify Inc.(Humanify公司);Zhejiang University(浙江大学)