The ability to reason about temporal and causal events from videos lies at the core of human intelligence. Most video reasoning benchmarks, however, focus on pattern recognition from complex visual and language input, instead of on causal structure. We study the complementary problem, exploring the temporal and causal structures behind videos of objects with simple visual appearance. To this end, we introduce the CoLlision Events for Video REpresentation and Reasoning (CLEVRER), a diagnostic video dataset for systematic evaluation of computational models on a wide range of reasoning tasks. Motivated by the theory of human casual judgment, CLEVRER includes four types of questions: descriptive (e.g., "what color"), explanatory ("what is responsible for"), predictive ("what will happen next"), and counterfactual ("what if"). We evaluate various state-of-the-art models for visual reasoning on our benchmark. While these models thrive on the perception-based task (descriptive), they perform poorly on the causal tasks (explanatory, predictive and counterfactual), suggesting that a principled approach for causal reasoning should incorporate the capability of both perceiving complex visual and language inputs, and understanding the underlying dynamics and causal relations. We also study an oracle model that explicitly combines these components via symbolic representations.
Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning
The University of Hong Kong(香港大学);MIT CSAIL(麻省理工学院计算机科学与人工智能实验室);Stanford University(斯坦福大学);MIT BCS, CBMM, CSAIL(麻省理工学院脑与认知科学系、大脑、心智与机器中心、计算机科学与人工智能实验室);MIT-IBM Watson AI Lab(MIT-IBM沃森人工智能实验室)
COMET: Contrastive Motion-Enhanced Temporal Reasoning for Video Multimodal Large Language Models
Peking University(北京大学);Nanyang Technological University(南洋理工大学);National University of Singapore(新加坡国立大学);Sun Yat-Sen University(中山大学);Pingan Technology(平安科技)
GRADEO: Towards Human-Like Evaluation for Text-to-Video Generation via Multi-Step Reasoning
Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China(清华大学深圳国际研究生院,清华大学,深圳,中国);CSE department, The Chinese University of Hong Kong, Hong Kong,China(中国香港大学计算机科学与工程系,中国香港,中国);Department of Computer Science(计算机科学系);Engineering, The Hong Kong University of Science(工程,香港科学大学)
VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection
Beihang University(北京航空航天大学);The University of Hong Kong(香港大学);Shanghai AI Lab(上海人工智能实验室);Central South University(中南大学);Sangfor Technologies Inc.(深信服科技股份有限公司);CUHK(香港中文大学)
Reasoning via Video: The First Evaluation of Video Models' Reasoning Abilities through Maze-Solving Tasks
DeepWisdom;Tsinghua University(清华大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Renmin University of China(中国人民大学);University of Oxford(牛津大学);National University of Singapore(新加坡国立大学);Xiamen University(厦门大学);Hong Kong University of Science and Technology (GuangZhou)(香港科技大学(广州))
VideoExplorer: Think With Videos For Agentic Long-Video Understanding
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学人工智能学院);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Beijing University of Posts and Telecommunications(北京邮电大学);Hong Kong Polytechnic University(香港理工大学);Peking University(北京大学);University of Trento(特伦托大学)
ViTCoT: Video-Text Interleaved Chain-of-Thought for Boosting Video Understanding in Large Language Models
School of Computer Science and Engineering, Central South University(计算机科学与工程学院,中南大学);Research Center for SCIR, Harbin Institute of Technology(SCIR研究中心,哈尔滨工业大学);NExT Research Center, National University of Singapore(NExT研究中心,新加坡国立大学)
DaMO: A Data-Efficient Multimodal Orchestrator for Temporal Reasoning with Video LLMs
College of Artificial Intelligence, National Yang Ming Chiao Tung University(人工智能学院,阳明交通大学);Institute of Population Health Sciences, National Health Research Institutes(人口健康科学研究所,国家健康研究院);Department of Computer Science, National Yang Ming Chiao Tung University(计算机科学系,阳明交通大学)
BoxTuning: Directly Injecting the Object Box for Multimodal Model Fine-Tuning
School of Computer Science and Technology, Tianjin University(天津大学计算机科学与技术学院);Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced Technology(深圳理工大学计算机科学与人工智能学院)
ReasVQA: Advancing VideoQA with Imperfect Reasoning Process
Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);National Key Laboratory of General Artificial Intelligence, Peking University(北京大学通用人工智能国家重点实验室)
UpstreamQA: A Modular Framework for Explicit Reasoning on Video Question Answering Tasks
Lincoln North Star High School(林肯北星高中);The Charter School of Wilmington(威尔明顿 charter 学校);Greenwich High School(格林威治高中);Santa Susana High School(圣塔苏娜高中);UC Berkeley(伯克利大学)