LlamaV-o1: Rethinking Step-by-step Visual Reasoning in LLMs
Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）；University of Central Florida（中佛罗里达大学）；Linköping University（林雪平大学）；Australian National University（澳大利亚国立大学）
Omkar Thawakar, Dinura Dissanayake, Ketan More, Ritesh Thawkar, Ahmed Heakl, Noor Ahsan, Yuhao Li, Mohammed Zumri, Jean Lahoud, Rao Muhammad Anwer, Hisham Chola… 展开作者
Omkar Thawakar, Dinura Dissanayake, Ketan More, Ritesh Thawkar, Ahmed Heakl, Noor Ahsan, Yuhao Li, Mohammed Zumri, Jean Lahoud, Rao Muhammad Anwer, Hisham Cholakkal, Ivan Laptev, Mubarak Shah, Fahad Shahbaz Khan, Salman Khan
Reasoning is a fundamental capability for solving complex multi-step problems, particularly in visual contexts where sequential step-wise understanding is essential. Existing approaches lack a comprehensive framework for evaluating visual reasoning and do not emphasize step-wise problem-solving. To this end, we propose a comprehensive framework for advancing step-by-step visual reasoning in large language models (LMMs) through three key contributions. First, we introduce a visual reasoning benchmark specifically designed to evaluate multi-step reasoning tasks. The benchmark presents a diverse set of challenges with eight different categories ranging from complex visual perception to scientific reasoning with over 4k reasoning steps in total, enabling robust evaluation of LLMs' abilities to perform accurate and interpretable visual reasoning across multiple steps. Second, we propose a novel metric that assesses visual reasoning quality at the granularity of individual steps, emphasizing both correctness and logical coherence. The proposed metric offers deeper insights into reasoning performance compared to traditional end-task accuracy metrics. Third, we present a new multimodal visual reasoning model, named LlamaV-o1, trained using a multi-step curriculum learning approach, where tasks are progressively organized to facilitate incremental skill acquisition and problem-solving. The proposed LlamaV-o1 is designed for multi-step reasoning and learns step-by-step through a structured training paradigm. Extensive experiments show that our LlamaV-o1 outperforms existing open-source models and performs favorably against close-source proprietary models. Compared to the recent Llava-CoT, our LlamaV-o1 achieves an average score of 67.3 with an absolute gain of 3.8\% across six benchmarks while being 5 times faster during inference scaling. Our benchmark, model, and code are publicly available.
RSGround-R1: Rethinking Remote Sensing Visual Grounding through Spatial Reasoning
School of Electrical and Electronic Engineering, Nanyang Technological University（电气电子工程学院，南洋理工大学）；MoE Key Laboratory of Interdisciplinary Research of Computation and Economics, Shanghai University of Finance and Economics（教育部计算与经济交叉学科重点实验室，上海财经大学）
GeoBench: Rethinking Multimodal Geometric Problem-Solving via Hierarchical Evaluation
Shanghai Jiao Tong University（上海交通大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Fudan University（复旦大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
Emotion-LLaMAv2 and MMEVerse: A New Framework and Benchmark for Multimodal Emotion Understanding
Shenzhen Technology University（深圳技术大学）；Shenzhen University of Information Technology（深圳信息科技学院）；University of Washington（华盛顿大学）；Foundation Model Team, Meituan（美团基础模型团队）；National University of Singapore（新加坡国立大学）；Tongji University（同济大学）
GThinker: Towards General Multimodal Reasoning via Cue-Guided Rethinking
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences（基础模型研究中心，自动化研究所，中国科学院）；ByteDance（字节跳动）；School of Artificial Intelligence, University of Chinese Academy of Sciences（人工智能学院，中国科学院大学）；Peng Cheng Laboratory（鹏城实验室）；Wuhan AI Research（武汉人工智能研究所）；Xi’an Jiaotong University（西安交通大学）；Renmin University of China（中国人民大学）
LatentOmni: Rethinking Omni-Modal Understanding via Unified Audio-Visual Latent Reasoning
School of AI, Shanghai Jiao Tong University（上海交通大学人工智能学院）；Kling Team, Kuaishou Technology（快手科技 Kling 团队）；Peking University（北京大学）；HKUST（香港科技大学）；CASIA（中国科学院自动化研究所）；Nanjing University（南京大学）；Renmin University of China（中国人民大学）；Tsinghua University（清华大学）
本文提出LatentOmni框架，通过统一的音频-视觉潜在空间进行多模态推理，利用特征级监督和Omni-Sync Position Embedding保持时间一致性，从而在多个音频-视觉推理基准测试中取得最佳性能。
Rethinking Foundation Model Collaboration: Enhancing Specialized Models through Proxy Task Reasoning
School of Vehicle and Mobility, Tsinghua University（清华大学车辆与运载学院）；Department of Urban Studies and Planning, Massachusetts Institute of Technology（麻省理工学院城市研究与规划系）
Rethinking the Video Sampling and Reasoning Strategies for Temporal Sentence Grounding
ProtagoLabs Inc（ProtagoLabs公司）；Microsoft Research（微软研究院）；Beijing Institute of Technology（北京理工大学）；Huazhong University of Science and Technology（华中科技大学）；Sichuan University（四川大学）；Dalian University of Technology（大连理工大学）；Lehigh University（利哈伊大学）；Peking University（北京大学）
Rethinking Facial Expression Recognition in the Era of Multimodal Large Language Models: Benchmark, Datasets, and Beyond
The Chinese University of Hong Kong（香港中文大学）；Peking University（北京大学）；Tencent（腾讯）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Tsinghua University（清华大学）；Nanyang Technological University（南洋理工大学）；Westlake University（西湖大学）
Mind Meets Space: Rethinking Agentic Spatial Intelligence from a Neuroscience-inspired Perspective
School of EEE, Nanyang Technological University（南洋理工大学电子工程系）；Department of Civil Engineering, Tsinghua University（清华大学土木工程系）；Dr. B. R. Ambedkar National Institute of Technology, Jalandhar（B.R.阿姆贝卡尔国家理工学院，贾兰德赫）；KTH Royal Institute of Technology（皇家理工学院）；College of AI, Tsinghua University（清华大学人工智能学院）；CCDS, Nanyang Technological University（南洋理工大学CCDS）
