SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Models
University of California, Santa Cruz(加州大学圣克鲁兹分校);University of Texas at Dallas(德克萨斯大学达拉斯分校);The Pennsylvania State University(宾夕法尼亚州立大学);Amazon Research(亚马逊研究院)
Hardy Chen, Haoqin Tu, Fali Wang, Hui Liu, Xianfeng Tang, Xinya Du, Yuyin Zhou, Cihang Xie
This work revisits the dominant supervised fine-tuning (SFT) then reinforcement learning (RL) paradigm for training Large Vision-Language Models (LVLMs), and reveals a key finding: SFT can significantly undermine subsequent RL by inducing ``pseudo reasoning paths'' imitated from expert models. While these paths may resemble the native reasoning paths of RL models, they often involve prolonged, hesitant, less informative steps, and incorrect reasoning. To systematically study this effect, we introduce VLAA-Thinking, a new multimodal dataset designed to support reasoning in LVLMs. Constructed via a six-step pipeline involving captioning, reasoning distillation, answer rewrite and verification, VLAA-Thinking comprises high-quality, step-by-step visual reasoning traces for SFT, along with a more challenging RL split from the same data source. Using this dataset, we conduct extensive experiments comparing SFT, RL and their combinations. Results show that while SFT helps models learn reasoning formats, it often locks aligned models into imitative, rigid reasoning modes that impede further learning. In contrast, building on the Group Relative Policy Optimization (GRPO) with a novel mixed reward module integrating both perception and cognition signals, our RL approach fosters more genuine, adaptive reasoning behavior. Notably, our model VLAA-Thinker, based on Qwen2.5VL 3B, achieves top-1 performance on Open LMM Reasoning Leaderboard (https://huggingface.co/spaces/opencompass/Open_LMM_Reasoning_Leaderboard) among 4B scale LVLMs, surpassing the previous state-of-the-art by 1.8%. We hope our findings provide valuable insights in developing reasoning-capable LVLMs and can inform future research in this area.
Reassessing the Role of Supervised Fine-Tuning: An Empirical Study in VLM Reasoning
NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences(人工智能研究院 & 模式识别与人工智能研究所,中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(人工智能学院,中国科学院大学);Meituan(美团)
Med-R2: Perception and Reflection-driven Complex Reasoning for Medical Report Generation
The School of Computer Science, The University of Sydney(悉尼大学计算机科学学院);The School of Computing, Macquarie University(麦考瑞大学计算机学院);Doubao Medical Group, ByteDance(字节跳动 doubao 医疗集团)
CrowdVLM-R1: Expanding R1 Ability to Vision Language Model for Crowd Counting using Fuzzy Group Relative Policy Reward
Florida Atlantic University(佛罗里达大学);University of Southern California(南加州大学);University of Texas at Dallas(德克萨斯大学达拉斯分校);Georgia Institute of Technology(佐治亚理工学院);University of Utah(犹他大学)
Learning to Adapt SFT Data for Better Reasoning Generalization
Beihang University(北京航空航天大学);Tsinghua University(清华大学);Nanyang Technological University(南洋理工大学);Hangzhou International Innovation Institute(杭州国际创新研究院);Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing(北京未来区块链与隐私计算高级创新中心)
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China(人工智能安全国家重点实验室,计算技术研究所,中国科学院,北京);University of Chinese Academy of Sciences, Beijing, China(中国科学院大学,北京)
RLKD: Distilling LLMs' Reasoning via Reinforcement Learning
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS(人工智能安全国家重点实验室,计算技术研究所,中国科学院);University of Chinese Academy of Sciences(中国科学院大学);Huawei Inc.(华为公司)
RASFT: Rollout-Adaptive Supervised Fine-Tuning for Reasoning
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);New Jersey Institute of Technology(新泽西理工学院);Institute of Computing Technology, CAS(中国科学院计算技术研究所)
Rethinking Expert Trajectory Utilization in LLM Post-training for Mathematical Reasoning
Zhejiang University(浙江大学);School of Engineering, Westlake University(西湖大学工程学院);Institute of Advanced Technology, Westlake Institute for Advanced Study(西湖先进研究院技术研究所);Huawei Noah’s Ark Lab(华为诺亚实验室)
Enhancing Large Language Model Reasoning via Selective Critical Token Fine-Tuning
Southern University of Science and Technology(南方科技大学);Peking University(北京大学);Shanghai University of Finance and Economics(上海金融学院);Tsinghua University(清华大学)
Teaching Thinking Models to Reason with Tools: A Full-Pipeline Recipe for Tool-Integrated Reasoning
Zhejiang University(浙江大学);Shanghai AI Laboratory(上海人工智能实验室);Peking University(北京大学);The Chinese University of Hong Kong(香港中文大学);Tsinghua University(清华大学);University of Science and Technology of China(中国科学技术大学);Shanghai Jiao Tong University(上海交通大学)
Learning When to Reason for Text-to-SQL via SFT and DPO
Department of Electrical and Computer Engineering, Seoul National University(首尔国立大学电气与计算机工程系);AI Center, Samsung Electronics(三星电子人工智能中心);AIIS, ASRI, INMC, ISRC, and IPAI, Seoul National University(首尔国立大学人工智能研究所、高级科学研究所以及综合纳米技术中心、信息存储研究中心和人工智能平台研究所)