What Makes for Good Visual Instructions? Synthesizing Complex Visual Reasoning Instructions for Visual Instruction Tuning
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；School of Information, Renmin University of China（中国人民大学信息学院）；Meituan Group（美团集团）
Yifan Du, Hangyu Guo, Kun Zhou, Wayne Xin Zhao, Jinpeng Wang, Chuyuan Wang, Mingchen Cai, Ruihua Song, Ji-Rong Wen
Visual instruction tuning is crucial for enhancing the zero-shot generalization capability of Multi-modal Large Language Models (MLLMs). In this paper, we aim to investigate a fundamental question: ''what makes for good visual instructions''. Through a comprehensive empirical study, we find that instructions focusing on complex visual reasoning tasks are particularly effective in improving the performance of MLLMs, with results correlating to instruction complexity. Based on this insight, we develop a systematic approach to automatically create high-quality complex visual reasoning instructions. Our approach employs a synthesize-complicate-reformulate paradigm, leveraging multiple stages to gradually increase the complexity of the instructions while guaranteeing quality. Based on this approach, we create the ComVint dataset with 32K examples, and fine-tune four MLLMs on it. Experimental results consistently demonstrate the enhanced performance of all compared MLLMs, such as a 27.86% and 27.60% improvement for LLaVA on MME-Perception and MME-Cognition, respectively. Our code and data are publicly available at the link: https://github.com/RUCAIBox/ComVint.
Synthesizing Instruction-Tuning Datasets with Contrastive Decoding
Department of Computer Science, School of Computing, Institute of Science Tokyo（东京科学大学计算机科学系）；National Institute of Advanced Industrial Science and Technology（国家先进工业科学与技术研究院）；Research and Development Center for Large Language Models, NII（大型语言模型研究与开发中心，日本信息产业技术综合研究所）
ImpRIF: Stronger Implicit Reasoning Leads to Better Complex Instruction Following
ByteDance China（字节跳动中国）；School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；State Key Laboratory of Virtual Reality Technology and Systems, Beihang University（北京航空航天大学虚拟现实技术与系统国家重点实验室）
SQLForge: Synthesizing Reliable and Diverse Data to Enhance Text-to-SQL Reasoning in LLMs
University of Science and Technology of China（中国科学技术大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）
Towards an AI Musician: Synthesizing Sheet Music Problems for Musical Reasoning
University of Science and Technology of China（中国科学技术大学）；Shanghai AI Laboratory（上海人工智能实验室）；Sichuan University（四川大学）；Shanghai Jiao Tong University（上海交通大学）；University of Macau（澳门大学）；Tsinghua University（清华大学）；The Chinese University of Hong Kong（香港中文大学）
From Templates to Natural Language: Generalization Challenges in Instruction-Tuned LLMs for Spatial Reasoning
Computational Linguistics, Department of Linguistics University of Potsdam（乌特雷赫特大学语言学系计算语言学部）；German Research Center for Artificial Intelligence (DFKI), Berlin（德国人工智能研究中心（DFKI）柏林）
RAIN-Merging: A Gradient-Free Method to Enhance Instruction Following in Large Reasoning Models with Preserved Thinking Format
Institute of Image Processing and Pattern Recoginition, School of Automation and Intelligent Sensing, Shanghai Jiao Tong University（图像处理与模式识别研究所，自动化与智能感知学院，上海交通大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；MoE Key Laboratory of System Control and Information Processing (Shanghai)（系统控制与信息处理国家重点实验室（上海））
EVIT: Event-Oriented Instruction Tuning for Event Reasoning
School of Computer Science, Peking University（北京大学计算机学院）；Key Laboratory of High Confidence Software Technologies (PKU), MOE, China（高可信软件技术重点实验室（北京大学））；Advanced Institute of Big Data（大数据先进研究院）
Select2Reason: Efficient Instruction-Tuning Data Selection for Long-CoT Reasoning
DataArc Tech Ltd.（DataArc科技有限公司）；Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；IDEA Research, International Digital Economy Academy（IDEA研究所，国际数字经济学院）；Hithink RoyalFlush Information Network Co., Ltd（Hithink RoyalFlush信息网络有限公司）
Reasoning Up the Instruction Ladder for Controllable Language Models
Department of Computer Science and Engineering, The Ohio State University（俄亥俄州立大学计算机科学与工程系）；Microsoft Research（微软研究院）；Allen Institute for AI（艾伦人工智能研究所）
Reflective Instruction Tuning: Mitigating Hallucinations in Large Vision-Language Models
Southern University of Science and Technology（南方科技大学）；The University of Hong Kong（香港大学）；Shenzhen Polytechnic University（深圳职业技术大学）；ZTE Corporation（中兴通讯股份有限公司）；Peng Cheng Laboratory（鹏城实验室）
Bridging Auxiliary Constraints to Resolve Instruction Following in Large Reasoning Models
The Chinese University of Hong Kong（香港中文大学）；University of International Relations（国际关系大学）；Tencent Jarvis Lab（腾讯Jarvis实验室）；Westlake University（西湖大学）；King’s College London（伦敦国王学院）
SilVar: Speech Driven Multimodal Model for Reasoning Visual Question Answering and Object Localization
Florida Institute of Technology（佛罗里达理工学院）；FPT University（FPT大学）；Uppsala University（乌普萨拉大学）；Knovel Engineering Lab（诺弗尔工程实验室）；University of Alabama at Birmingham（阿拉巴马大学伯明翰分校）
Does Reasoning Preserve Alignment? On the Trustworthiness of Large Reasoning Models
University of Colorado Boulder（科罗拉多大学博尔德分校）；University of Central Florida（中佛罗里达大学）；University of Maryland College Park（马里兰大学帕克分校）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）
