Thanks to the emerging of foundation models, the large language and vision models are integrated to acquire the multimodal ability of visual captioning, question answering, etc. Although existing multimodal models present impressive performance of visual understanding and reasoning, their limits are still largely under-explored due to the scarcity of high-quality instruction tuning data. To push the limits of multimodal capability, we Scale up Visual Instruction Tuning (SVIT) by constructing a dataset of 4.2 million visual instruction tuning data including 1.6M conversation question-answer (QA) pairs, 1.6M complex reasoning QA pairs, 1.0M referring QA pairs and 106K detailed image descriptions. Besides the volume, the proposed dataset is also featured by the high quality and rich diversity, which is generated by prompting GPT-4 with the abundant manual annotations of images. We also propose a new data recipe to select subset with better diversity and balance, which evokes model's superior capabilities. Extensive experiments verify that SVIT-v1.5, trained on the proposed dataset, outperforms state-of-the-art Multimodal Large Language Models on popular benchmarks. The data and code are publicly available at https://github.com/BAAI-DCAI/Visual-Instruction-Tuning.
MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversity
Shanghai AI Laboratory（上海人工智能实验室）；SenseTime Research（商汤科技研究院）；Tsinghua University（清华大学）；Nanjing University（南京大学）；Fudan University（复旦大学）；The Chinese University of Hong Kong（香港中文大学）；Shanghai Jiao Tong University（上海交通大学）
CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Model
Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China（电子科技大学深圳高等研究院）；University of Electronic Science and Technology of China（电子科技大学）；Tongji University（同济大学）
Continual Instruction Tuning for Large Multimodal Models
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；The Peng Cheng Laboratory（鹏城实验室）；Wuhan AI Research（武汉人工智能研究院）
Instruction-Free Tuning of Large Vision Language Models for Medical Instruction Following
Department of Electrical and Computer Engineering, The University of British Columbia（英属哥伦比亚大学电气与计算机工程系）；Department of Robotics and Mechatronics Engineering, Daegu Gyeongbuk Institute of Science and Technology (DGIST)（大邱庆北科学技术院机器人与机电工程系）；Division of Intelligent Robot, Daegu Gyeongbuk Institute of Science and Technology (DGIST)（大邱庆北科学技术院智能机器人系）；Vector Institute（向量研究所）；Department of Computer Science and Engineering, Pohang University of Science and Technology (POSTECH)（釜山科学技术大学计算机科学与工程系）
Reflective Instruction Tuning: Mitigating Hallucinations in Large Vision-Language Models
Southern University of Science and Technology（南方科技大学）；The University of Hong Kong（香港大学）；Shenzhen Polytechnic University（深圳职业技术大学）；ZTE Corporation（中兴通讯股份有限公司）；Peng Cheng Laboratory（鹏城实验室）
Do we Really Need Visual Instructions? Towards Visual Instruction-Free Fine-tuning for Large Vision-Language Models
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；School of Information, Renmin University of China（中国人民大学信息学院）；Alibaba Group（阿里巴巴集团）
What Makes for Good Visual Instructions? Synthesizing Complex Visual Reasoning Instructions for Visual Instruction Tuning
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；School of Information, Renmin University of China（中国人民大学信息学院）；Meituan Group（美团集团）
Pilot: Building the Federated Multimodal Instruction Tuning Framework
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Pengcheng Laboratory（鹏城实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)（中国科学院大学人工智能学院）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））
Biomedical Visual Instruction Tuning with Clinician Preference Alignment
Stanford University（斯坦福大学）；Emory University（埃默里大学）；University of California, Berkeley（加州大学伯克利分校）；University of Washington（华盛顿大学）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）
