Large multimodal models (LMM) have recently shown encouraging progress with visual instruction tuning. In this note, we show that the fully-connected vision-language cross-modal connector in LLaVA is surprisingly powerful and data-efficient. With simple modifications to LLaVA, namely, using CLIP-ViT-L-336px with an MLP projection and adding academic-task-oriented VQA data with simple response formatting prompts, we establish stronger baselines that achieve state-of-the-art across 11 benchmarks. Our final 13B checkpoint uses merely 1.2M publicly available data, and finishes full training in ~1 day on a single 8-A100 node. We hope this can make state-of-the-art LMM research more accessible. Code and model will be publicly available.
LongForm: Effective Instruction Tuning with Reverse Instructions
Center for Information and Language Processing, LMU Munich（慕尼黑大学信息与语言处理中心）；Munich Center for Machine Learning（慕尼黑机器学习中心）；Language Technology Lab, University of Cambridge（剑桥大学语言技术实验室）
Biomedical Visual Instruction Tuning with Clinician Preference Alignment
Stanford University（斯坦福大学）；Emory University（埃默里大学）；University of California, Berkeley（加州大学伯克利分校）；University of Washington（华盛顿大学）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）
On the Effect of Instruction Tuning Loss on Generalization
Dept. of Electrical Engineering（电气工程系）；Indian Institute of Technology Delhi（印度理工学院德里）；Media and Data Science Research（媒体与数据科学研究）；Adobe Inc., India（Adobe公司，印度）
Instruction tuning enables pretrained language models to perform new tasks from inference-time natural language descriptions. These approaches rely on vast amounts of human supervision in the form of crowdsourced datasets or user interactions. In this work, we introduce Unnatural Instructions: a large dataset of creative and diverse instructions, collected with virtually no human labor. We collect 64,000 examples by prompting a language model with three seed examples of instructions and eliciting a fourth. This set is then expanded by prompting the model to rephrase each instruction, creating …
We study the design decisions of publicly available instruction tuning methods, and break down the development of Flan 2022 (Chung et al., 2022). Through careful ablation studies on the Flan Collection of tasks and methods, we tease apart the effect of design decisions which enable Flan-T5 to outperform prior work by 3-17%+ across evaluation settings. We find task balancing and enrichment techniques are overlooked but critical to effective instruction tuning, and in particular, training with mixed prompt settings (zero-shot, few-shot, and chain-of-thought) actually yields stronger (2%+) perfor…
University of Southern California（南加州大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；The Pennsylvania State University（宾夕法尼亚州立大学）；University of California, Davis（加利福尼亚大学戴维斯分校）
Demystifying Instruction Mixing for Fine-tuning Large Language Models
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；LibrAI；Monash University（莫纳什大学）；University of British Columbia（不列颠哥伦比亚大学）；The University of Melbourne（墨尔本大学）
What Makes for Good Visual Instructions? Synthesizing Complex Visual Reasoning Instructions for Visual Instruction Tuning
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；School of Information, Renmin University of China（中国人民大学信息学院）；Meituan Group（美团集团）
Do we Really Need Visual Instructions? Towards Visual Instruction-Free Fine-tuning for Large Vision-Language Models
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；School of Information, Renmin University of China（中国人民大学信息学院）；Alibaba Group（阿里巴巴集团）
SwitchCIT: Switching for Continual Instruction Tuning
Coordinated Science Laboratory（协调科学实验室）；Department of Electrical and Computer Engineering（电气与计算机工程系）；Department of Mathematics（数学系）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Instruction-Free Tuning of Large Vision Language Models for Medical Instruction Following
Department of Electrical and Computer Engineering, The University of British Columbia（英属哥伦比亚大学电气与计算机工程系）；Department of Robotics and Mechatronics Engineering, Daegu Gyeongbuk Institute of Science and Technology (DGIST)（大邱庆北科学技术院机器人与机电工程系）；Division of Intelligent Robot, Daegu Gyeongbuk Institute of Science and Technology (DGIST)（大邱庆北科学技术院智能机器人系）；Vector Institute（向量研究所）；Department of Computer Science and Engineering, Pohang University of Science and Technology (POSTECH)（釜山科学技术大学计算机科学与工程系）
