We propose to use automatically generated instruction-following data to improve the zero-shot capabilities of a large multimodal model with additional support for generative and image editing tasks. We achieve this by curating a new multimodal instruction-following set using GPT-4V and existing datasets for image generation and editing. Using this instruction set and the existing LLaVA-Finetune instruction set for visual understanding tasks, we produce GenLLaVA, a Generative Large Language and Visual Assistant. GenLLaVA is built through a strategy that combines three types of large pretrained models through instruction finetuning: Mistral for language modeling, SigLIP for image-text matching, and StableDiffusion for text-to-image generation. Our model demonstrates visual understanding capabilities superior to LLaVA and additionally demonstrates competitive results with native multimodal models such as Unified-IO 2, paving the way for building advanced general-purpose visual assistants by effectively re-using existing multimodal models. We open-source our dataset, codebase, and model checkpoints to foster further research and application in this domain.
Instruction-Free Tuning of Large Vision Language Models for Medical Instruction Following
Department of Electrical and Computer Engineering, The University of British Columbia(英属哥伦比亚大学电气与计算机工程系);Department of Robotics and Mechatronics Engineering, Daegu Gyeongbuk Institute of Science and Technology (DGIST)(大邱庆北科学技术院机器人与机电工程系);Division of Intelligent Robot, Daegu Gyeongbuk Institute of Science and Technology (DGIST)(大邱庆北科学技术院智能机器人系);Vector Institute(向量研究所);Department of Computer Science and Engineering, Pohang University of Science and Technology (POSTECH)(釜山科学技术大学计算机科学与工程系)
MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversity
Shanghai AI Laboratory(上海人工智能实验室);SenseTime Research(商汤科技研究院);Tsinghua University(清华大学);Nanjing University(南京大学);Fudan University(复旦大学);The Chinese University of Hong Kong(香港中文大学);Shanghai Jiao Tong University(上海交通大学)
CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Model
Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China(电子科技大学深圳高等研究院);University of Electronic Science and Technology of China(电子科技大学);Tongji University(同济大学)
Reflective Instruction Tuning: Mitigating Hallucinations in Large Vision-Language Models
Southern University of Science and Technology(南方科技大学);The University of Hong Kong(香港大学);Shenzhen Polytechnic University(深圳职业技术大学);ZTE Corporation(中兴通讯股份有限公司);Peng Cheng Laboratory(鹏城实验室)
Adapting Large VLMs with Iterative and Manual Instructions for Generative Low-light Enhancement
Dalian University of Technology(大连理工大学);National University of Singapore(新加坡国立大学);Hong Kong Polytechnic University(香港理工大学);University of California, San Francisco(加州大学旧金山分校);Nanjing University of Science and Technology(南京理工大学)
Instruction-Grounded Visual Projectors for Continual Learning of Generative Vision-Language Models
School of Computer Science and Engineering, Chung-Ang University(Chung-Ang 大学计算机科学与工程学院);School of Computer Science, University of Birmingham(布拉德福德大学计算机科学学院)
Do we Really Need Visual Instructions? Towards Visual Instruction-Free Fine-tuning for Large Vision-Language Models
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);School of Information, Renmin University of China(中国人民大学信息学院);Alibaba Group(阿里巴巴集团)
What Makes for Good Visual Instructions? Synthesizing Complex Visual Reasoning Instructions for Visual Instruction Tuning
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);School of Information, Renmin University of China(中国人民大学信息学院);Meituan Group(美团集团)
Continual Instruction Tuning for Large Multimodal Models
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);The Peng Cheng Laboratory(鹏城实验室);Wuhan AI Research(武汉人工智能研究院)
Biomedical Visual Instruction Tuning with Clinician Preference Alignment
Stanford University(斯坦福大学);Emory University(埃默里大学);University of California, Berkeley(加州大学伯克利分校);University of Washington(华盛顿大学);Massachusetts General Hospital(麻省总医院);Harvard Medical School(哈佛医学院)