Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);University of Hong Kong(香港大学);MEGVII Technology(旷视科技);StepFun(阶跃星辰);Centre for Artificial Intelligence and Robotics(人工智能与机器人中心);Hong Kong Institute of Science & Innovation, Chinese Academy of Science(中国科学院香港创新研究院)
This paper introduces reconstructive visual instruction tuning (ROSS), a family of Large Multimodal Models (LMMs) that exploit vision-centric supervision signals. In contrast to conventional visual instruction tuning approaches that exclusively supervise text outputs, ROSS prompts LMMs to supervise visual outputs via reconstructing input images. By doing so, it capitalizes on the inherent richness and detail present within input images themselves, which are often lost in pure text supervision. However, producing meaningful feedback from natural images is challenging due to the heavy spatial redundancy of visual signals. To address this issue, ROSS employs a denoising objective to reconstruct latent representations of input images, avoiding directly regressing exact raw RGB values. This intrinsic activation design inherently encourages LMMs to maintain image detail, thereby enhancing their fine-grained comprehension capabilities and reducing hallucinations. Empirically, ROSS consistently brings significant improvements across different visual encoders and language models. In comparison with extrinsic assistance state-of-the-art alternatives that aggregate multiple visual experts, ROSS delivers competitive performance with a single SigLIP visual encoder, demonstrating the efficacy of our vision-centric supervision tailored for visual outputs.
Biomedical Visual Instruction Tuning with Clinician Preference Alignment
Stanford University(斯坦福大学);Emory University(埃默里大学);University of California, Berkeley(加州大学伯克利分校);University of Washington(华盛顿大学);Massachusetts General Hospital(麻省总医院);Harvard Medical School(哈佛医学院)
LongForm: Effective Instruction Tuning with Reverse Instructions
Center for Information and Language Processing, LMU Munich(慕尼黑大学信息与语言处理中心);Munich Center for Machine Learning(慕尼黑机器学习中心);Language Technology Lab, University of Cambridge(剑桥大学语言技术实验室)
On the Effect of Instruction Tuning Loss on Generalization
Dept. of Electrical Engineering(电气工程系);Indian Institute of Technology Delhi(印度理工学院德里);Media and Data Science Research(媒体与数据科学研究);Adobe Inc., India(Adobe公司,印度)
University of Southern California(南加州大学);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);The Pennsylvania State University(宾夕法尼亚州立大学);University of California, Davis(加利福尼亚大学戴维斯分校)
Demystifying Instruction Mixing for Fine-tuning Large Language Models
Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);LibrAI;Monash University(莫纳什大学);University of British Columbia(不列颠哥伦比亚大学);The University of Melbourne(墨尔本大学)
SwitchCIT: Switching for Continual Instruction Tuning
Coordinated Science Laboratory(协调科学实验室);Department of Electrical and Computer Engineering(电气与计算机工程系);Department of Mathematics(数学系);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
What Makes for Good Visual Instructions? Synthesizing Complex Visual Reasoning Instructions for Visual Instruction Tuning
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);School of Information, Renmin University of China(中国人民大学信息学院);Meituan Group(美团集团)
An Empirical Study of Instruction-tuning Large Language Models in Chinese
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);APUS AiLMe Lab(APUS AiLMe实验室)
Roles of Scaling and Instruction Tuning in Language Perception: Model vs. Human Attention
National Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);Department of Linguistics and Translation, City University of Hong Kong(香港城市大学语言学及翻译学系)