arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.09575cs.CVcs.AIcs.CLcs.LG
收藏

重建式视觉指令微调

Reconstructive Visual Instruction Tuning

Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);University of Hong Kong(香港大学);MEGVII Technology(旷视科技);StepFun(阶跃星辰);Centre for Artificial Intelligence and Robotics(人工智能与机器人中心);Hong Kong Institute of Science & Innovation, Chinese Academy of Science(中国科学院香港创新研究院)

Haochen Wang, Anlin Zheng, Yucheng Zhao, Tiancai Wang, Zheng Ge, Xiangyu Zhang, Zhaoxiang Zhang

AI总结:

本文提出重建式视觉指令微调方法ROSS,通过去噪重建输入图像的潜在表示来监督视觉输出,从而增强多模态模型的细粒度理解能力并减少幻觉,在单个视觉编码器下取得与多专家方法相当的性能。

摘要:

本文提出重建式视觉指令微调(ROSS),这是一类利用以视觉为中心的监督信号的大型多模态模型(LMMs)。与仅监督文本输出的传统视觉指令微调方法不同,ROSS 通过重建输入图像来促使 LMMs 监督视觉输出。通过这种方式,它充分利用了输入图像本身固有的丰富性和细节,而这些在纯文本监督中往往会丢失。然而,由于视觉信号存在严重的空间冗余,从自然图像中产生有意义的反馈具有挑战性。为解决这一问题,ROSS 采用去噪目标来重建输入图像的潜在表示,避免直接回归精确的原始 RGB 值。这种内在激活设计天然地鼓励 LMMs 保持图像细节,从而增强其细粒度理解能力并减少幻觉。实验表明,ROSS 在不同的视觉编码器和语言模型上均持续带来显著提升。与聚合多个视觉专家的外部辅助型最先进替代方案相比,ROSS 仅使用单个 SigLIP 视觉编码器即可提供具有竞争力的性能,证明了我们针对视觉输出定制的以视觉为中心的监督的有效性。

英文摘要:

This paper introduces reconstructive visual instruction tuning (ROSS), a family of Large Multimodal Models (LMMs) that exploit vision-centric supervision signals. In contrast to conventional visual instruction tuning approaches that exclusively supervise text outputs, ROSS prompts LMMs to supervise visual outputs via reconstructing input images. By doing so, it capitalizes on the inherent richness and detail present within input images themselves, which are often lost in pure text supervision. However, producing meaningful feedback from natural images is challenging due to the heavy spatial redundancy of visual signals. To address this issue, ROSS employs a denoising objective to reconstruct latent representations of input images, avoiding directly regressing exact raw RGB values. This intrinsic activation design inherently encourages LMMs to maintain image detail, thereby enhancing their fine-grained comprehension capabilities and reducing hallucinations. Empirically, ROSS consistently brings significant improvements across different visual encoders and language models. In comparison with extrinsic assistance state-of-the-art alternatives that aggregate multiple visual experts, ROSS delivers competitive performance with a single SigLIP visual encoder, demonstrating the efficacy of our vision-centric supervision tailored for visual outputs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑