arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.15232cs.CVcs.CL
收藏

DEEM:扩散模型作为大型语言模型进行图像感知的眼睛

DEEM: Diffusion Models Serve as the Eyes of Large Language Models for Image Perception

Shenzhen Key Laboratory for High Performance Data Mining, SIAT, CAS(中国科学院深圳先进技术研究院);University of Chinese Academy of Sciences(中国科学院大学);Tsinghua University(清华大学);Huazhong University of Science and Technology(华中科技大学);University of New South Wales(新南威尔士大学);National University of Singapore(新加坡国立大学);University of Science and Technology of China(中国科学技术大学);The University of Sydney(悉尼大学);Alibaba Group(阿里巴巴集团)

Run Luo, Yunshui Li, Longze Chen, Wanwei He, Ting-En Lin, Ziqiang Liu, Lei Zhang, Zikai Song, Xiaobo Xia, Tongliang Liu, Min Yang, Binyuan Hui

AI总结:

本文提出DEEM方法,利用扩散模型的生成反馈对齐图像编码器语义分布,以增强大型多模态模型对分布外样本的鲁棒性并减少视觉幻觉,在多个基准上显著提升感知性能且训练开销更低。

摘要:

大型语言模型(LLMs)的发展显著推动了大型多模态模型(LMMs)的出现。尽管LMMs通过促进多模态理解与生成之间的协同取得了巨大成功,但它们在面对分布外数据时常常面临挑战,例如难以区分方向、数量、颜色、结构等。这主要是由于它们依赖的图像编码器被训练为将图像编码为与任务相关的特征,这可能导致它们忽略不相关的细节。深入探究扩散模型对图像的建模能力自然引出一个问题:扩散模型能否作为大型语言模型进行图像感知的眼睛?在本文中,我们提出DEEM,一种简单但有效的方法,利用扩散模型的生成反馈来对齐图像编码器的语义分布。这解决了以往仅依赖CLIP-ViT等图像编码器的方法的缺陷,从而增强了模型对分布外样本的鲁棒性并减少了视觉幻觉。重要的是,这一目标无需额外的训练模块,且训练参数更少即可实现。我们在新构建的RobustVQA基准以及POPE和MMVP等其他知名基准上对DEEM进行了广泛评估,以测试视觉幻觉和感知能力。特别是,DEEM大幅提升了LMM的视觉感知性能(例如,在RobustVQA上提高4%,在MMVP上提高6.5%,在POPE上提高12.8%)。与最先进的交错内容生成模型相比,DEEM表现出更强的鲁棒性和更优越的缓解模型幻觉的能力,同时使用的可训练参数更少、预训练数据更少(10%)且基础模型规模更小。

英文摘要:

The development of large language models (LLMs) has significantly advanced the emergence of large multimodal models (LMMs). While LMMs have achieved tremendous success by promoting the synergy between multimodal comprehension and creation, they often face challenges when confronted with out-of-distribution data, such as which can hardly distinguish orientation, quantity, color, structure, etc. This is primarily due to their reliance on image encoders trained to encode images into task-relevant features, which may lead them to disregard irrelevant details. Delving into the modeling capabilities of diffusion models for images naturally prompts the question: Can diffusion models serve as the eyes of large language models for image perception? In this paper, we propose DEEM, a simple but effective approach that utilizes the generative feedback of diffusion models to align the semantic distributions of the image encoder. This addresses the drawbacks of previous methods that solely relied on image encoders like CLIP-ViT, thereby enhancing the model's resilience against out-of-distribution samples and reducing visual hallucinations. Importantly, this is achieved without requiring additional training modules and with fewer training parameters. We extensively evaluated DEEM on both our newly constructed RobustVQA benchmark and other well-known benchmarks, POPE and MMVP, for visual hallucination and perception. In particular, DEEM improves LMM's visual perception performance to a large extent (e.g., 4% higher on RobustVQA, 6.5% higher on MMVP and 12.8 % higher on POPE ). Compared to the state-of-the-art interleaved content generation models, DEEM exhibits enhanced robustness and a superior capacity to alleviate model hallucinations while utilizing fewer trainable parameters, less pre-training data (10%), and a smaller base model size.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑