DEEM: Diffusion Models Serve as the Eyes of Large Language Models for Image Perception
Shenzhen Key Laboratory for High Performance Data Mining, SIAT, CAS(中国科学院深圳先进技术研究院);University of Chinese Academy of Sciences(中国科学院大学);Tsinghua University(清华大学);Huazhong University of Science and Technology(华中科技大学);University of New South Wales(新南威尔士大学);National University of Singapore(新加坡国立大学);University of Science and Technology of China(中国科学技术大学);The University of Sydney(悉尼大学);Alibaba Group(阿里巴巴集团)
Run Luo, Yunshui Li, Longze Chen, Wanwei He, Ting-En Lin, Ziqiang Liu, Lei Zhang, Zikai Song, Xiaobo Xia, Tongliang Liu, Min Yang, Binyuan Hui
备注:
25 pages. arXiv admin note: text overlap with arXiv:2401.10208 by other authors
The development of large language models (LLMs) has significantly advanced the emergence of large multimodal models (LMMs). While LMMs have achieved tremendous success by promoting the synergy between multimodal comprehension and creation, they often face challenges when confronted with out-of-distribution data, such as which can hardly distinguish orientation, quantity, color, structure, etc. This is primarily due to their reliance on image encoders trained to encode images into task-relevant features, which may lead them to disregard irrelevant details. Delving into the modeling capabilities of diffusion models for images naturally prompts the question: Can diffusion models serve as the eyes of large language models for image perception? In this paper, we propose DEEM, a simple but effective approach that utilizes the generative feedback of diffusion models to align the semantic distributions of the image encoder. This addresses the drawbacks of previous methods that solely relied on image encoders like CLIP-ViT, thereby enhancing the model's resilience against out-of-distribution samples and reducing visual hallucinations. Importantly, this is achieved without requiring additional training modules and with fewer training parameters. We extensively evaluated DEEM on both our newly constructed RobustVQA benchmark and other well-known benchmarks, POPE and MMVP, for visual hallucination and perception. In particular, DEEM improves LMM's visual perception performance to a large extent (e.g., 4% higher on RobustVQA, 6.5% higher on MMVP and 12.8 % higher on POPE ). Compared to the state-of-the-art interleaved content generation models, DEEM exhibits enhanced robustness and a superior capacity to alleviate model hallucinations while utilizing fewer trainable parameters, less pre-training data (10%), and a smaller base model size.
Dementia Through Different Eyes: Explainable Modeling of Human and LLM Perceptions for Early Awareness
Faculty of Data and Decision Sciences, Technion(技术学院数据与决策科学学院);Paul G. Allen School of Computer Science & Engineering, University of Washington(华盛顿大学保罗·G·艾伦计算机科学与工程学院)
Bridging Ears and Eyes: Analyzing Audio and Visual Large Language Models to Humans in Visible Sound Recognition and Reducing Their Sensory Gap via Cross-Modal Distillation
Columbia University(哥伦比亚大学);University of Washington(华盛顿大学)
Better Eyes, Better Thoughts: Why Vision Chain-of-Thought Fails in Medicine
Department of Computer Science, City University of Hong Kong (Dongguan)(香港城市大学(东莞)计算机科学系);College of Computing and Data Science (CCDS), Nanyang Technological University(南洋理工大学计算与数据科学学院);School of Computer Science, Northwestern Polytechnical University(西北工业大学计算机科学学院)
Open Eyes, Then Reason: Fine-grained Visual Mathematical Understanding in MLLMs
Australian Institute for Machine Learning, University of Adelaide(阿德莱德大学澳大利亚机器学习研究所);Georgia Institute of Technology(佐治亚理工学院);Nanjing University of Science and Technology(南京理工大学);University of Oxford(牛津大学);The Ohio State University(俄亥俄州立大学)
A Perception CNN for Facial Expression Recognition
School of Computer Science and Technology, Harbin Institute of Technology(计算机科学与技术学院,哈尔滨工业大学);School of Software, Northwestern Polytechnical University(软件学院,西北工业大学);School of Software, Zhengzhou University of Light Industry(软件学院,郑州轻工业学院);School of Computer Science, Northwestern Polytechnical University(计算机科学学院,西北工业大学);School of Science and Engineering, Chinese University of Hong Kong, Shenzhen(科学与工程学院,香港中文大学(深圳))
When Eye-Tracking Meets Machine Learning: A Systematic Review on Applications in Medical Image Analysis
Macquarie University(麦考瑞大学);Macquarie Medical School(麦考瑞医学院);Faculty of Medicine, Health and Human Sciences(医学与健康科学学院);School of Computing(计算机学院);Faculty of Science and Engineering(科学与工程学院);Centre for Health Informatics(健康信息学中心);Australian Institute of Health Innovation(澳大利亚健康创新研究院)