自监督学习视觉编码器中对大型视觉语言模型的隐蔽后门攻击
Stealthy Backdoor Attack in Self-Supervised Learning Vision Encoders for Large Vision Language Models
AI总结：
提出BadVision，首个针对大型视觉语言模型中自监督视觉编码器的隐蔽后门攻击方法，通过触发器优化实现高成功率视觉幻觉诱导，且能规避现有检测。
摘要：
自监督学习（SSL）视觉编码器能够学习高质量的图像表示，因此已成为开发大型视觉语言模型（LVLMs）视觉模态的关键组成部分。由于训练此类编码器成本高昂，预训练编码器被广泛共享并部署到许多LVLMs中，这些模型具有安全关键性或具有社会意义。在此实际场景下，我们揭示了一种新的后门威胁：仅通过破坏视觉编码器即可在LVLMs中诱导显著的视觉幻觉。由于这些编码器的共享和重用，许多下游LVLMs可能从编码器继承后门行为，导致广泛的后门传播。在本工作中，我们提出了BadVision，这是首个利用LVLMs的SSL视觉编码器中此漏洞的方法，采用了新颖的触发器优化和后门学习技术。我们在八项基准上对两种类型的SSL编码器和LVLMs评估了BadVision。我们表明，BadVision能有效驱动LVLMs产生攻击者选择的幻觉，攻击成功率超过99%，导致相对视觉理解误差达77.6%，同时保持隐蔽性。最先进的后门检测方法无法有效检测我们的攻击。
英文摘要：
Self-supervised learning (SSL) vision encoders learn high-quality image representations and thus have become a vital part of developing vision modality of large vision language models (LVLMs). Due to the high cost of training such encoders, pre-trained encoders are widely shared and deployed into many LVLMs, which are security-critical or bear societal significance. Under this practical scenario, we reveal a new backdoor threat that significant visual hallucinations can be induced into these LVLMs by merely compromising vision encoders. Because of the sharing and reuse of these encoders, many downstream LVLMs may inherit backdoor behaviors from encoders, leading to widespread backdoors. In this work, we propose BadVision, the first method to exploit this vulnerability in SSL vision encoders for LVLMs with novel trigger optimization and backdoor learning techniques. We evaluate BadVision on two types of SSL encoders and LVLMs across eight benchmarks. We show that BadVision effectively drives the LVLMs to attacker-chosen hallucination with over 99% attack success rate, causing a 77.6% relative visual understanding error while maintaining the stealthiness. SoTA backdoor detection methods cannot detect our attack effectively.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
