大视觉语言模型中压缩投影器的安全威胁
The Security Threat of Compressed Projectors in Large Vision-Language Models
AI总结:
本研究针对大视觉语言模型中两类视觉语言投影器的安全特性展开评估,发现压缩投影器存在重大安全漏洞,非压缩投影器安全性更稳健,为VLP选型提供了关键安全指导。
摘要:
选择合适的视觉语言投影器(VLP)对大视觉语言模型(LVLM)的成功训练至关重要。主流VLP大致可分为压缩投影器和非压缩投影器两类,二者在性能和计算效率上各有独特优势,但它们的安全影响尚未得到彻底研究。我们的综合评估揭示了二者安全特性的显著差异:压缩投影器存在重大漏洞,攻击者即使仅掌握极少结构信息也能成功攻破LVLM。与之形成鲜明对比的是,非压缩投影器表现出稳健的安全属性,不会引入额外漏洞。这些发现为研究人员选择最优VLP、提升视觉语言模型的安全性和可靠性提供了关键指导。代码可在https://github.com/btzyd/TCP获取。
英文摘要:
The choice of a suitable visual language projector (VLP) is critical to the successful training of large visual language models (LVLMs). Mainstream VLPs can be broadly categorized into compressed and uncompressed projectors, and each offers distinct advantages in performance and computational efficiency. However, their security implications have not been thoroughly examined. Our comprehensive evaluation reveals significant differences in their security profiles: compressed projectors exhibit substantial vulnerabilities, allowing adversaries to successfully compromise LVLMs even with minimal knowledge of structure information. In stark contrast, uncompressed projectors demonstrate robust security properties and do not introduce additional vulnerabilities. These findings provide critical guidance for researchers in selecting optimal VLPs that enhance the security and reliability of visual language models. The code is available at https://github.com/btzyd/TCP.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。