VIP：通过对视觉语言模型的对抗攻击实现视觉信息保护
VIP: Visual Information Protection through Adversarial Attacks on Vision-Language Models
AI总结：
本研究将视觉语言模型的隐私保护构建为对抗攻击问题，提出VIP策略选择性地隐藏图像指定感兴趣区域的信息，有效阻止模型访问敏感内容，同时保持图像其余部分的语义完整性。
摘要：
近年来，能够同时处理文本和视觉输入的视觉语言模型（VLMs）的开发取得了显著进展。这些模型展现出了令人印象深刻的性能，从而在各种应用中被广泛采用。然而，这种广泛使用引发了关于用户隐私的严重担忧，特别是当模型无意中处理或暴露私人视觉信息时。在本研究中，我们将VLMs中的隐私保护构建为一个对抗攻击问题。我们提出了一种新颖的攻击策略，该策略选择性地隐藏图像中指定感兴趣区域（ROIs）内的信息，有效地阻止VLMs访问敏感内容，同时保留图像其余部分的语义完整性。与通常会破坏整个图像的传统对抗攻击不同，我们的方法在未掩码区域保持了高度的一致性。在三个最先进的VLMs（即LLaVA、Instruct-BLIP和BLIP2-T5）上的实验结果表明，目标ROIs的检测率降低了高达98%，同时保持了全局图像语义的完整性，这一点通过干净输出和对抗输出之间的高相似度得分得到了证实。我们相信这项工作有助于更加注重隐私的多模态模型使用，并为进一步研究提供了实用工具，源代码已公开于：https://github.com/hbrachemi/Vlm_defense-attack。
英文摘要：
Recent years have witnessed remarkable progress in developing Vision-Language Models (VLMs) capable of processing both textual and visual inputs. These models have demonstrated impressive performance, leading to their widespread adoption in various applications. However, this widespread raises serious concerns regarding user privacy, particularly when models inadvertently process or expose private visual information. In this work, we frame the preservation of privacy in VLMs as an adversarial attack problem. We propose a novel attack strategy that selectively conceals information within designated Region Of Interests (ROIs) in an image, effectively preventing VLMs from accessing sensitive content while preserving the semantic integrity of the remaining image. Unlike conventional adversarial attacks that often disrupt the entire image, our method maintains high coherence in unmasked areas. Experimental results across three state-of-the-art VLMs namely LLaVA, Instruct-BLIP, and BLIP2-T5 demonstrate up to 98% reduction in detecting targeted ROIs, while maintaining global image semantics intact, as confirmed by high similarity scores between clean and adversarial outputs. We believe that this work contributes to a more privacy conscious use of multimodal models and offers a practical tool for further research, with the source code publicly available at: https://github.com/hbrachemi/Vlm_defense-attack.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
