面向大视觉语言模型中压缩触发隐蔽故障的特征感知令牌攻击
Feature-Aware Token Attack for Compression-Triggered Stealthy Failures in Large Vision-Language Models
浏览论文内容
中文总结 AI 辅助
提出特征感知令牌攻击（FATA），通过注意力抑制与特征保留，在压缩推理中引发隐蔽故障，同时保持全令牌正确性，实验显示高成功率与低检测率。
中文摘要 AI 辅助
视觉令牌压缩提高了大视觉语言模型的效率，但可能暴露全令牌评估所遗漏的故障。我们研究了对抗性图像，这些图像在全令牌推理下保持正确性，但在压缩后却引发错误，即使两种推理路径在干净图像上均成功。创建此类故障具有挑战性，因为扰动令牌重要性也可能损害全令牌推理所需的视觉内容。我们提出了特征感知令牌攻击（FATA），该方法在固定的显著干净图像令牌集上，将注意力抑制与基于余弦的特征保留相结合。在主要的LLaVA-1.5-7B设置中，FATA仅使用视觉编码器梯度，无需访问部署的压缩器、令牌预算或下游任务。在受控重建协议下，跨四个视觉依赖任务子集和四个压缩器，FATA实现了SR=96.3%的全令牌准确率保留和CBR=22.1%的条件盲化，而CAA分别为89.8%和15.7%。消融研究支持了这两个目标在平衡压缩路径故障与全令牌保留方面的作用。在三个评估检测器上，FATA在5%假阳性率下也拥有四种攻击中最低的检测率。这些发现促使我们在全令牌和压缩推理中联合评估对抗鲁棒性。
英文摘要
Visual-token compression improves the efficiency of large vision-language models, but can expose failures that full-token evaluation misses. We study adversarial images that preserve full-token correctness yet induce errors after compression, even when both inference paths succeed on the clean image. Creating such failures is challenging because perturbing token importance can also damage the visual content needed for full-token inference. We propose Feature-Aware Token Attack (FATA), which couples attention suppression with cosine-based feature preservation on a fixed set of salient clean-image tokens. In the primary LLaVA-1.5-7B setting, FATA uses only vision-encoder gradients, without access to the deployed compressor, token budget, or downstream task. Across four visually dependent task subsets and four compressors under a controlled reconstruction protocol, FATA achieves SR = 96.3% full-token accuracy retention and CBR = 22.1% conditional blinding, compared with 89.8% and 15.7% for CAA. Ablations support the role of both objectives in balancing compressed-path failure against full-token preservation. FATA also has the lowest measured detection rate among four attacks across three evaluated detectors at a 5% false-positive rate. These findings motivate assessing adversarial robustness jointly across full-token and compressed inference.
发表机构
- Beijing University of Posts and Telecommunications（北京邮电大学）
- Jiangnan University（江南大学）
- Nanyang Technological University（南洋理工大学）
- Chongqing University of Posts and Telecommunications（重庆邮电大学）
机构由 AI 辅助整理，请以论文原文为准。
