Dyslexify: 一种针对CLIP中印刷攻击的机制性防御
Dyslexify: A Mechanistic Defense Against Typographic Attacks in CLIP
AI总结：
Dyslexify通过消融CLIP中的印刷电路，有效防御印刷攻击，提升性能并保持应用安全性。
摘要：
印刷攻击通过向图像中注入文本，利用多模态系统导致目标误分类、恶意内容生成，甚至视觉语言模型劫持。在本工作中，我们分析了CLIP视觉编码器在印刷攻击下的行为，发现模型后半部分的特定注意力头会因果性地提取并传递印刷信息到cls token。基于这些见解，我们引入Dyslexify——一种通过选择性消融印刷电路（由注意力头组成）来防御CLIP模型对抗印刷攻击的方法。无需微调，Dyslexify在印刷变体的ImageNet-100上将性能提升高达22.06%，同时在标准ImageNet-100上的准确率下降小于1%。我们还展示了其在皮肤病变诊断医疗基础模型中的实用性。值得注意的是，我们的无训练方法在当前最先进的印刷防御方法（依赖微调）中仍具有竞争力。为此，我们发布了Dyslexic CLIP模型家族，这些模型在对抗印刷攻击方面显著更稳健。这些模型适用于广泛的安全关键应用，其中文本基于操控的风险超过了文本识别的效用。
英文摘要：
Typographic attacks exploit multi-modal systems by injecting text into images, leading to targeted misclassifications, malicious content generation and even Vision-Language Model jailbreaks. In this work, we analyze how CLIP vision encoders behave under typographic attacks, locating specialized attention heads in the latter half of the model's layers that causally extract and transmit typographic information to the cls token. Building on these insights, we introduce Dyslexify - a method to defend CLIP models against typographic attacks by selectively ablating a typographic circuit, consisting of attention heads. Without requiring finetuning, dyslexify improves performance by up to 22.06% on a typographic variant of ImageNet-100, while reducing standard ImageNet-100 accuracy by less than 1%, and demonstrate its utility in a medical foundation model for skin lesion diagnosis. Notably, our training-free approach remains competitive with current state-of-the-art typographic defenses that rely on finetuning. To this end, we release a family of dyslexic CLIP models which are significantly more robust against typographic attacks. These models serve as suitable drop-in replacements for a broad range of safety-critical applications, where the risks of text-based manipulation outweigh the utility of text recognition.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
