SRD：基于强化学习的语义扰动用于视觉语言模型（VLMs）的后门防御
SRD: Reinforcement-Learned Semantic Perturbation for Backdoor Defense in VLMs
AI总结：
针对视觉语言模型（VLMs）易受后门攻击的问题，提出基于强化学习的语义奖励防御（SRD）框架，通过对图像敏感区域施加离散扰动混淆注意力，以语义保真度分数优化策略，有效降低两类攻击的ASR且对干净输入影响小。
摘要：
视觉语言模型（VLMs）在图像 captioning 任务中取得了显著进展，但最近的研究发现它们容易受到后门攻击。攻击者可以在推理期间向数据中注入无法检测的扰动，触发异常行为并生成恶意 caption。由于触发信号的隐蔽性和跨模态传播性，这些攻击特别难以检测和防御。本文通过分析现有攻击模式确定了两个关键漏洞：（1）模型对输入图像的某些区域表现出异常的注意力集中；（2）后门攻击通常会导致语义漂移和句子不连贯。基于这些见解，我们提出了语义奖励防御（SRD），这是一种强化学习框架，无需任何触发模式的先验知识即可缓解后门行为。SRD 通过深度 Q 网络策略学习对图像输入的敏感上下文区域应用离散扰动，旨在混淆注意力并破坏恶意路径的激活。为了指导策略优化，我们设计了一种名为语义保真度分数的奖励信号，该信号联合评估生成 caption 的语义一致性和语言流畅性，鼓励智能体实现稳健且忠实的输出。SRD 提供了一种与触发无关、策略可解释的防御范式，有效缓解局部（TrojVLM）和全局（Shadowcast）后门攻击，分别将攻击成功率（ASR）降至 3.6% 和 5.6%，同时对干净输入的平均 CIDEr 下降不到 15%。我们的代码可在 https://github.com/Ciconey/SRD.git 获取。
英文摘要：
Visual language models (VLMs) have made significant progress in image captioning tasks, yet recent studies have found they are vulnerable to backdoor attacks. Attackers can inject undetectable perturbations into the data during inference, triggering abnormal behavior and generating malicious captions. These attacks are particularly challenging to detect and defend against due to the stealthiness and cross-modal propagation of the trigger signals. In this paper, we identify two key vulnerabilities by analyzing existing attack patterns: (1) the model exhibits abnormal attention concentration on certain regions of the input image, and (2) backdoor attacks often induce semantic drift and sentence incoherence. Based on these insights, we propose Semantic Reward Defense (SRD), a reinforcement learning framework that mitigates backdoor behavior without requiring any prior knowledge of trigger patterns. SRD learns to apply discrete perturbations to sensitive contextual regions of image inputs via a deep Q-network policy, aiming to confuse attention and disrupt the activation of malicious paths. To guide policy optimization, we design a reward signal named semantic fidelity score, which jointly assesses the semantic consistency and linguistic fluency of the generated captions, encouraging the agent to achieve a robust yet faithful output. SRD offers a trigger-agnostic, policy-interpretable defense paradigm that effectively mitigates local (TrojVLM) and global (Shadowcast) backdoor attacks, reducing ASR to 3.6% and 5.6% respectively, with less than 15% average CIDEr drop on the clean inputs. Our codes can be found at https://github.com/Ciconey/SRD.git.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
