通过细粒度细节靶向推动大视觉-语言模型攻击的前沿
Pushing the Frontier of Black-Box LVLM Attacks via Fine-Grained Detail Targeting
AI总结：
通过细粒度细节靶向改进M-Attack，显著提升黑盒攻击效果，成功率达100%
摘要：
针对大视觉-语言模型（LVLM）的黑盒对抗攻击具有挑战性，因为缺乏梯度和复杂的多模态边界。尽管先前最先进的基于转移的方法如M-Attack通过源图像和目标图像之间的局部裁剪匹配表现良好，但我们发现这会诱导高方差、几乎正交的梯度，违反了相干的局部对齐并破坏了优化。我们将其归因于（i）ViT翻译敏感性导致尖峰状梯度，以及（ii）源和目标裁剪之间的结构不对称性。我们重新公式化局部匹配为源转换和目标语义之间的不对称期望，并构建了M-Attack的梯度去噪升级。在源侧，多裁剪对齐（MCA）在每次迭代中平均多个独立采样的局部视图的梯度以减少方差。在目标侧，辅助目标对齐（ATA）用一个来自语义相关分布的小辅助集取代了激进的目标增强，产生更平滑、方差更低的目标流形。我们进一步将动量重新解释为补丁动量，回放历史裁剪梯度；结合改进的补丁大小集合（PE+），这加强了可转移的方向。这些模块共同构成了M-Attack-V2，这是M-Attack的简单、模块化的增强版本，显著提高了基于转移的黑盒攻击效果：将Claude-4.0的成功率从8%提升到30%，Gemini-2.5-Pro从83%提升到97%，GPT-5从98%提升到100%，优于先前的黑盒LVLM攻击。代码和数据可在：https://github.com/vila-lab/M-Attack-V2上公开获取。
英文摘要：
Black-box adversarial attacks on Large Vision-Language Models (LVLMs) are challenging due to missing gradients and complex multimodal boundaries. While prior state-of-the-art transfer-based approaches like M-Attack perform well using local crop-level matching between source and target images, we find this induces high-variance, nearly orthogonal gradients across iterations, violating coherent local alignment and destabilizing optimization. We attribute this to (i) ViT translation sensitivity that yields spike-like gradients and (ii) structural asymmetry between source and target crops. We reformulate local matching as an asymmetric expectation over source transformations and target semantics, and build a gradient-denoising upgrade to M-Attack. On the source side, Multi-Crop Alignment (MCA) averages gradients from multiple independently sampled local views per iteration to reduce variance. On the target side, Auxiliary Target Alignment (ATA) replaces aggressive target augmentation with a small auxiliary set from a semantically correlated distribution, producing a smoother, lower-variance target manifold. We further reinterpret momentum as Patch Momentum, replaying historical crop gradients; combined with a refined patch-size ensemble (PE+), this strengthens transferable directions. Together these modules form M-Attack-V2, a simple, modular enhancement over M-Attack that substantially improves transfer-based black-box attacks on frontier LVLMs: boosting success rates on Claude-4.0 from 8% to 30%, Gemini-2.5-Pro from 83% to 97%, and GPT-5 from 98% to 100%, outperforming prior black-box LVLM attacks. Code and data are publicly available at: https://github.com/vila-lab/M-Attack-V2.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
