FoCLIP：一种面向基于CLIP的图像篡改与检测的特征空间错位框架
FoCLIP: A Feature-Space Misalignment Framework for CLIP-Based Image Manipulation and Detection
AI总结：
本文提出FoCLIP框架，通过特征对齐、分数分布平衡和像素保护正则化构造欺骗样本以操纵CLIPscore，并基于灰度转换导致的特征退化现象设计颜色通道敏感性驱动的篡改检测机制，在基准上达到91%准确率。
摘要：
基于CLIP模型良好的对齐特性，使其能够被有效应用于诸如CLIPscore这类被广泛采用的图像质量评估指标。然而，这种基于CLIP的指标因其精细的多模态对齐而存在脆弱性。在本文中，我们提出了FoCLIP，一种用于欺骗基于CLIP的图像质量指标的特征空间错位框架。基于随机梯度下降技术，FoCLIP整合了三个关键组件来构造欺骗样本：以特征对齐作为核心模块来缩小图像-文本模态差距，分数分布平衡模块和像素保护正则化，它们共同优化CLIPscore性能与图像质量之间的多模态输出均衡。这种设计可以被用于在多样化的输入提示下最大化CLIPscore预测，尽管从人类感知角度来看，这些图像与对应的对抗提示之间要么表现出视觉不可识别性，要么表现出语义不一致性。在十个艺术杰作提示和ImageNet子集上的实验表明，优化后的图像能够在保持高视觉保真度的同时实现CLIPscore的显著提升。此外，我们发现灰度转换会导致欺骗图像出现显著的特征退化，表现出明显的CLIPscore下降，同时保持与原始图像的统计一致性。受这一现象启发，我们提出了一种颜色通道敏感性驱动的篡改检测机制，在标准基准上达到了91%的准确率。总之，本工作为基于CLIP的多模态系统中的特征错位及其相应的防御方法建立了一条实用路径。
英文摘要：
The well-aligned attribute of CLIP-based models enables its effective application like CLIPscore as a widely adopted image quality assessment metric. However, such a CLIP-based metric is vulnerable for its delicate multimodal alignment. In this work, we propose FoCLIP, a feature-space misalignment framework for fooling CLIP-based image quality metric. Based on the stochastic gradient descent technique, FoCLIP integrates three key components to construct fooling examples: feature alignment as the core module to reduce image-text modality gaps, the score distribution balance module and pixel-guard regularization, which collectively optimize multimodal output equilibrium between CLIPscore performance and image quality. Such a design can be engineered to maximize the CLIPscore predictions across diverse input prompts, despite exhibiting either visual unrecognizability or semantic incongruence with the corresponding adversarial prompts from human perceptual perspectives. Experiments on ten artistic masterpiece prompts and ImageNet subsets demonstrate that optimized images can achieve significant improvement in CLIPscore while preserving high visual fidelity. In addition, we found that grayscale conversion induces significant feature degradation in fooling images, exhibiting noticeable CLIPscore reduction while preserving statistical consistency with original images. Inspired by this phenomenon, we propose a color channel sensitivity-driven tampering detection mechanism that achieves 91% accuracy on standard benchmarks. In conclusion, this work establishes a practical pathway for feature misalignment in CLIP-based multimodal systems and the corresponding defense method.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
