arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.07555cs.CV
收藏

对风格迁移的抗性是否等同于全局形状偏好?测量网络对全局形状构型的敏感性

Does resistance to style-transfer equal Global Shape Bias? Measuring network sensitivity to global shape configuration

School of Computer Science(计算机学院);Carnegie Mellon University(卡内基梅隆大学)

Ziqi Wen, Tianqin Li, Zhi Jing, Tai Sing Lee

AI总结:

本文提出 DiST 测试台直接测量网络对全局形状构型的敏感性,发现抗风格迁移不等于全局形状偏好,且自监督 ViT 更具全局结构敏感性。

摘要:

众所周知,深度学习模型表现出强烈的纹理偏好,而人类在物体识别中往往高度依赖全局形状结构。当前用于评估模型全局形状偏好的基准是一组风格迁移图像,其假设是:模型对风格迁移攻击的抗性与其全局结构敏感性的发展相关。在本研究中,我们表明,使用风格迁移图像训练的网络确实学会了忽略风格,但其形状偏好主要来自局部细节。我们提供了一个 Disrupted Structure Testbench(DiST,结构破坏测试台),用于直接测量全局结构敏感性。我们的测试包含来自 ImageNet-1K 的2400张原始图像,每张图像都配有两张通过纹理合成程序在保留其纹理的同时破坏原始图像全局形状的图像。我们发现:(1)在以往线索冲突数据集上表现良好的模型,在所提出的 DiST 上表现并不好;(2)监督训练的 Vision Transformer(ViT)会从位置编码中丢失全局空间信息,导致其在 DiST 上相比 Convolutional Neural Networks(CNNs)没有显著优势;而自监督学习方法,尤其是掩码自编码器,显著提升了 ViT 的全局结构敏感性。(3)提升全局结构敏感性与对风格迁移的抗性是正交的,这表明全局形状结构与局部纹理细节之间并非非此即彼的关系。使用 DiST 图像和风格迁移图像进行训练具有互补性,可以结合起来共同训练网络,以增强全局形状敏感性和局部特征的鲁棒性。我们的代码将托管在 GitHub:https://github.com/leelabcnbc/DiST

英文摘要:

Deep learning models are known to exhibit a strong texture bias, while human tends to rely heavily on global shape structure for object recognition. The current benchmark for evaluating a model's global shape bias is a set of style-transferred images with the assumption that resistance to the attack of style transfer is related to the development of global structure sensitivity in the model. In this work, we show that networks trained with style-transfer images indeed learn to ignore style, but its shape bias arises primarily from local detail. We provide a \textbf{Disrupted Structure Testbench (DiST)} as a direct measurement of global structure sensitivity. Our test includes 2400 original images from ImageNet-1K, each of which is accompanied by two images with the global shapes of the original image disrupted while preserving its texture via the texture synthesis program. We found that \textcolor{black}{(1) models that performed well on the previous cue-conflict dataset do not fare well in the proposed DiST; (2) the supervised trained Vision Transformer (ViT) lose its global spatial information from positional embedding, leading to no significant advantages over Convolutional Neural Networks (CNNs) on DiST. While self-supervised learning methods, especially mask autoencoder significantly improves the global structure sensitivity of ViT. (3) Improving the global structure sensitivity is orthogonal to resistance to style-transfer, indicating that the relationship between global shape structure and local texture detail is not an either/or relationship. Training with DiST images and style-transferred images are complementary, and can be combined to train network together to enhance the global shape sensitivity and robustness of local features.} Our code will be hosted in github: https://github.com/leelabcnbc/DiST

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑