In this work we demonstrate the vulnerability of vision transformers (ViTs) to gradient-based inversion attacks. During this attack, the original data batch is reconstructed given model weights and the corresponding gradients. We introduce a method, named GradViT, that optimizes random noise into naturally looking images via an iterative process. The optimization objective consists of (i) a loss on matching the gradients, (ii) image prior in the form of distance to batch-normalization statistics of a pretrained CNN model, and (iii) a total variation regularization on patches to guide correct recovery locations. We propose a unique loss scheduling function to overcome local minima during optimization. We evaluate GadViT on ImageNet1K and MS-Celeb-1M datasets, and observe unprecedentedly high fidelity and closeness to the original (hidden) data. During the analysis we find that vision transformers are significantly more vulnerable than previously studied CNNs due to the presence of the attention mechanism. Our method demonstrates new state-of-the-art results for gradient inversion in both qualitative and quantitative metrics. Project page at https://gradvit.github.io/.
Sparse Model Inversion: Efficient Inversion of Vision Transformers for Data-Free Applications
Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China;College of Computing \& Data Science, Nanyang Technological University, Singapore;School of Cyber Science;Technology, Sun Yat-sen University, Shenzhen, China;JD Explore Academy, China;University of Maryland, College Park, USA
Physics informed Transformer-VAE for biophysical parameter estimation: PROSAIL model inversion in Sentinel-2 imagery
African Institute for Mathematical Sciences (AIMS), Accra, Ghana(非洲数学科学研究所(AIMS));African Institute for Mathematical Sciences (AIMS), Cape Town, South Africa(非洲数学科学研究所(AIMS))
Joint rotational invariance and adversarial training of a dual-stream Transformer yields state of the art Brain-Score for Area V4
The Center for Brains, Minds and Machines(大脑、心智与机器中心);Massachusetts Institute of Technology(麻省理工学院)
现代在大脑评分竞赛中获得高分的视觉模型并不源于Vision Transformers。然而,在本文中,我们通过展示双流Transformer,CrossViT $~\textit{a la}$ Chen et al.(2021),在联合旋转不变和对抗优化程序下,在2022年总Brain-Score竞赛中获得第二名,提供了反对Vision Transformers(ViT)在感知上与人类视觉表示不一致的意外趋势的证据(Schrimpf et al.,2020 b)在所有视觉类别中平均,并且在比赛时在V4区域的最高可解释方差中排名第一。此外,我们目前的基于变换器的模型还实现了区域V4、IT和行为的更大的可解释方差,而不是生物启发的CNN(ResNet 50),其集成了类似于额叶V1的计算模块(Dapello等人,2020年)。为了评估优化方案对CrossViT架构的贡献,我们对不同优化的CrossViT的对抗鲁棒性,常见的腐败基准,中腹部刺激解释和特征反转进行了几次额外的实验。与我们最初的预期,我们的家庭的结果提供了临时支持$\textit{"All roads lead to Rome"}$参数通过联合优化规则,即使是非生物学动机的模型,如Vision Transformers的视觉。代码可在https://github.com/williamberrios/BrainScore-T…
SatelliteCalculator: A Multi-Task Vision Foundation Model for Quantitative Remote Sensing Inversion
Faculty of Computer Science and Information Technology, Universiti Malaya(马来亚大学计算机科学与信息技术学院);Faculty of Information Engineering and Automation, Kunming University of Science and Technology(昆明理工大学信息工程与自动化学院)
DREMnet: An Interpretable Denoising Framework for Semi-Airborne Transient Electromagnetic Signal
Key Laboratory of Earth Exploration and Information Techniques, Ministry of Education, College of Geophysics, Chengdu University of Technology(地球探测与信息处理重点实验室,教育部,成都理工大学地球物理学院)
Masked Jigsaw Puzzle: A Versatile Position Embedding for Vision Transformers
University of Pisa(比萨大学);University of Trento(特伦托大学);Tencent AI Lab(腾讯人工智能实验室);Beijing Jiaotong University(北京交通大学);University of Modena and Reggio Emilia(摩德纳-雷焦艾米利亚大学)