In this work, we introduce an unconditional video generative model, InMoDeGAN, targeted to (a) generate high quality videos, as well as to (b) allow for interpretation of the latent space. For the latter, we place emphasis on interpreting and manipulating motion. Towards this, we decompose motion into semantic sub-spaces, which allow for control of generated samples. We design the architecture of InMoDeGAN-generator in accordance to proposed Linear Motion Decomposition, which carries the assumption that motion can be represented by a dictionary, with related vectors forming an orthogonal basis in the latent space. Each vector in the basis represents a semantic sub-space. In addition, a Temporal Pyramid Discriminator analyzes videos at different temporal resolutions. Extensive quantitative and qualitative analysis shows that our model systematically and significantly outperforms state-of-the-art methods on the VoxCeleb2-mini and BAIR-robot datasets w.r.t. video quality related to (a). Towards (b) we present experimental results, confirming that decomposed sub-spaces are interpretable and moreover, generated motion is controllable.
Nanjing University of Science and Technology（南京理工大学）；Peking University（北京大学）
Despite the success of deep learning across various domains, it remains vulnerable to adversarial attacks. Although many existing adversarial attack methods achieve high success rates, they typically rely on $\ell_{p}$-norm perturbation constraints, which do not align with human perceptual capabilities. Consequently, researchers have shifted their focus toward generating natural, unrestricted adversarial examples (UAEs). GAN-based approaches suffer from inherent limitations, such as poor image quality due to instability and mode collapse. Meanwhile, diffusion models have been employed for UAE …
This paper describes a simple technique to analyze Generative Adversarial Networks (GANs) and create interpretable controls for image synthesis, such as change of viewpoint, aging, lighting, and time of day. We identify important latent directions based on Principal Components Analysis (PCA) applied either in latent space or feature space. Then, we show that a large number of interpretable controls can be defined by layer-wise perturbation along the principal directions. Moreover, we show that BigGAN can be controlled with layer-wise inputs in a StyleGAN-like manner. We show results on differe…
The effects of adversarial training on semantic segmentation networks has not been thoroughly explored. While previous work has shown that adversarially-trained image classifiers can be used to perform image synthesis, we have yet to understand how best to leverage an adversarially-trained segmentation network to do the same. Using a simple optimizer, we demonstrate that adversarially-trained semantic segmentation networks can be used to perform image inpainting and generation. Our experiments demonstrate that adversarially-trained segmentation networks are more robust and indeed exhibit perce…
Georgia Institute of Technology（佐治亚理工学院）；H. Milton Stewart School of Industrial and Systems Engineering, Georgia Tech（佐治亚理工学院H.米尔顿·斯图尔特工业与系统工程学院）；School of Computational Science and Engineering, Georgia Tech（佐治亚理工学院计算科学与工程学院）；Department of Industrial Engineering and Engineering Management, National Tsing Hua University（国立清华大学工业工程与工程管理系）；Marcus Heart Valve Center, Piedmont Heart Institute（皮埃蒙特心脏研究所马库斯心脏瓣膜中心）；School of Materials Science and Engineering, Georgia Tech（佐治亚理工学院材料科学与工程学院）
EPFL（洛桑联邦理工学院）；Technical University of Munich（慕尼黑工业大学）；Boğaziçi University（博阿齐奇大学）
Recent research has shown that it is possible to find interpretable directions in the latent spaces of pre-trained Generative Adversarial Networks (GANs). These directions enable controllable image generation and support a wide range of semantic editing operations, such as zoom or rotation. The discovery of such directions is often done in a supervised or semi-supervised manner and requires manual annotations which limits their use in practice. In comparison, unsupervised discovery allows finding subtle directions that are difficult to detect a priori. In this work, we propose a contrastive le…
Learning disentangled and interpretable representations is an important step towards accomplishing comprehensive data representations on the manifold. In this paper, we propose a novel representation learning algorithm which combines the inference abilities of Variational Autoencoders (VAE) with the generalization capability of Generative Adversarial Networks (GAN). The proposed model, called InfoVAEGAN, consists of three networks~: Encoder, Generator and Discriminator. InfoVAEGAN aims to jointly learn discrete and continuous interpretable representations in an unsupervised manner by using two…
Towards Interpretable Deep Neural Networks by Leveraging Adversarial Examples
Tsinghua National Lab for Information Science and Technology（清华大学信息科学与技术国家实验室）；State Key Lab of Intelligent Technology and Systems（智能技术与系统国家重点实验室）；Center for Bio-Inspired Computing Research（类脑计算研究中心）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）
Zhejiang University of Technology（浙江工业大学）；Zhejiang University（浙江大学）；Hangzhou Dianzi University（杭州电子科技大学）；Zhejiang Police College（浙江警察学院）
Recent years have witnessed unprecedented success achieved by deep learning models in the field of computer vision. However, their vulnerability towards carefully crafted adversarial examples has also attracted the increasing attention of researchers. Motivated by the observation that adversarial examples are due to the non-robust feature learned from the original dataset by models, we propose the concepts of salient feature(SF) and trivial feature(TF). The former represents the class-related feature, while the latter is usually adopted to mislead the model. We extract these two features with …
Training-Free and Interpretable Hateful Video Detection via Multi-stage Adversarial Reasoning
Multimodal Intelligence Lab, Department of Computer Science, University of Exeter, United Kingdom（埃克塞特大学计算机科学系多模态智能实验室）；Institute for Analytics and Data Science, University of Essex, United Kingdom（埃塞克斯大学分析与数据科学研究所）
Deep learning methods have gained increased attention in various applications due to their outstanding performance. For exploring how this high performance relates to the proper use of data artifacts and the accurate problem formulation of a given task, interpretation models have become a crucial component in developing deep learning-based systems. Interpretation models enable the understanding of the inner workings of deep learning models and offer a sense of security in detecting the misuse of artifacts in the input data. Similar to prediction models, interpretation models are also susceptib…
UnICLAM:Contrastive Representation Learning with Adversarial Masking for Unified and Interpretable Medical Vision Question Answering
Zhejiang University（浙江大学）；University of Illinois Urbana–Champaign Institute（伊利诺伊大学厄巴纳-香槟分校研究院）；Sixth Medical Center of Chinese PLA General Hospital（中国人民解放军总医院第六医学中心）
OTI: A Model-free and Visually Interpretable Measure of Image Attackability
Faculty of Science and Technology, University of Macau（澳门大学科技学院）；Chongqing Key Laboratory of Image Cognition, Chongqing University of Posts and Telecommunications（重庆邮电大学图像认知重点实验室）
Breaking the Illusion of Security via Interpretation: Interpretable Vision Transformer Systems under Attack
Department of Computer Science and Engineering, Sungkyunkwan University（计算机科学与工程系，全州大学）；Department of Computer Science, Loyola University（计算机科学系，洛伊拉大学）
ZHAW Datalab & School of Engineering（苏黎世应用科技大学数据实验室与工程学院）；Institute of Neural Information Processing, Ulm University（乌尔姆大学神经信息处理研究所）
The existence of adversarial attacks on convolutional neural networks (CNN) questions the fitness of such models for serious applications. The attacks manipulate an input image such that misclassification is evoked while still looking normal to a human observer -- they are thus not easily detectable. In a different context, backpropagated activations of CNN hidden layers -- "feature responses" to a given input -- have been helpful to visualize for a human "debugger" what the CNN "looks at" while computing its output. In this work, we propose a novel detection method for adversarial examples to…
We propose a new adversarial attack to Deep Neural Networks for image classification. Different from most existing attacks that directly perturb input pixels, our attack focuses on perturbing abstract features, more specifically, features that denote styles, including interpretable styles such as vivid colors and sharp outlines, and uninterpretable ones. It induces model misclassfication by injecting imperceptible style changes through an optimization procedure. We show that our attack can generate adversarial samples that are more natural-looking than the state-of-the-art unbounded attacks. T…
Learning useful representations of complex data has been the subject of extensive research for many years. With the diffusion of Deep Neural Networks, Variational Autoencoders have gained lots of attention since they provide an explicit model of the data distribution based on an encoder/decoder architecture which is able to both generate images and encode them in a low-dimensional subspace. However, the latent space is not easily interpretable and the generation capabilities show some limitations since images typically look blurry and lack details. In this paper, we propose the Introspective V…
Institute of Machine Learning in Biomedical Imaging, Helmholtz Munich（亥姆霍兹慕尼黑生物医学成像机器学习研究所）；Helmholtz AI, Helmholtz Munich（亥姆霍兹慕尼黑人工智能中心）；School of Computation, Information and Technology, Technical University of Munich（慕尼黑工业大学计算、信息与技术学院）；School of Biomedical Engineering and Imaging Sciences, King's College London（伦敦国王学院生物医学工程与影像科学学院）
