Temporal Consistency Objectives Regularize the Learning of Disentangled Representations
IMT School for Advanced Studies Lucca（IMT卢卡高级研究学院）；School of Engineering, University of Edinburgh（爱丁堡大学工程学院）
Gabriele Valvano, Agisilaos Chartsias, Andrea Leo, Sotirios A. Tsaftaris
备注：
9 pages, 4 figures (1 .gif), 1 table
发表信息：
Domain Adaptation and Representation Transfer and Medical Image Learning with Less Labels and Imperfect Data. DART 2019, MIL3ID 2019. Lecture Notes in Computer Science, vol 11795 pp 11-19. Springer, Cham
There has been an increasing focus in learning interpretable feature representations, particularly in applications such as medical image analysis that require explainability, whilst relying less on annotated data (since annotations can be tedious and costly). Here we build on recent innovations in style-content representations to learn anatomy, imaging characteristics (appearance) and temporal correlations. By introducing a self-supervised objective of predicting future cardiac phases we improve disentanglement. We propose a temporal transformer architecture that given an image conditioned on phase difference, it predicts a future frame. This forces the anatomical decomposition to be consistent with the temporal cardiac contraction in cine MRI and to have semantic meaning with less need for annotations. We demonstrate that using this regularization, we achieve competitive results and improve semi-supervised segmentation, especially when very few labelled data are available. Specifically, we show Dice increase of up to 19\% and 7\% compared to supervised and semi-supervised approaches respectively on the ACDC dataset. Code is available at: https://github.com/gvalvano/sdtnet .
The University of Hong Kong（香港大学）；Tencent AI Lab（腾讯AI实验室）；ETH Zürich（苏黎世联邦理工学院）；Tencent Data Platform（腾讯数据平台）
Image virtual try-on replaces the clothes on a person image with a desired in-shop clothes image. It is challenging because the person and the in-shop clothes are unpaired. Existing methods formulate virtual try-on as either in-painting or cycle consistency. Both of these two formulations encourage the generation networks to reconstruct the input image in a self-supervised manner. However, existing methods do not differentiate clothing and non-clothing regions. A straight-forward generation impedes virtual try-on quality because of the heavily coupled image contents. In this paper, we propose …
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Computed medical imaging systems require a computational reconstruction procedure for image formation. In order to recover a useful estimate of the object to-be-imaged when the recorded measurements are incomplete, prior knowledge about the nature of object must be utilized. In order to improve the conditioning of an ill-posed imaging inverse problem, deep learning approaches are being actively investigated for better representing object priors and constraints. This work proposes to use a style-based generative adversarial network (StyleGAN) to constrain an image reconstruction problem in the …
Dual Consistent Constraint via Disentangled Consistency and Complementarity for Multi-view Clustering
College of Data Science and Application（数据科学与应用学院）；Inner Mongolia Autonomous Region Engineering and Technology Research Center of Big Data-Based Software Service（内蒙古自治区大数据软件服务工程技术研究中心）；Inner Mongolia Key Laboratory of Beijiang Cyberspace Security（内蒙古北疆网络空间安全重点实验室）
Image deblurring aims to restore the latent sharp images from the corresponding blurred ones. In this paper, we present an unsupervised method for domain-specific single-image deblurring based on disentangled representations. The disentanglement is achieved by splitting the content and blur features in a blurred image using content encoders and blur encoders. We enforce a KL divergence loss to regularize the distribution range of extracted blur attributes such that little content information is contained. Meanwhile, to handle the unpaired training data, a blurring branch and the cycle-consiste…
University of Southern California（南加州大学）；Harvard University（哈佛大学）；Toyota Technological Institute at Chicago（芝加哥丰田技术研究院）
We study the problem of learning disentangled representations for data across multiple domains and its applications in human retargeting. Our goal is to map an input image to an identity-invariant latent representation that captures intrinsic factors such as expressions and poses. To this end, we present a novel multi-view learning approach that leverages various data sources such as images, keypoints, and poses. Our model consists of multiple id-conditioned VAEs for different views of the data. During training, we encourage the latent embeddings to be consistent across these views. Our observ…
As recent generative models can generate photo-realistic images, people seek to understand the mechanism behind the generation process. Interpretable generation process is beneficial to various image editing applications. In this work, we propose a framework to discover interpretable directions in the latent space given arbitrary pre-trained generative adversarial networks. We propose to learn the transformation from prior one-hot vectors representing different attributes to the latent space used by pre-trained models. Furthermore, we apply a centroid loss function to improve consistency and s…
We address the problem of person re-identification (reID), that is, retrieving person images from a large dataset, given a query image of the person of interest. A key challenge is to learn person representations robust to intra-class variations, as different persons can have the same attribute and the same person's appearance looks different with viewpoint changes. Recent reID methods focus on learning discriminative features but robust to only a particular factor of variations (e.g., human pose), which requires corresponding supervisory signals (e.g., pose annotations). To tackle this proble…
From Consistency to Complementarity: Aligned and Disentangled Multi-modal Learning for Time Series Understanding and Reasoning
The Hong Kong University of Science（香港科学与技术大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）
IBM Research AI（IBM研究人工智能部门）；Yorktown Heights, NY（纽约州约克镇高地（这里注意：根据提取规则，地址信息不应提取，但原HTML中“Affiliation: Yorktown Heights, NY”可能是地址而非机构，结合上下文，正确机构应为“IBM Research AI”，另一个“Affiliation”可能是地址或重复，需确认。但根据提取规则，只提取明确的机构。原文本中明确的机构是“IBM Research AI”，“Yorktown Heights, NY”是地点，非机构。因此正确应为[{）
Disentangled representations, where the higher level data generative factors are reflected in disjoint latent dimensions, offer several benefits such as ease of deriving invariant representations, transferability to other tasks, interpretability, etc. We consider the problem of unsupervised learning of disentangled representations from large pool of unlabeled observations, and propose a variational inference based approach to infer disentangled latent factors. We introduce a regularizer on the expectation of the approximate posterior over observed data that encourages the disentanglement. We a…
Generative models that learn disentangled representations for different factors of variation in an image can be very useful for targeted data augmentation. By sampling from the disentangled latent subspace of interest, we can efficiently generate new data necessary for a particular task. Learning disentangled representations is a challenging problem, especially when certain factors of variation are difficult to label. In this paper, we introduce a novel architecture that disentangles the latent space into two complementary subspaces by using only weak supervision in form of pairwise similarity…
Institute of Advanced Research in Artificial Intelligence（高级人工智能研究所）；Helmholtz-Zentrum Dresden-Rossendorf（亥姆霍兹德累斯顿罗森多夫研究中心）；Helmholtz Institute Freiberg for Resource Technology（亥姆霍兹弗赖贝格资源技术研究所）
Deep learning algorithms have obtained great success in semantic segmentation of very high-resolution (VHR) images. Nevertheless, training these models generally requires a large amount of accurate pixel-wise annotations, which is very laborious and time-consuming to collect. To reduce the annotation burden, this paper proposes a consistency-regularized region-growing network (CRGNet) to achieve semantic segmentation of VHR images with point-level annotations. The key idea of CRGNet is to iteratively select unlabeled pixels with high confidence to expand the annotated area from the original sp…
University of California, Merced（加州大学默塞德分校）；Virginia Tech（弗吉尼亚理工学院暨州立大学）；Verisk Analytics（Verisk分析公司）；Google Cloud（谷歌云）
Image-to-image translation aims to learn the mapping between two visual domains. There are two main challenges for many applications: 1) the lack of aligned training pairs and 2) multiple possible outputs from a single input image. In this work, we present an approach based on disentangled representation for producing diverse outputs without paired training images. To achieve diversity, we propose to embed images onto two spaces: a domain-invariant content space capturing shared information across domains and a domain-specific attribute space. Our model takes the encoded content features extra…
Virginia Tech（弗吉尼亚理工学院）；Verisk Analytics（Verisk分析公司）；University of California at Merced（加州大学默塞德分校）；Peking University（北京大学）
Image-to-image translation aims to learn the mapping between two visual domains. There are two main challenges for this task: 1) lack of aligned training pairs and 2) multiple possible outputs from a single input image. In this work, we present an approach based on disentangled representation for generating diverse outputs without paired training images. To synthesize diverse outputs, we propose to embed images onto two spaces: a domain-invariant content space capturing shared information across domains and a domain-specific attribute space. Our model takes the encoded content features extract…
Cardiac segmentation is an essential step for the diagnosis of cardiovascular diseases. However, pixel-wise dense labeling is both costly and time-consuming. Scribble, as a form of sparse annotation, is more accessible than full annotations. However, it's particularly challenging to train a segmentation network with weak supervision from scribbles. To tackle this problem, we propose a new scribble-guided method for cardiac segmentation, based on the Positive-Unlabeled (PU) learning framework and global consistency regularization, and termed as ShapePU. To leverage unlabeled pixels via PU learn…
DisentTalk: Cross-lingual Talking Face Generation via Semantic Disentangled Diffusion Model
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）
Although various methods have been proposed for multi-label classification, most approaches still follow the feature learning mechanism of the single-label (multi-class) classification, namely, learning a shared image feature to classify multiple labels. However, we find this One-shared-Feature-for-Multiple-Labels (OFML) mechanism is not conducive to learning discriminative label features and makes the model non-robustness. For the first time, we mathematically prove that the inferiority of the OFML mechanism is that the optimal learned image feature cannot maintain high similarities with mult…
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；University of Salzburg（萨尔茨堡大学）
Learning maps between data samples is fundamental. Applications range from representation learning, image translation and generative modeling, to the estimation of spatial deformations. Such maps relate feature vectors, or map between feature spaces. Well-behaved maps should be regular, which can be imposed explicitly or may emanate from the data itself. We explore what induces regularity for spatial transformations, e.g., when computing image registrations. Classical optimization-based models compute maps between pairs of samples and rely on an appropriate regularizer for well-posedness. Rece…
Institute for Infocomm Research (I2R), Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局资讯通信研究院）；Nanyang Technological University（南洋理工大学）
Deep learning has achieved promising segmentation performance on 3D left atrium MR images. However, annotations for segmentation tasks are expensive, costly and difficult to obtain. In this paper, we introduce a novel hierarchical consistency regularized mean teacher framework for 3D left atrium segmentation. In each iteration, the student model is optimized by multi-scale deep supervision and hierarchical consistency regularization, concurrently. Extensive experiments have shown that our method achieves competitive performance as compared with full annotation, outperforming other state-of-the…
Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）；NAVER AI Lab（NAVER人工智能实验室）；AITRICS
Inspired by the impressive performance of recent face image editing methods, several studies have been naturally proposed to extend these methods to the face video editing task. One of the main challenges here is temporal consistency among edited frames, which is still unresolved. To this end, we propose a novel face video editing framework based on diffusion autoencoders that can successfully extract the decomposed features - for the first time as a face video editing model - of identity and motion from a given video. This modeling allows us to edit the video by simply manipulating the tempor…
Tianjin University（天津大学）；University of Cambridge（剑桥大学）；Inception Institute of Artificial Intelligence（起源人工智能研究院）；The Hong Kong Polytechnic University（香港理工大学）；Sun Yat-sen University（中山大学）
Single image dehazing is a challenging task, for which the domain shift between synthetic training data and real-world testing images usually leads to degradation of existing methods. To address this issue, we propose a novel image dehazing framework collaborating with unlabeled real data. First, we develop a disentangled image dehazing network (DID-Net), which disentangles the feature representations into three component maps, i.e. the latent haze-free image, the transmission map, and the global atmospheric light estimate, respecting the physical model of a haze process. Our DID-Net predicts …
Consistency-Regularized GAN for Few-Shot SAR Target Recognition
College of Electronics and Information Engineering, Wuyi University（威夷大学电子与信息工程学院）；College of Electronic and Information Engineering, South China University of Technology（华南理工大学电子与信息工程学院）；Department of Geography, The University of Hong Kong（香港大学地理系）；Faculty of Innovation Engineering, Macau University of Science and Technology（澳门科技大学创新工程学院）；Faculty of Computer Science and Engineering, South China University of Technology（华南理工大学计算机科学与工程学院）
Aiming at inferring 3D shapes from 2D images, 3D shape reconstruction has drawn huge attention from researchers in computer vision and deep learning communities. However, it is not practical to assume that 2D input images and their associated ground truth 3D shapes are always available during training. In this paper, we propose a framework for semi-supervised 3D reconstruction. This is realized by our introduced 2D-3D self-consistency, which aligns the predicted 3D models and the projected 2D foreground segmentation masks. Moreover, our model not only enables recovering 3D shapes with the corr…
Hierarchical Context Alignment with Disentangled Geometric and Temporal Modeling for Semantic Occupancy Prediction
Shanghai Jiao Tong University（上海交通大学）；Eastern Institute of Technology（东部技术研究所）；Ningbo Institute of Digital Twin（宁波数字孪生研究所）；University of Adelaide（阿德莱德大学）；Tokyo Institute of Technology（东京技术大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Goal-conditioned reinforcement learning is a crucial yet challenging algorithm which enables agents to achieve multiple user-specified goals when learning a set of skills in a dynamic environment. However, it typically requires millions of the environmental interactions explored by agents, which is sample-inefficient. In the paper, we propose a skill learning framework DR-GRL that aims to improve the sample efficiency and policy generalization by combining the Disentangled Representation learning and Goal-conditioned visual Reinforcement Learning. In a weakly supervised manner, we propose a Sp…
Huazhong University of Science and Technology（华中科技大学）；Peng Cheng Laboratory（鹏城实验室）
Existing deep learning-based image deraining methods have achieved promising performance for synthetic rainy images, typically rely on the pairs of sharp images and simulated rainy counterparts. However, these methods suffer from significant performance drop when facing the real rain, because of the huge gap between the simplified synthetic rain and the complex real rain. In this work, we argue that the rain generation and removal are the two sides of the same coin and should be tightly coupled. To close the loop, we propose to jointly learn real rain generation and removal procedure within a …
Xiamen University（厦门大学）；University of Chinese Academy of Sciences（中国科学院大学）；Huawei Cloud & AI（华为云与AI）
Unsupervised domain adaptation (UDA) aims to address the domain-shift problem between a labeled source domain and an unlabeled target domain. Many efforts have been made to address the mismatch between the distributions of training and testing data, but unfortunately, they ignore the task-oriented information across domains and are inflexible to perform well in complicated open-set scenarios. Many efforts have been made to eliminate the mismatch between the distributions of training and testing data by learning domain-invariant representations. However, the learned representations are usually …
Inferring geometrically consistent dense 3D scenes across a tuple of temporally consecutive images remains challenging for self-supervised monocular depth prediction pipelines. This paper explores how the increasingly popular transformer architecture, together with novel regularized loss formulations, can improve depth consistency while preserving accuracy. We propose a spatial attention module that correlates coarse depth predictions to aggregate local geometric information. A novel temporal attention mechanism further processes the local geometric information in a global context across conse…
Disentangled Fine-Grained Prototype Learning for Incomplete Image-Tabular Classification
School of Eye and Vision Sciences, University of Liverpool, U.K.（利物浦大学眼科与视觉科学学院）；Department of Cardiovascular and Metabolic Medicine, University of Liverpool, U.K.（利物浦大学心血管与代谢医学系）；School of Computer Science, University of Exeter, U.K.（埃克塞特大学计算机科学学院）；School of Computer Science and Engineering, South China University of Technology, China（华南理工大学计算机科学与工程学院）；School of Computing and Mathematical Sciences, University of Leicester, U.K.（莱斯特大学计算科学与数学科学学院）；Ningbo Institute of Industrial Technology, Chinese Academy of Sciences, China（中国科学院宁波工业技术研究所）；Bioengineering Program, Biological and Environmental Science and Engineering Division (BESE), King Abdullah University of Science and Technology (KAUST), Saudi Arabia（卡尔斯塔德大学科学与技术学院（KAUST）生物工程项目，沙特阿拉伯）
