We evaluate the out-of-distribution (OOD) detection performance of self-supervised learning (SSL) techniques with a new evaluation framework. Unlike the previous evaluation methods, the proposed framework adjusts the distance of OOD samples from the in-distribution samples. We evaluate an extensive combination of OOD detection algorithms on three different implementations of the proposed framework using simulated samples, images, and text. SSL methods consistently demonstrated the improved OOD detection performance in all evaluation settings.
In this work, we propose a zero-shot voice conversion method using speech representations trained with self-supervised learning. First, we develop a multi-task model to decompose a speech utterance into features such as linguistic content, speaker characteristics, and speaking style. To disentangle content and speaker representations, we propose a training strategy based on Siamese networks that encourages similarity between the content representations of the original and pitch-shifted audio. Next, we develop a synthesis model with pitch and duration predictors that can effectively reconstruct…
Rice University（莱斯大学）；Nvidia（英伟达）；University of Toronto（多伦多大学）；California Institute of Technology（加州理工学院）
Disentanglement learning is crucial for obtaining disentangled representations and controllable generation. Current disentanglement methods face several inherent limitations: difficulty with high-resolution images, primarily focusing on learning disentangled representations, and non-identifiability due to the unsupervised setting. To alleviate these limitations, we design new architectures and loss functions based on StyleGAN (Karras et al., 2019), for semi-supervised high-resolution disentanglement learning. We create two complex high-resolution synthetic datasets for systematic testing. We i…
The Hong Kong University of Science and Technology（香港科技大学）；Zhejiang Laboratory（之江实验室）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）
This paper proposes a Disentangled gEnerative cAusal Representation (DEAR) learning method under appropriate supervised information. Unlike existing disentanglement methods that enforce independence of the latent variables, we consider the general case where the underlying factors of interests can be causally related. We show that previous methods with independent priors fail to disentangle causally related factors even under supervision. Motivated by this finding, we propose a new disentangled learning method called DEAR that enables causal controllable generation and causal representation le…
Autodesk AI Lab（欧特克人工智能实验室）；Imperial College London（伦敦帝国理工学院）
Learning interpretable and human-controllable representations that uncover factors of variation in data remains an ongoing key challenge in representation learning. We investigate learning group-disentangled representations for groups of factors with weak supervision. Existing techniques to address this challenge merely constrain the approximate posterior by averaging over observations of a shared group. As a result, observations with a common set of variations are encoded to distinct latent representations, reducing their capacity to disentangle and generalize to downstream tasks. In contrast…
We propose using self-supervised discrete representations for the task of speech resynthesis. To generate disentangled representation, we separately extract low-bitrate representations for speech content, prosodic information, and speaker identity. This allows to synthesize speech in a controllable manner. We analyze various state-of-the-art, self-supervised representation learning methods and shed light on the advantages of each method while considering reconstruction quality and disentanglement properties. Specifically, we evaluate the F0 reconstruction, speaker identification performance (f…
EPFL（洛桑联邦理工学院）；Technical University of Munich（慕尼黑工业大学）；Boğaziçi University（博阿齐奇大学）
Recent research has shown that it is possible to find interpretable directions in the latent spaces of pre-trained Generative Adversarial Networks (GANs). These directions enable controllable image generation and support a wide range of semantic editing operations, such as zoom or rotation. The discovery of such directions is often done in a supervised or semi-supervised manner and requires manual annotations which limits their use in practice. In comparison, unsupervised discovery allows finding subtle directions that are difficult to detect a priori. In this work, we propose a contrastive le…
L’Oréal Research and Innovation（欧莱雅研发与创新中心）；LTCI, Télécom Paris, Institut Polytechnique de Paris（巴黎综合理工学院电信巴黎理工高等通信学院LTCI实验室）
While existing makeup style transfer models perform an image synthesis whose results cannot be explicitly controlled, the ability to modify makeup color continuously is a desirable property for virtual try-on applications. We propose a new formulation for the makeup style transfer task, with the objective to learn a color controllable makeup style synthesis. We introduce CA-GAN, a generative model that learns to modify the color of specific objects (e.g. lips or eyes) in the image to an arbitrary target color while preserving background. Since color labels are rare and costly to acquire, our m…
Trajectory generation and prediction are two interwoven tasks that play important roles in planner evaluation and decision making for intelligent vehicles. Most existing methods focus on one of the two and are optimized to directly output the final generated/predicted trajectories, which only contain limited information for critical scenario augmentation and safe planning. In this work, we propose a novel behavior-aware Trajectory Autoencoder (TAE) that explicitly models drivers' behavior such as aggressiveness and intention in the latent space, using semi-supervised adversarial autoencoder an…
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；ETH Zurich（苏黎世联邦理工学院）
To successfully tackle challenging manipulation tasks, autonomous agents must learn a diverse set of skills and how to combine them. Recently, self-supervised agents that set their own abstract goals by exploiting the discovered structure in the environment were shown to perform well on many different tasks. In particular, some of them were applied to learn basic manipulation skills in compositional multi-object environments. However, these methods learn skills without taking the dependencies between objects into account. Thus, the learned skills are difficult to combine in realistic environme…
Reinforcement learning (RL) is a powerful framework for learning to take actions to solve tasks. However, in many settings, an agent must winnow down the inconceivably large space of all possible tasks to the single task that it is currently being asked to solve. Can we instead constrain the space of tasks to those that are semantically meaningful? In this work, we introduce a framework for using weak supervision to automatically disentangle this semantically meaningful subspace of tasks from the enormous space of nonsensical "chaff" tasks. We show that this learned subspace enables efficient …
Fast Controllable Diffusion Models for Undersampled MRI Reconstruction
School of Electrical Engineering and Computer Science, University of Queensland（昆士兰大学电气工程与计算机科学学院）；School of Mathematics and Physics, University of Queensland（昆士兰大学数学与物理学院）
Keep It on a Leash: Controllable Pseudo-label Generation Towards Realistic Long-Tailed Semi-Supervised Learning
School of Computer Science and Engineering, Southeast University, Nanjing 210096, China（东南大学计算机科学与工程学院）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China（新一代人工智能技术及其交叉应用国家重点实验室）；School of Computing Information Sciences, Saint Francis University, Hong Kong, China（圣弗朗西斯大学计算信息科学学院）；Department of Computer Science, City University of Hong Kong, Hong Kong, China（香港城市大学计算机科学系）
Learning controllable dynamics through informative exploration
School of Science（科学与技术学院）；University of New England Australia（新英格兰大学澳大利亚分校）；Redwood Center for Theoretical Neuroscience（红木理论神经科学中心）；University of California at Berkeley USA（加州大学伯克利分校美国分校）
Disentangling the independently controllable factors of variation by interacting with the world
MILA（米拉人工智能研究所）；Université de Montréal（蒙特利尔大学）；Element AI（元素人工智能公司）；McGill University（麦吉尔大学）；ENS Paris（巴黎高等师范学院）；Google Brain（谷歌大脑）；Facebook AI Research（脸书人工智能研究院）；Google Deepmind（谷歌DeepMind）
MILA, Université de Montréal（蒙特利尔大学 MILA）；ElementAI（ElementAI 公司）；ENS Paris（巴黎高等师范学院）；McGill University（麦吉尔大学）；École Polytechnique（巴黎综合理工学院）；Université du Québec à Montréal（魁北克大学蒙特利尔分校）
This paper proposes the Transition Motion Tensor, a data-driven framework that creates novel and physically accurate transitions outside of the motion dataset. It enables simulated characters to adopt new motion skills efficiently and robustly without modifying existing ones. Given several physically simulated controllers specializing in different motions, the tensor serves as a temporal guideline to transition between them. Through querying the tensor for transitions that best fit user-defined preferences, we can create a unified controller capable of producing novel transitions and solving c…
Motion synthesis in a dynamic environment has been a long-standing problem for character animation. Methods using motion capture data tend to scale poorly in complex environments because of their larger capturing and labeling requirement. Physics-based controllers are effective in this regard, albeit less controllable. In this paper, we present CARL, a quadruped agent that can be controlled with high-level directives and react naturally to dynamic environments. Starting with an agent that can imitate individual animation clips, we use Generative Adversarial Networks to adapt high-level control…
Indian Institute of Technology, Kharagpur（印度理工学院克勒格布尔分校）
With the rising participation of the common mass in social media, it is increasingly common now for policymakers/journalists to create online polls on social media to understand the political leanings of people in specific locations. The caveat here is that only influential people can make such an online polling and reach out at a mass scale. Further, in such cases, the distribution of voters is not controllable and may be, in fact, biased. On the other hand,if we can interpret the publicly available data over social media to probe the political inclination of users, we will be able to have co…
Supervise Thyself: Examining Self-Supervised Representations in Interactive Environments
Mila（米拉研究所）；Université de Montréal（蒙特利尔大学）；École Polytechnique de Montréal（蒙特利尔理工学院）；Element AI（元素人工智能公司）
自我监督的方法，其中代理仅通过观察其行为的结果来学习表示，在不提供密集奖励信号或具有标签的环境中变得至关重要。在大多数情况下，这些方法用于“下游”任务的预训练或辅助任务，例如控制、探索或模仿学习。然而，目前尚不清楚哪种方法的表示最能捕捉环境的有意义的特征，以及哪种方法最适合于哪种类型的环境。我们提出了一个小规模的研究自我监督的方法在两个视觉环境：Flappy Bird和Sonic The Hedgehog。特别是，我们在两个方面定量评估从这些任务中学习到的表示：a）表示捕获代理的真实状态信息的程度和b）这些表示如何推广到新的情况，如新的水平和纹理。最后，我们评估这些自我监督的功能，通过可视化的部分，他们专注于环境。我们的研究结果表明，效用的表示是高度依赖于视觉和动态的环境。
Robust Semi-Supervised Learning in Open Environments
Nanjing University（南京大学）；School of Intelligence Science and Technology, Nanjing University（南京大学智能科学与技术学院）；School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；ETH Zurich（苏黎世联邦理工学院）
Learning diverse skills is one of the main challenges in robotics. To this end, imitation learning approaches have achieved impressive results. These methods require explicitly labeled datasets or assume consistent skill execution to enable learning and active control of individual behaviors, which limits their applicability. In this work, we propose a cooperative adversarial method for obtaining single versatile policies with controllable skill sets from unlabeled datasets containing diverse state transition patterns by maximizing their discriminability. Moreover, we show that by utilizing un…
SALIENT: Frequency-Aware Paired Diffusion for Controllable Long-Tail CT Detection
Trauma Radiology AI Laboratory (TRAIL)（创伤放射AI实验室）；Department of Radiology and Nuclear Medicine（放射科与核医学科）；University of Maryland School of Medicine（马里兰大学医学院）
University of California San Diego（加利福尼亚大学圣迭戈分校）；Bar-Ilan University（巴伊兰大学）；Technical University Denmark（丹麦技术大学）
We propose a semi-supervised approach to acoustic source localization in reverberant environments based on deep generative modeling. Localization in reverberant environments remains an open challenge. Even with large data volumes, the number of labels available for supervised learning in reverberant environments is usually small. We address this issue by performing semi-supervised learning (SSL) with convolutional variational autoencoders (VAEs) on reverberant speech signals recorded with microphone arrays. The VAE is trained to generate the phase of relative transfer functions (RTFs) between …
