Learning disentangled representations is a key step towards effectively discovering and modelling the underlying structure of environments. In the natural sciences, physics has found great success by describing the universe in terms of symmetry preserving transformations. Inspired by this formalism, we propose a framework, built upon the theory of group representation, for learning representations of a dynamical environment structured around the transformations that generate its evolution. Experimentally, we learn the structure of explicitly symmetric environments without supervision from observational data generated by sequential interactions. We further introduce an intuitive disentanglement regularisation to ensure the interpretability of the learnt representations. We show that our method enables accurate long-horizon predictions, and demonstrate a correlation between the quality of predictions and disentanglement in the latent space.
Finding a generally accepted formal definition of a disentangled representation in the context of an agent behaving in an environment is an important challenge towards the construction of data-efficient autonomous agents. Higgins et al. recently proposed Symmetry-Based Disentangled Representation Learning, a definition based on a characterization of symmetries in the environment using group theory. We build on their work and make observations, theoretical and empirical, that lead us to argue that Symmetry-Based Disentangled Representation Learning cannot only be based on static observations: a…
Nara Institute of Science and Technology（奈良先端科学技术大学院大学）
A variational autoencoder (VAE) derived from Tsallis statistics called q-VAE is proposed. In the proposed method, a standard VAE is employed to statistically extract latent space hidden in sampled data, and this latent space helps make robots controllable in feasible computational time and cost. To improve the usefulness of the latent space, this paper focuses on disentangled representation learning, e.g., $β$-VAE, which is the baseline for it. Starting from a Tsallis statistics perspective, a new lower bound for the proposed q-VAE is derived to maximize the likelihood of the sampled data, whi…
Intrinsically motivated goal exploration processes enable agents to autonomously sample goals to explore efficiently complex environments with high-dimensional continuous actions. They have been applied successfully to real world robots to discover repertoires of policies producing a wide diversity of effects. Often these algorithms relied on engineered goal spaces but it was recently shown that one can use deep representation learning algorithms to learn an adequate goal space in simple environments. However, in the case of more complex environments containing multiple objects or distractors,…
The University of Tokyo（东京大学）；RIKEN AIP（理化学研究所先进智能项目中心）；Microsoft Research Asia（微软亚洲研究院）
Machine learning systems may encounter unexpected problems when the data distribution changes in the deployment environment. A major reason is that certain combinations of domains and labels are not observed during training but appear in the test environment. Although various invariance-based algorithms can be applied, we find that the performance gain is often marginal. To formally analyze this issue, we provide a unique algebraic formulation of the combination shift problem based on the concepts of homomorphism, equivariance, and a refined definition of disentanglement. The algebraic require…
Hidden parameters are latent variables in reinforcement learning (RL) environments that are constant over the course of a trajectory. Understanding what, if any, hidden parameters affect a particular environment can aid both the development and appropriate usage of RL systems. We present an unsupervised method to map RL trajectories into a feature space where distance represents the relative difference in system behavior due to hidden parameters. Our approach disentangles the effects of hidden parameters by leveraging a recurrent neural network (RNN) world model as used in model-based RL. Firs…
Biological intelligence can learn to solve many diverse tasks in a data efficient manner by re-using basic knowledge and skills from one task to another. Furthermore, many of such skills are acquired without explicit supervision in an intrinsically driven fashion. This is in contrast to the state-of-the-art reinforcement learning agents, which typically start learning each new task from scratch and struggle with knowledge transfer. In this paper we propose a principled way to learn a basis set of policies, which, when recombined through generalised policy improvement, come with guarantees on t…
ETH Zurich（苏黎世联邦理工学院）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Google Research（谷歌研究院）
Intelligent agents should be able to learn useful representations by observing changes in their environment. We model such observations as pairs of non-i.i.d. images sharing at least one of the underlying factors of variation. First, we theoretically show that only knowing how many factors have changed, but not which ones, is sufficient to learn disentangled representations. Second, we provide practical algorithms that learn disentangled representations from pairs of images without requiring annotation of groups, individual factors, or the number of factors that have changed. Third, we perform…
Disentangled World Models: Learning to Transfer Semantic Knowledge from Distracting Videos for Reinforcement Learning
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（上海交通大学人工智能研究院教育部人工智能重点实验室）；Ningbo Institute of Digital Twin, Eastern Institute of Technology, Ningbo, China（东方理工高等研究院宁波数字孪生研究院）；Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative, Ningbo, China（宁波市空间智能与数字衍生重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Shenyang Institute of Computing Technology, Chinese Academy of Sciences（中国科学院沈阳计算技术研究所）；Shenyang CASNC Technology Co., Ltd（沈阳中科数控技术股份有限公司）
Graduate School of Engineering, The University of Tokyo（东京大学工学系研究科）
There is considerable interest in designing meta-reinforcement learning (meta-RL) algorithms, which enable autonomous agents to adapt new tasks from small amount of experience. In meta-RL, the specification (such as reward function) of current task is hidden from the agent. In addition, states are hidden within each task owing to sensor noise or limitations in realistic environments. Therefore, the meta-RL agent faces the challenge of specifying both the hidden task and states based on small amount of experience. To address this, we propose estimating disentangled belief about task and states,…
Department of Computer Science at UIUC（伊利诺伊大学厄巴纳-香槟分校计算机科学系）；College of Control Science and Engineering, Zhejiang University（浙江大学控制科学与工程学院）
This paper challenges the common assumption that the weight $β$, in $β$-VAE, should be larger than $1$ in order to effectively disentangle latent factors. We demonstrate that $β$-VAE, with $β< 1$, can not only attain good disentanglement but also significantly improve reconstruction accuracy via dynamic control. The paper removes the inherent trade-off between reconstruction accuracy and disentanglement for $β$-VAE. Existing methods, such as $β$-VAE and FactorVAE, assign a large weight to the KL-divergence term in the objective function, leading to high reconstruction errors for the sake of be…
Southeast University（东南大学）；University of New South Wales（新南威尔士大学）；University of California, Irvine（加州大学欧文分校）
Deep learning (DL) applied to a device's radio-frequency fingerprint~(RFF) has attracted significant attention in physical-layer authentication due to its extraordinary classification performance. Conventional DL-RFF techniques are trained by adopting maximum likelihood estimation~(MLE). Although their discriminability has recently been extended to unknown devices in open-set scenarios, they still tend to overfit the channel statistics embedded in the training dataset. This restricts their practical applications as it is challenging to collect sufficient training data capturing the characteris…
Recently in the field of unsupervised representation learning, strong identifiability results for disentanglement of causally-related latent variables have been established by exploiting certain side information, such as class labels, in addition to independence. However, most existing work is constrained by functional form assumptions such as independent sources or further with linear transitions, and distribution assumptions such as stationary, exponential family distribution. It is unknown whether the underlying latent variables and their causal relations are identifiable if they have arbit…
Technical University of Denmark（丹麦技术大学）；DeepMind（深度思考）
In model-based reinforcement learning, generative and temporal models of environments can be leveraged to boost agent performance, either by tuning the agent's representations during training or via use as part of an explicit planning mechanism. However, their application in practice has been limited to simplistic environments, due to the difficulty of training such models in larger, potentially partially-observed and 3D environments. In this work we introduce a novel action-conditioned generative model of such challenging environments. The model features a non-parametric spatial memory system…
Collaborative Cognitive Diagnosis with Disentangled Representation Learning for Learner Modeling
University of Science and Technology of China（中国科学技术大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）
Learning from demonstration is an effective method for human users to instruct desired robot behaviour. However, for most non-trivial tasks of practical interest, efficient learning from demonstration depends crucially on inductive bias in the chosen structure for rewards/costs and policies. We address the case where this inductive bias comes from an exchange with a human user. We propose a method in which a learning agent utilizes the information bottleneck layer of a high-parameter variational neural model, with auxiliary loss terms, in order to ground abstract concepts such as spatial relat…
We present a novel method for learning a set of disentangled reward functions that sum to the original environment reward and are constrained to be independently obtainable. We define independent obtainability in terms of value functions with respect to obtaining one learned reward while pursuing another learned reward. Empirically, we illustrate that our method can learn meaningful reward decompositions in a variety of domains and that these decompositions exhibit some form of generalization performance when the environment's reward is modified. Theoretically, we derive results about the effe…
University of Edinburgh（爱丁堡大学）；Aalto University（阿尔托大学）；University of Wisconsin – Madison（威斯康星大学麦迪逊分校）
Reinforcement Learning (RL) agents are often unable to generalise well to environment variations in the state space that were not observed during training. This issue is especially problematic for image-based RL, where a change in just one variable, such as the background colour, can change many pixels in the image. The changed pixels can lead to drastic changes in the agent's latent representation of the image, causing the learned policy to fail. To learn more robust representations, we introduce TEmporal Disentanglement (TED), a self-supervised auxiliary task that leads to disentangled image…
Semimage: HSV-Based Semantic Image Encoding for Disentangled Text Representation
AI Lab at Department of Computer Engineering（计算机工程系人工智能实验室）；AriooBarzan Engineering Team and Information Technology（AriooBarzan工程团队和信息技术）；Shiraz University of Technology（谢兹大学技术学院）
We construct an unsupervised learning model that achieves nonlinear disentanglement of underlying factors of variation in naturalistic videos. Previous work suggests that representations can be disentangled if all but a few factors in the environment stay constant at any point in time. As a result, algorithms proposed for this problem have only been tested on carefully constructed datasets with this exact property, leaving it unclear whether they will transfer to natural scenes. Here we provide evidence that objects in segmented natural movies undergo transitions that are typically small in ma…
Conformal Disentanglement and Latent-Space Curation: A Neural Framework for Perspective Synthesis, Differentiation and Targeted Generation
Department of Applied Mathematics and Statistics, Johns Hopkins University（应用数学与统计学系，约翰霍普金斯大学）；Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室）；Faculty of Science, Technology and Medicine, University of Luxembourg（科学、技术与医学学院，卢森堡大学）；Department of Civil and Systems Engineering, Johns Hopkins University（土木与系统工程系，约翰霍普金斯大学）；Department of Chemical and Biomolecular Engineering, Johns Hopkins University（化学与生物分子工程系，约翰霍普金斯大学）
Sorbonne Université（索邦大学）；Thales（泰雷兹）；Criteo AI Lab（Criteo人工智能实验室）
When modeling dynamical systems from real-world data samples, the distribution of data often changes according to the environment in which they are captured, and the dynamics of the system itself vary from one environment to another. Generalizing across environments thus challenges the conventional frameworks. The classical settings suggest either considering data as i.i.d. and learning a single model to cover all situations or learning environment-specific models. Both are sub-optimal: the former disregards the discrepancies between environments leading to biased solutions, while the latter d…
Environment Inference for Learning Generalizable Dynamical System
College of Computer Science and Technology, National University of Defense Technology（国防科技大学计算机科学与技术学院）；School of Information, Renmin University of China（中国人民大学信息学院）；College of Systems Engineering, National University of Defense Technology（国防科技大学系统工程学院）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）；Laboratory for Big Data and Decision, National University of Defense Technology（国防科技大学大数据与决策实验室）
Tarbiat Modares University（塔比亚特·莫达雷斯大学）；Hamedan University of Technology（哈马丹工业大学）
In this paper, a review of model-free reinforcement learning for learning of dynamical systems in uncertain environments has discussed. For this purpose, the Markov Decision Process (MDP) will be reviewed. Furthermore, some learning algorithms such as Temporal Difference (TD) learning, Q-Learning, and Approximate Q-learning as model-free algorithms which constitute the main part of this article have been investigated, and benefits and drawbacks of each algorithm will be discussed. The discussed concepts in each section are explaining with details and examples.
