Reinforcement Learning to Disentangle Multiqubit Quantum States from Partial Observations
Department of Mathematics and Informatics（数学与信息学系）；St. Kliment Ohridski University of Sofia（索菲亚圣克莱门特·欧赫里迪斯大学）；Joint Quantum Institute（联合量子研究所）；Department of Physics（物理系）；University of Maryland（马里兰大学）；National Quantum Laboratory (QLab)（国家量子实验室（QLab））；Institute of Physics（物理研究所）；École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院（EPFL））；Center for Quantum Science and Engineering（量子科学与工程中心）；Quantum Systems Unit（量子系统单元）；OIST Graduate University（OIST研究生大学）；Duke Quantum Center（杜克量子中心）；Max Planck Institute for the Physics of Complex Systems（复杂系统物理研究所）
Pavel Tashev, Stefan Petrov, Matthew T. Diaz, Friederike Metz, Alaina M. Green, Norbert M. Linke, Marin Bukov
Using partial knowledge of a quantum state to control multiqubit entanglement is a largely unexplored paradigm in the emerging field of quantum interactive dynamics with the potential to address outstanding challenges in quantum state preparation and compression, quantum control, and quantum complexity. We present a deep reinforcement learning (RL) approach using an actor-critic algorithm for constructing short disentangling circuits for states with up to 16 qubits. With access to only two-qubit reduced density matrices, our agent decides which pairs of qubits to apply two-qubit gates on; requiring only local information makes it directly applicable on modern NISQ devices, as we demonstrated experimentally on a trapped-ion quantum computer. Utilizing a permutation-equivariant transformer architecture, the agent can autonomously identify qubit permutations within the state, and adjusts the disentangling protocol accordingly. Once trained, it provides circuits from different initial states without further optimization. We demonstrate the agent's ability to identify and exploit the entanglement structure of multi-qubit states. We analyze the disentangling circuits constructed by the agent for 4- and 5-qubit Haar-random states, and observe strong correlations between consecutive gates and among the qubits involved. Through extensive benchmarking, we show the efficacy of the RL approach to find disentangling protocols with minimal gate resources. We explore the resilience of our trained agents to noise, highlighting their potential for real-world quantum computing applications. Analyzing optimal disentangling protocols, we report a general circuit to prepare an arbitrary 4-qubit state using at most 5 two-qubit (10 CNOT) gates.
University of Edinburgh（爱丁堡大学）；Aalto University（阿尔托大学）；University of Wisconsin – Madison（威斯康星大学麦迪逊分校）
Reinforcement Learning (RL) agents are often unable to generalise well to environment variations in the state space that were not observed during training. This issue is especially problematic for image-based RL, where a change in just one variable, such as the background colour, can change many pixels in the image. The changed pixels can lead to drastic changes in the agent's latent representation of the image, causing the learned policy to fail. To learn more robust representations, we introduce TEmporal Disentanglement (TED), a self-supervised auxiliary task that leads to disentangled image…
Hidden parameters are latent variables in reinforcement learning (RL) environments that are constant over the course of a trajectory. Understanding what, if any, hidden parameters affect a particular environment can aid both the development and appropriate usage of RL systems. We present an unsupervised method to map RL trajectories into a feature space where distance represents the relative difference in system behavior due to hidden parameters. Our approach disentangles the effects of hidden parameters by leveraging a recurrent neural network (RNN) world model as used in model-based RL. Firs…
University of California, Berkeley（加州大学伯克利分校）；Intrinsic Innovation LLC（Intrinsic创新有限责任公司）；Tsinghua University（清华大学）
Humans are capable of abstracting various tasks as different combinations of multiple attributes. This perspective of compositionality is vital for human rapid learning and adaption since previous experiences from related tasks can be combined to generalize across novel compositional settings. In this work, we aim to achieve zero-shot policy generalization of Reinforcement Learning (RL) agents by leveraging the task compositionality. Our proposed method is a meta- RL algorithm with disentangled task representation, explicitly encoding different aspects of the tasks. Policy generalization is th…
Acceleration of Actor-Critic Deep Reinforcement Learning for Visual Grasping in Clutter by State Representation Learning Based on Disentanglement of a Raw Input Image
In recent years, there are great interests as well as challenges in applying reinforcement learning (RL) to recommendation systems (RS). In this paper, we summarize three key practical challenges of large-scale RL-based recommender systems: massive state and action spaces, high-variance environment, and the unspecific reward setting in recommendation. All these problems remain largely unexplored in the existing literature and make the application of RL challenging. We develop a model-based reinforcement learning framework, called GoalRec. Inspired by the ideas of world model (model-based), val…
Disentangled World Models: Learning to Transfer Semantic Knowledge from Distracting Videos for Reinforcement Learning
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（上海交通大学人工智能研究院教育部人工智能重点实验室）；Ningbo Institute of Digital Twin, Eastern Institute of Technology, Ningbo, China（东方理工高等研究院宁波数字孪生研究院）；Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative, Ningbo, China（宁波市空间智能与数字衍生重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Shenyang Institute of Computing Technology, Chinese Academy of Sciences（中国科学院沈阳计算技术研究所）；Shenyang CASNC Technology Co., Ltd（沈阳中科数控技术股份有限公司）
Rethinking State Disentanglement in Causal Reinforcement Learning
Australia Institute for Machine Learning, The University of Adelaide（阿德莱德大学澳大利亚机器学习研究所）；Shanghai Jiao Tong University（上海交通大学）；University of California San Diego（加利福尼亚大学圣迭戈分校）；The University of Melbourne（墨尔本大学）
Multiqubit and multilevel quantum reinforcement learning with quantum technologies
Universidad de Santiago de Chile (USACH)（智利圣地亚哥大学）；Center for the Development of Nanoscience and Nanotechnology（纳米科学与纳米技术发展中心）；University of the Basque Country UPV/EHU（巴斯克大学）；IKERBASQUE, Basque Foundation for Science（伊克尔巴斯克巴斯克科学基金会）；Shanghai University（上海大学）
Task Adaptation from Skills: Information Geometry, Disentanglement, and New Objectives for Unsupervised Reinforcement Learning
Eindhoven University of Technology（埃因霍温理工大学）；University of Maryland, College Park（马里兰大学学院市分校）；Ruhr University Bochum（波鸿鲁尔大学）；Tencent Robotics X（腾讯机器人科技）；University of Liverpool（利物浦大学）
The University of Hong Kong（香港大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；Tianjin University（天津大学）；Tsinghua University（清华大学）
Adapting to the changes in transition dynamics is essential in robotic applications. By learning a conditional policy with a compact context, context-aware meta-reinforcement learning provides a flexible way to adjust behavior according to dynamics changes. However, in real-world applications, the agent may encounter complex dynamics changes. Multiple confounders can influence the transition dynamics, making it challenging to infer accurate context for decision-making. This paper addresses such a challenge by Decomposed Mutual INformation Optimization (DOMINO) for context learning, which expli…
In this paper, we propose a max-min entropy framework for reinforcement learning (RL) to overcome the limitation of the soft actor-critic (SAC) algorithm implementing the maximum entropy RL in model-free sample-based learning. Whereas the maximum entropy RL guides learning for policies to reach states with high entropy in the future, the proposed max-min entropy framework aims to learn to visit states with low entropy and maximize the entropy of these low-entropy states to promote better exploration. For general Markov decision processes (MDPs), an efficient algorithm is constructed under the …
Leiden Institute of Advanced Computer Science, Leiden University（莱顿大学莱顿高级计算机科学研究所）；TU Delft（代尔夫特理工大学）
Sequential decision making, commonly formalized as optimization of a Markov Decision Process, is a key challenge in artificial intelligence. Two successful approaches to MDP optimization are reinforcement learning and planning, which both largely have their own research communities. However, if both research fields solve the same problem, then we might be able to disentangle the common factors in their solution approaches. Therefore, this paper presents a unifying algorithmic framework for reinforcement learning and planning (FRAP), which identifies underlying dimensions on which MDP planning …
Tsinghua University（清华大学）；UC Berkeley（加州大学伯克利分校）；UC San Diego（加州大学圣地亚哥分校）；Shanghai Qi Zhi Institute（上海期智研究院）
We address the problem of safely solving complex bimanual robot manipulation tasks with sparse rewards. Such challenging tasks can be decomposed into sub-tasks that are accomplishable by different robots concurrently or sequentially for better efficiency. While previous reinforcement learning approaches primarily focus on modeling the compositionality of sub-tasks, two fundamental issues are largely ignored particularly when learning cooperative strategies for two robots: (i) domination, i.e., one robot may try to solve a task by itself and leaves the other idle; (ii) conflict, i.e., one robot…
A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspective
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；State Key Laboratory of Blockchain and Data Security, Zhejiang University（浙江大学区块链与数据安全国家重点实验室）；Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security（杭州高新区（滨江）区块链与数据安全研究院）
Bitdefender（比特梵德）；Technical University of Cluj-Napoca（克卢日-纳波卡技术大学）；Imperial College London（帝国理工学院）；DeepMind（深度思维）；University College London（伦敦大学学院）
Most of the recent deep reinforcement learning advances take an RL-centric perspective and focus on refinements of the training objective. We diverge from this view and show we can recover the performance of these developments not by changing the objective, but by regularising the value-function estimator. Constraining the Lipschitz constant of a single layer using spectral normalisation is sufficient to elevate the performance of a Categorical-DQN agent to that of a more elaborated \rainbow{} agent on the challenging Atari domain. We conduct ablation studies to disentangle the various effects…
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；ETH Zurich（苏黎世联邦理工学院）
Autonomous agents need large repertoires of skills to act reasonably on new tasks that they have not seen before. However, acquiring these skills using only a stream of high-dimensional, unstructured, and unlabeled observations is a tricky challenge for any autonomous agent. Previous methods have used variational autoencoders to encode a scene into a low-dimensional vector that can be used as a goal for an agent to discover new skills. Nevertheless, in compositional/multi-object environments it is difficult to disentangle all the factors of variation into such a fixed-length representation of …
Reinforcement learning (RL) is a powerful framework for learning to take actions to solve tasks. However, in many settings, an agent must winnow down the inconceivably large space of all possible tasks to the single task that it is currently being asked to solve. Can we instead constrain the space of tasks to those that are semantically meaningful? In this work, we introduce a framework for using weak supervision to automatically disentangle this semantically meaningful subspace of tasks from the enormous space of nonsensical "chaff" tasks. We show that this learned subspace enables efficient …
Inverse reinforcement learning (IRL) seeks to infer a cost function that explains the underlying goals and preferences of expert demonstrations. This paper presents receding horizon inverse reinforcement learning (RHIRL), a new IRL algorithm for high-dimensional, noisy, continuous systems with black-box dynamic models. RHIRL addresses two key challenges of IRL: scalability and robustness. To handle high-dimensional continuous systems, RHIRL matches the induced optimal trajectories with expert demonstrations locally in a receding horizon manner and 'stitches' together the local solutions to lea…
Semantically Aligned Task Decomposition in Multi-Agent Reinforcement Learning
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳人工智能与机器人研究院）；East China Normal University（华东师范大学）；Tongji University（同济大学）
University of California San Diego（加州大学圣地亚哥分校）；Amazon（亚马逊）
We present a two-step hybrid reinforcement learning (RL) policy that is designed to generate interpretable and robust hierarchical policies on the RL problem with graph-based input. Unlike prior deep reinforcement learning policies parameterized by an end-to-end black-box graph neural network, our approach disentangles the decision-making process into two steps. The first step is a simplified classification problem that maps the graph input to an action group where all actions share a similar semantic meaning. The second step implements a sophisticated rule-miner that conducts explicit one-hop…
Impressive results in natural language processing (NLP) based on the Transformer neural network architecture have inspired researchers to explore viewing offline reinforcement learning (RL) as a generic sequence modeling problem. Recent works based on this paradigm have achieved state-of-the-art results in several of the mostly deterministic offline Atari and D4RL benchmarks. However, because these methods jointly model the states and actions as a single sequencing problem, they struggle to disentangle the effects of the policy and world dynamics on the return. Thus, in adversarial or stochast…
Graduate School of Engineering, The University of Tokyo（东京大学工学系研究科）
There is considerable interest in designing meta-reinforcement learning (meta-RL) algorithms, which enable autonomous agents to adapt new tasks from small amount of experience. In meta-RL, the specification (such as reward function) of current task is hidden from the agent. In addition, states are hidden within each task owing to sensor noise or limitations in realistic environments. Therefore, the meta-RL agent faces the challenge of specifying both the hidden task and states based on small amount of experience. To address this, we propose estimating disentangled belief about task and states,…
University of Oxford（牛津大学）；University of California, Berkeley（加州大学伯克利分校）；Google Research（谷歌研究院）；DeepMind（深度思维）
We study reinforcement learning (RL) with no-reward demonstrations, a setting in which an RL agent has access to additional data from the interaction of other agents with the same environment. However, it has no access to the rewards or goals of these agents, and their objectives and levels of expertise may vary widely. These assumptions are common in multi-agent settings, such as autonomous driving. To effectively use this data, we turn to the framework of successor features. This allows us to disentangle shared features and dynamics of the environment from agent-specific rewards and policies…
Learning Interactive World Model for Object-Centric Reinforcement Learning
University of California San Diego（加州大学圣地亚哥分校）；Mohamed bin Zayed University of Artificial Intelligence（马尔代夫穆罕默德·本·扎耶德人工智能大学）；University of Amsterdam（阿姆斯特丹大学）
Towards Governing Agent's Efficacy: Action-Conditional $β$-VAE for Deep Transparent Reinforcement Learning
Seoul National University（首尔国立大学）；Samsung Electronics（三星电子）
我们在强化学习（RL）的设置中解决了深度神经网络的黑盒问题，其中神经代理以不可控的方式学习最大化奖励收益。当交互环境包括广阔的状态空间时，这种学习方法是有风险的，因为几乎不可能预见所有不想要的结果并预先用负奖励惩罚它们。与以前的作品中学习的神经功能的反向分析不同，我们提出的方法\nj{tackles the blackbox issue by encouraging}是一个RL策略网络，通过实现一个解纠缠的表示学习方法来学习可解释的潜在特征。为此，我们的方法允许RL代理通过将其影响与不可控的环境因素区分开来来理解自我效能，这与人类理解场景的方式非常相似。我们的实验结果表明，学习的潜在因素不仅是可解释的，而且还可以建模整个访问状态空间的分布与特定的动作条件。我们已经实验表明，拟议结构的这一特征可以导致对RL代理的期望行为进行事后治理。
A zero-shot RL agent is an agent that can solve any RL task in a given environment, instantly with no additional planning or learning, after an initial reward-free learning phase. This marks a shift from the reward-centric RL paradigm towards "controllable" agents that can follow arbitrary instructions in an environment. Current RL agents can solve families of related tasks at best, or require planning anew for each task. Strategies for approximate zero-shot RL ave been suggested using successor features (SFs) [BBQ+ 18] or forward-backward (FB) representations [TO21], but testing has been limi…
Network function virtualization (NFV) and software-defined network (SDN) have become emerging network paradigms, allowing virtualized network function (VNF) deployment at a low cost. Even though VNF deployment can be flexible, it is still challenging to optimize VNF deployment due to its high complexity. Several studies have approached the task as dynamic programming, e.g., integer linear programming (ILP). However, optimizing VNF deployment for highly complex networks remains a challenge. Alternatively, reinforcement learning (RL) based approaches have been proposed to optimize this task, esp…
