The development of quantum computational techniques has advanced greatly in recent years, parallel to the advancements in techniques for deep reinforcement learning. This work explores the potential for quantum computing to facilitate reinforcement learning problems. Quantum computing approaches offer important potential improvements in time and space complexity over traditional algorithms because of its ability to exploit the quantum phenomena of superposition and entanglement. Specifically, we investigate the use of quantum variational circuits, a form of quantum machine learning. We present our techniques for encoding classical data for a quantum variational circuit, we further explore pure and hybrid quantum algorithms for DQN and Double DQN. Our results indicate both hybrid and pure quantum variational circuit have the ability to solve reinforcement learning tasks with a smaller parameter space. These comparison are conducted with two OpenAI Gym environments: CartPole and Blackjack, The success of this work is indicative of a strong future relationship between quantum machine learning and deep reinforcement learning.
National Taiwan University（台湾大学）；Georgia Institute of Technology（佐治亚理工学院）；IBM Research（IBM研究院）
The state-of-the-art machine learning approaches are based on classical von Neumann computing architectures and have been widely used in many industrial and academic domains. With the recent development of quantum computing, researchers and tech-giants have attempted new quantum circuits for machine learning tasks. However, the existing quantum computing platforms are hard to simulate classical deep learning models or problems because of the intractability of deep quantum circuits. Thus, it is necessary to design feasible quantum algorithms for quantum machine learning for noisy intermediate s…
We consider undiscounted reinforcement learning in Markov decision processes (MDPs) where both the reward functions and the state-transition probabilities may vary (gradually or abruptly) over time. For this problem setting, we propose an algorithm and provide performance guarantees for the regret evaluated against the optimal non-stationary policy. The upper bound on the regret is given in terms of the total variation in the MDP. This is the first variational regret bound for the general reinforcement learning setting.
We propose a method for finding approximate compilations of quantum unitary transformations, based on techniques from policy gradient reinforcement learning. The choice of a stochastic policy allows us to rephrase the optimization problem in terms of probability distributions, rather than variational gates. In this framework, the optimal configuration is found by optimizing over distribution parameters, rather than over free angles. We show numerically that this approach can be more competitive than gradient-free methods, for a comparable amount of resources, both for noiseless and noisy circu…
Quantum Machine Learning (QML) is considered to be one of the most promising applications of near term quantum devices. However, the optimization of quantum machine learning models presents numerous challenges arising from the imperfections of hardware and the fundamental obstacles in navigating an exponentially scaling Hilbert space. In this work, we evaluate the potential of contemporary methods in deep reinforcement learning to augment gradient based optimization routines in quantum variational circuits. We find that reinforcement learning augmented optimizers consistently outperform gradie…
University of Minho（米尼奥大学）；INESC TEC（葡萄牙系统与计算机工程研究所（INESC TEC））；International Nanotechnology Laboratory (INL)（国际纳米技术实验室（INL））
Variational Quantum Circuits are being used as versatile Quantum Machine Learning models. Some empirical results exhibit an advantage in supervised and generative learning tasks. However, when applied to Reinforcement Learning, less is known. In this work, we considered a Variational Quantum Circuit composed of a low-depth hardware-efficient ansatz as the parameterized policy of a Reinforcement Learning agent. We show that an $ε$-approximation of the policy gradient can be obtained using a logarithmic number of samples concerning the total number of parameters. We empirically verify that such …
Exploration in environments with continuous control and sparse rewards remains a key challenge in reinforcement learning (RL). Recently, surprise has been used as an intrinsic reward that encourages systematic and efficient exploration. We introduce a new definition of surprise and its RL implementation named Variational Assorted Surprise Exploration (VASE). VASE uses a Bayesian neural network as a model of the environment dynamics and is trained using variational inference, alternately updating the accuracy of the agent's model and policy. Our experiments show that in continuous control spars…
Aiming at a comprehensive and concise tutorial survey, recap of variational inference and reinforcement learning with Probabilistic Graphical Models are given with detailed derivations. Reviews and comparisons on recent advances in deep reinforcement learning are made from various aspects. We offer detailed derivations to a taxonomy of Probabilistic Graphical Model and Variational Inference methods in deep reinforcement learning, which serves as a complementary material on top of the original contributions.
Deep reinforcement learning (DRL) has achieved significant success in various robot tasks: manipulation, navigation, etc. However, complex visual observations in natural environments remains a major challenge. This paper presents Contrastive Variational Reinforcement Learning (CVRL), a model-based method that tackles complex visual observations in DRL. CVRL learns a contrastive variational model by maximizing the mutual information between latent states and observations discriminatively, through contrastive learning. It avoids modeling the complex observation space unnecessarily, as the common…
Institute for Computational & Mathematical Engineering, Stanford University（斯坦福大学计算与数学工程研究所）；Management Science & Engineering, Stanford University（斯坦福大学管理科学与工程系）
This paper provides a unifying view of a wide range of problems of interest in machine learning by framing them as the minimization of functionals defined on the space of probability measures. In particular, we show that generative adversarial networks, variational inference, and actor-critic methods in reinforcement learning can all be seen through the lens of our framework. We then discuss a generic optimization algorithm for our formulation, called probability functional descent (PFD), and show how this algorithm recovers existing methods developed independently in the settings mentioned ea…
IRT Saint Exupery（圣埃克苏佩里技术研究所）；Université Paul Sabatier（保罗·萨巴捷大学）；IRIT（图卢兹信息学研究所）
Quantum walks on binary trees are used in many quantum algorithms to achieve important speedup over classical algorithms. The formulation of this kind of algorithms as quantum circuit presents the advantage of being easily readable, executable on circuit based quantum computers and simulators and optimal on the usage of resources. We propose a strategy to compose quantum circuit that performs quantum walk on binary trees following universal gate model quantum computation principles. We give a particular attention to NAND formula evaluation algorithm as it could have many applications in game t…
University of Oxford（牛津大学）；University of British Columbia（不列颠哥伦比亚大学）
Many real-world sequential decision making problems are partially observable by nature, and the environment model is typically unknown. Consequently, there is great need for reinforcement learning methods that can tackle such problems given only a stream of incomplete and noisy observations. In this paper, we propose deep variational reinforcement learning (DVRL), which introduces an inductive bias that allows an agent to learn a generative model of the environment and perform inference in that model to effectively aggregate the available information. We develop an n-step approximation to the …
Equivariant Reinforcement Learning for Clifford Quantum Circuit Synthesis
Department of Computer Science, University of Oxford（牛津大学计算机科学系）；Quantinuum（Quantinuum公司）；College of Computing and Data Science, Nanyang Technological University, Singapore（新加坡南洋理工大学计算科学与数据科学学院）
Ant Financial Services Group（蚂蚁金融服务集团）；Shanghai Jiao Tong University（上海交通大学）；College of Computing, Georgia Institute of Technology（佐治亚理工学院计算学院）
We propose a \emph{collaborative} multi-agent reinforcement learning algorithm named variational policy propagation (VPP) to learn a \emph{joint} policy through the interactions over agents. We prove that the joint policy is a Markov Random Field under some mild conditions, which in turn reduces the policy space effectively. We integrate the variational inference as special differentiable layers in policy such that the actions can be efficiently sampled from the Markov Random Field and the overall policy is differentiable. We evaluate our algorithm on several large scale challenging tasks and …
University of California, Berkeley（加州大学伯克利分校）；Stanford University（斯坦福大学）；St. Kliment Ohridski University of Sofia（索非亚圣克里门特·奥赫里德大学）；Max Planck Institute for the Physics of Complex Systems（马克斯·普朗克复杂系统物理研究所）；Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室）
Variational quantum algorithms stand at the forefront of simulations on near-term and future fault-tolerant quantum devices. While most variational quantum algorithms involve only continuous optimization variables, the representational power of the variational ansatz can sometimes be significantly enhanced by adding certain discrete optimization variables, as is exemplified by the generalized quantum approximate optimization algorithm (QAOA). However, the hybrid discrete-continuous optimization problem in the generalized QAOA poses a challenge to the optimization. We propose a new algorithm ca…
École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）
Modern reinforcement learning algorithms reach super-human performance on many board and video games, but they are sample inefficient, i.e. they typically require significantly more playing experience than humans to reach an equal performance level. To improve sample efficiency, an agent may build a model of the environment and use planning methods to update its policy. In this article we introduce Variational State Tabulation (VaST), which maps an environment with a high-dimensional state space (e.g. the space of visual inputs) to an abstract tabular model. Prioritized sweeping with small bac…
University of Oxford（牛津大学）；University of California, Berkeley（加州大学伯克利分校）
While reinforcement learning algorithms provide automated acquisition of optimal policies, practical application of such methods requires a number of design decisions, such as manually designing reward functions that not only define the task, but also provide sufficient shaping to accomplish it. In this paper, we view reinforcement learning as inferring policies that achieve desired outcomes, rather than as a problem of maximizing rewards. To solve this inference problem, we establish a novel variational inference formulation that allows us to derive a well-shaped reward function which can be …
One problem in the application of reinforcement learning to real-world problems is the curse of dimensionality on the action space. Macro actions, a sequence of primitive actions, have been studied to diminish the dimensionality of the action space with regard to the time axis. However, previous studies relied on humans defining macro actions or assumed macro actions as repetitions of the same primitive actions. We present Factorized Macro Action Reinforcement Learning (FaMARL) which autonomously learns disentangled factor representation of a sequence of actions to generate macro actions that …
Mitsubishi Electric Research Laboratories（三菱电机研究实验室）；Washington University in St. Louis（圣路易斯华盛顿大学）
The design automation of analog circuits is a longstanding challenge in the integrated circuit field. This paper presents a deep reinforcement learning method to expedite the design of analog circuits at the pre-layout stage, where the goal is to find device parameters to fulfill desired circuit specifications. Our approach is inspired by experienced human designers who rely on domain knowledge of analog circuit design (e.g., circuit topology and couplings between circuit specifications) to tackle the problem. Unlike all prior methods, our method originally incorporates such key domain knowled…
Adaptive sampling using variational autoencoder and reinforcement learning
Department of Engineering Cybernetics, Norwegian University of Science and Technology（工程 cybernetics 系，挪威科学与技术大学）；Department of Engineering Sciences, University of Agder（工程科学系，阿格德大学）
Reinforcement learning has shown great potential in generalizing over raw sensory data using only a single neural network for value optimization. There are several challenges in the current state-of-the-art reinforcement learning algorithms that prevent them from converging towards the global optima. It is likely that the solution to these problems lies in short- and long-term planning, exploration and memory management for reinforcement learning algorithms. Games are often used to benchmark reinforcement learning algorithms as they provide a flexible, reproducible, and easy to control environ…
Sub-universal variational circuits for combinatorial optimization problems
Department of Physics, University of Colorado Boulder, Boulder, CO 80309, USA（科罗拉多大学博尔德分校物理系）；University of Technology Sydney, Centre for Quantum Software（悉尼技术大学量子软件中心）；Department of Electrical, Computer and Energy Engineering, University of Colorado Boulder, Boulder, Colorado 80309, USA（科罗拉多大学博尔德分校电气、计算机与能源工程系）
National University of Defense Technology（国防科技大学）；National Innovation Institute of Defense Technology（国防科技创新研究院）
Quantum state tomography is a key process in most quantum experiments. In this work, we employ quantum machine learning for state tomography. Given an unknown quantum state, it can be learned by maximizing the fidelity between the output of a variational quantum circuit and this state. The number of parameters of the variational quantum circuit grows linearly with the number of qubits and the circuit depth, so that only polynomial measurements are required, even for highly-entangled states. After that, a subsequent classical circuit simulator is used to transform the information of the target …
We propose a novel algorithm for fast training of variational classifiers by processing multiple samples parallelly. The algorithm can be adapted for any ansatz used in the variational circuit. The presented algorithm utilizes qRAM and other quantum circuits in the forward pass. Further, instead of the usual practice of computing the loss classically, we calculate the loss using a Swap-test circuit. The algorithm thus brings down the training cost of a variational classifier to O(logN)from the usual O(N)when training on a dataset of N samples. Although we discuss only binary classification in …
All you need is spin: SU(2) equivariant variational quantum circuits based on spin networks
School of Integrated Technology, Yonsei University, Incheon 21983, Republic of Korea（延世大学集成技术学院）；Department of Quantum Information, Yonsei University, Incheon 21983, Republic of Korea（延世大学量子信息系）
Delhi Technological University（德里理工大学）；CHRIST (Deemed to be University)（克里斯大学（被视为大学））
There are numerous models of quantum neural networks that have been applied to variegated problems such as image classification, pattern recognition etc.Quantum inspired algorithms have been relevant for quite awhile. More recently, in the NISQ era, hybrid quantum classical models have shown promising results. Multi-feature regression is common problem in classical machine learning. Hence we present a comparative analysis of continuous variable quantum neural networks (Variational circuits) and quantum backpropagating multi layer perceptron (QBMLP). We have chosen the contemporary problem of p…
We demonstrate the simulation of a noncollinear molecule, e.g. H2O molecule using Variational Quantum Eigensolver (VQE) with high chemical accuracy. The 2D and 3D potential energy surface (PES) were reported. Taking advantage of the potential speedup in Qiskit runtime program, the optimal initial parameters for the variational quantum circuits were obtained after several consecutive iterations, thus resulting in accurate prediction of water's PES matching result obtained from exact diagonalization of the full Hamiltonian.
Circuit design for quantum machine learning remains a formidable challenge. Inspired by the applications of tensor networks across different fields and their novel presence in the classical machine learning context, one proposed method to design variational circuits is to base the circuit architecture on tensor networks. Here, we comprehensively describe tensor-network quantum circuits and how to implement them in simulations. This includes leveraging circuit cutting, a technique used to evaluate circuits with more qubits than those available on current quantum devices. We then illustrate the …
We explore how to build quantum circuits that compute the lowest energy state corresponding to a given Hamiltonian within a Symmetry subspace by explicitly encoding it into the circuit. We create an explicit unitary and a variationally trained unitary that maps any vector output by ansatz A(~α) from a defined subspace to a vector in the symmetry space. The parameters are trained varitionally to minimize the energy thus keeping the output within the labelled symmetry value. The method was tested for a spin XXZ hamiltonian using rotation and reflection symmetry and H2 hamiltonian within S_z = 0 …
Rensselaer Polytechnic Institute（伦斯勒理工学院）；Department of Computer Science（计算机科学系）；Troy NY, USA（美国纽约州特洛伊市（此处仅为地址信息，根据提取规则应忽略，正确仅保留大学））
Quantum Variational Circuits (QVCs) are often claimed as one of the most potent uses of both near term and long term quantum hardware. The standard approaches to optimizing these circuits rely on a classical system to compute the new parameters at every optimization step. However, this process can be extremely challenging, due to the nature of navigating the exponentially scaling complex Hilbert space, barren plateaus, and the noise present in all foreseeable quantum hardware. Although a variety of optimization algorithms are employed in practice, there is often a lack of theoretical or empiri…
We generalize solid-state tight-binding techniques for the spectral analysis of large superconducting circuits. We find that tight-binding states can be better suited for approximating the low-energy excitations than charge-basis states, as illustrated for the interesting example of the current-mirror circuit. The use of tight binding can dramatically lower the Hilbert space dimension required for convergence to the true spectrum, and allows for the accurate simulation of larger circuits that are out of reach of charge basis diagonalization.
