In the training of over-parameterized model functions via gradient descent, sometimes the parameters do not change significantly and remain close to their initial values. This phenomenon is called lazy training, and motivates consideration of the linear approximation of the model function around the initial parameters. In the lazy regime, this linear approximation imitates the behavior of the parameterized function whose associated kernel, called the tangent kernel, specifies the training performance of the model. Lazy training is known to occur in the case of (classical) neural networks with large widths. In this paper, we show that the training of geometrically local parameterized quantum circuits enters the lazy regime for large numbers of qubits. More precisely, we prove bounds on the rate of changes of the parameters of such a geometrically local parameterized quantum circuit in the training process, and on the precision of the linear approximation of the associated quantum model function; both of these bounds tend to zero as the number of qubits grows. We support our analytic results with numerical simulations.
CNRS（法国国家科学研究中心）；Université Paris-Sud（巴黎第十一大学）；CentraleSupelec（巴黎中央理工-高等电力学院）；INRIA（法国国家信息与自动化研究所）；ENS（巴黎高等师范学院）；PSL Research University（巴黎文理研究大学）
In a series of recent theoretical works, it was shown that strongly over-parameterized neural networks trained with gradient-based methods could converge exponentially fast to zero training loss, with their parameters hardly varying. In this work, we show that this "lazy training" phenomenon is not specific to over-parameterized neural networks, and is due to a choice of scaling, often implicit, that makes the model behave as its linearization around the initialization, thus yielding a model equivalent to learning with positive-definite kernels. Through a theoretical analysis, we exhibit vario…
Microsoft Research India（微软印度研究院）；IIT Kanpur（印度理工学院坎普尔分校）
Training multi-layer Graph Convolution Networks (GCN) using standard SGD techniques scales poorly as each descent step ends up updating node embeddings for a large portion of the graph. Recent attempts to remedy this sub-sample the graph that reduces compute but introduce additional variance and may offer suboptimal performance. This paper develops the IGLU method that caches intermediate computations at various GCN layers thus enabling lazy updates that significantly reduce the compute cost of descent. IGLU introduces bounded bias into the gradients but nevertheless converges to a first-order…
Duke University（杜克大学）；Princeton University（普林斯顿大学）；Stanford University（斯坦福大学）
Over-parametrization is an important technique in training neural networks. In both theory and practice, training a larger network allows the optimization algorithm to avoid bad local optimal solutions. In this paper we study a closely related tensor decomposition problem: given an $l$-th order tensor in $(R^d)^{\otimes l}$ of rank $r$ (where $r\ll d$), can variants of gradient descent find a rank $m$ decomposition where $m > r$? We show that in a lazy training regime (similar to the NTK regime for neural networks) one needs at least $m = Ω(d^{l-1})$, while a variant of gradient descent can fi…
Modern neural networks are often operated in a strongly overparametrized regime: they comprise so many parameters that they can interpolate the training set, even if actual labels are replaced by purely random ones. Despite this, they achieve good prediction error on unseen data: interpolating the training set does not lead to a large generalization error. Further, overparametrization appears to be beneficial in that it simplifies the optimization landscape. Here we study these phenomena in the context of two-layers neural networks in the neural tangent (NT) regime. We consider a simple data m…
University of Minnesota, Twin Cities（明尼苏达大学双城分校）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
In spite of advances in understanding lazy training, recent work attributes the practical success of deep learning to the rich regime with complex inductive bias. In this paper, we study rich regime training empirically with benchmark datasets, and find that while most parameters are lazy, there is always a small number of active parameters which change quite a bit during training. We show that re-initializing (resetting to their initial random values) the active parameters leads to worse generalization. Further, we show that most of the active parameters are in the bottom layers, close to the…
For small training set sizes $P$, the generalization error of wide neural networks is well-approximated by the error of an infinite width neural network (NN), either in the kernel or mean-field/feature-learning regime. However, after a critical sample size $P^*$, we empirically find the finite-width network generalization becomes worse than that of the infinite width network. In this work, we empirically study the transition from infinite-width behavior to this variance limited regime as a function of sample size $P$ and network width $N$. We find that finite-size effects can become relevant f…
Lazy quantum walks were presented by Andrew M. Childs to prove that the continuous-time quantum walk is a limit of the discrete-time quantum walk [Commun.Math.Phys.294,581-603(2010)]. In this paper, we discuss properties of lazy quantum walks. Our analysis shows that lazy quantum walks have $O(t^n)$ order of the n-th moment of the corresponding probability distribution, which is the same as that for normal quantum walks. Also, the lazy quantum walk with DFT (Discrete Fourier Transform) coin operator has a similar probability distribution concentrated interval to that of the normal Hadamard qua…
Quantum walks play an important role for developing quantum algorithms and quantum simulations. Here we present one dimensional three-state quantum walk(lazy quantum walk) and show its equivalence for circuit realization in ternary quantum logic for the first of its kind. Using an appropriate logical mapping of the position space on which a walker evolves onto the multi-qutrit states, we present efficient quantum circuits considering the nearest neighbour position space for the implementation of lazy quantum walks in one-dimensional position space in ternary quantum system. We also address sca…
Open quantum walks (OQWs) describe a quantum walker on an underlying graph whose dynamics is purely driven by dissipation and decoherence. Mathematically, they are formulated as completely positive trace preserving (CPTP) maps on the space of density matrices for the walker on the graph. Any microscopically derived OQW must include the possibility of remaining on the same site on the graph when the map is applied. We extend the CPTP map to describe a lazy OQW. We derive a central limit theorem for lazy OQWs on a $d$-dimensional lattice, where the distribution converges to a Gaussian. We show t…
Game-playing proofs constitute a powerful framework for non-quantum cryptographic security arguments, most notably applied in the context of indifferentiability. An essential ingredient in such proofs is lazy sampling of random primitives. We develop a quantum game-playing proof framework by generalizing two recently developed proof techniques. First, we describe how Zhandry's compressed quantum oracles~(Crypto'19) can be used to do quantum lazy sampling of a class of non-uniform function distributions. Second, we observe how Unruh's one-way-to-hiding lemma~(Eurocrypt'14) can also be applied t…
A recent line of work has shown that an overparametrized neural network can perfectly fit the training data, an otherwise often intractable nonconvex optimization problem. For (fully-connected) shallow networks, in the best case scenario, the existing theory requires quadratic over-parametrization as a function of the number of training samples. This paper establishes that linear overparametrization is sufficient to fit the training data, using a simple variant of the (stochastic) gradient descent. Crucially, unlike several related works, the training considered in this paper is not limited to…
In this paper, we study the properties of lackadaisical quantum walks on a line. This model is first proposed in~\cite{wong2015grover} as a quantum analogue of lazy random walks where each vertex is attached $τ$ self-loops. We derive an analytic expression for the localization probability of the walker at the origin after infinite steps, and obtain the peak velocities of the walker. We also calculate rigorously the wave function of the walker starting from the origin and obtain a long time approximation for the entire probability density function. As an application of the density function, we …
We find the necessary and sufficient conditions for the entropy rate of the system to be zero under any system-environment Hamiltonian interaction. We call the class of system-environment states that satisfy this condition lazy states. They are a generalization of classically correlated states defined by quantum discord, but based on projective measurements of any rank. The concept of lazy states permits the construction of a protocol for detecting global quantum correlations using only local dynamical information. We show how quantum correlations to the environment provide bounds to the entro…
The Graduate University for Advanced Studies, SOKENDAI（综合研究大学院大学（SOKENDAI））；National Institute of Informatics（国立信息学研究所）
A multiplicative constant scaling factor is often applied to the model output to adjust the dynamics of neural network parameters. This has been used as one of the key interventions in an empirical study of lazy and active behavior. However, we show that the combination of such scaling and a commonly used adaptive learning rate optimizer strongly affects the training behavior of the neural network. This is problematic as it can cause \emph{unintended behavior} of neural networks, resulting in the misinterpretation of experimental results. Specifically, for some scaling settings, the effect of …
We compare different strategies aimed to prepare an ensemble with a given density matrix $ρ$. Preparing the ensemble of eigenstates of $ρ$ with appropriate probabilities can be treated as `generous' strategy: it provides maximal accessible information about the state. Another extremity is the so-called `Scrooge' ensemble, which is mostly stingy to share the information. We introduce `lazy' ensembles which require minimal efforts to prepare the density matrix by selecting pure states with respect to completely random choice. We consider two parties, Alice and Bob, playing a kind of game. Bob wi…
Qubit routing is a key problematic related to quantum circuit compilation. It consists in rewriting a quantum circuit by adding the least possible number of instructions to make the circuit compliant with some architecture's connectivity constraints. Usually, this problem is tackled via either SWAP insertion techniques or re-synthesis of portions of the circuit using architecture aware synthesis algorithms. In this work, we propose a meta-heuristic that couples the iterative approach of SWAP insertion techniques with greedy architecture aware synthesis routines. We propose two new compilation …
With the motive of training all the parameters of a neural network, we study why and when one can achieve this by iteratively creating, training, and combining randomly selected subnetworks. Such scenarios have either implicitly or explicitly emerged in the recent literature: see e.g., the Dropout family of regularization techniques, or some distributed ML training protocols that reduce communication/computation complexities, such as the Independent Subnet Training protocol. While these methods are studied empirically and utilized in practice, they often enjoy partial or no theoretical support…
We propose a trade-off between the Lipschitz constants of the position and momentum probability distributions for arbitrary quantum states. We refer to the trade-off as a quantum reciprocity relation. The Lipschitz constant of a function may be considered to quantify the extent of fluctuations of that function, and is, in general, independent of its spread. The spreads of the position and momentum distributions are used to obtain the celebrated Heisenberg quantum uncertainty relations. We find that the product of the Lipschitz constants of position and momentum probability distributions is bou…
On the Learning Dynamics of Two-layer Linear Networks with Label Noise SGD
Sch. of Computer Science and Zhiyuan College, Shanghai Jiao Tong University（上海交通大学计算机科学学院和智远学院）；Peking University（北京大学）；University of Sydney（悉尼大学）；RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能项目中心）；The Institute of Statistical Mathematics（统计数学研究所）；The University of Tokyo（东京大学）
The joint state of a system that is in contact with an environment is called lazy, if the entropy rate of the system under any coupling to the environment is zero. Necessary and sufficient conditions have recently been established for a state to be lazy [ Phys. Rev. Lett. 106 050403 (2011)], and it was shown that almost all states of the system and the environment do not have this property [ Phys. Rev. A 81 052318 (2010)]. At first glance, this may lead us to believe that low entropy rates themselves form an exception, in the sense that most states are far from being lazy and have high entropy…
We consider optimizing two-layer neural networks in the mean-field regime where the learning dynamics of network weights can be approximated by the evolution in the space of probability measures over the weight parameters associated with the neurons. The mean-field regime is a theoretically attractive alternative to the NTK (lazy training) regime which is only restricted locally in the so-called neural tangent kernel space around specialized initializations. Several prior works (\cite{chizat2018global, mei2018mean}) establish the asymptotic global optimality of the mean-field regime, but it is…
The ability of neural networks to provide `best in class' approximation across a wide range of applications is well-documented. Nevertheless, the powerful expressivity of neural networks comes to naught if one is unable to effectively train (choose) the parameters defining the network. In general, neural networks are trained by gradient descent type optimization methods, or a stochastic variant thereof. In practice, such methods result in the loss function decreases rapidly at the beginning of training but then, after a relatively small number of steps, significantly slow down. The loss may ev…
Extensive quantum error correction is necessary in order to scale quantum hardware to the regime of practical applications. As a result, a significant amount of decoding hardware is necessary to process the colossal amount of data required to constantly detect and correct errors occurring over the millions of physical qubits driving the computation. The implementation of a recent highly optimized version of Shor's algorithm to factor a 2,048-bits integer would require more 7 TBit/s of bandwidth for the sole purpose of quantum error correction and up to 20,000 decoding units. To reduce the deco…
A Basin-Selection Perspective on Grokking via Singular Learning Theory
Department of Computer Science（计算机科学系）；Department of Mathematics（数学系）；Mathematical Institute（数学研究所）；University of Pisa（比萨大学）；Imperial College London（伦敦帝国学院）；University of Oxford（牛津大学）；Section of Mathematics and Artificial Intelligence（数学与人工智能系）；Max Planck Institute of Molecular Cell Biology and Genetics（马克斯·普朗克分子细胞生物学和遗传学研究所）；Center for Systems Biology（系统生物学中心）；Faculty of Mathematics（数学系）；TU Dresden（德累斯顿技术大学）
Lazy-Move Compilation for Neutral-Atom Quantum Computers via a Buffer-Relay Fabric
Beijing Academy of Quantum Information Sciences（北京量子信息科学研究院）；Department of Computer Science and Engineering, The Chinese University of Hong Kong（香港中文大学计算机科学与工程系）；Open Quantum Intelligence (Shanghai) Co., Ltd.（上海开放量子智能有限公司）；State Key Laboratory of Quantum Information Technologies and Materials, The Chinese University of Hong Kong（香港中文大学量子信息技术与材料国家重点实验室）；CAS Cold Atom Technology (Wuhan) Co., Ltd.（中科院冷原子技术（武汉）有限公司）；Key Laboratory of Quantum Optics and Center of Cold Atom Physics, Shanghai Institute of Optics and Fine Mechanics, Chinese Academy of Sciences（中国科学院上海光学精密机械研究所量子光学重点实验室和冷原子物理中心）；State Key Laboratory of Low Dimensional Quantum Physics, Department of Physics, Tsinghua University（清华大学物理系低维量子物理国家重点实验室）
Precise gradient descent training dynamics for finite-width multi-layer neural networks
Department of Statistics, Rutgers University, Piscataway, NJ 08854, USA.（统计学系，罗格斯大学）；Graduate School of Arts and Sciences, The University of Tokyo, Meguro, Tokyo 153-8902, Japan. RIKEN Center for Advanced Intelligence Project, Chuo, Tokyo 103-0027, Japan.（艺术与科学研究生院，东京大学）
The Diosi-Penrose model of quantum-classical boundary postulates gravity-related spontaneous wave function collapse of massive degrees of freedom. The decoherence effects of the collapses are in principle detectable if not masked by the overwhelming environmental decoherence. But the DP (or any other, like GRW, CSL) spontaneous collapses are not detectable themselves, they are merely the redundant formalism of spontaneous decoherence. To let DP collapses become testable physics, recently we extended the DP model and proposed that DP collapses are responsible for the emergence of the Newton gra…
