In this paper we study the properties of the quenched pressure of a multi-layer spin-glass model (a deep Boltzmann Machine in artificial intelligence jargon) whose pairwise interactions are allowed between spins lying in adjacent layers and not inside the same layer nor among layers at distance larger than one. We prove a theorem that bounds the quenched pressure of such a K-layer machine in terms of K Sherrington-Kirkpatrick spin glasses and use it to investigate its annealed region. The replica-symmetric approximation of the quenched pressure is identified and its relation to the annealed one is considered. The paper also presents some observation on the model's architectural structure related to machine learning. Since escaping the annealed region is mandatory for a meaningful training, by squeezing such region we obtain thermodynamical constraints on the form factors. Remarkably, its optimal escape is achieved by requiring the last layer to scale sub-linearly in the network size.
The parallel annealing method is one of the promising approaches for large scale simulations as potentially scalable on any parallel architecture. We present an implementation of the algorithm on the hybrid program architecture combining CUDA and MPI. The problem is to keep all general-purpose graphics processing unit devices as busy as possible redistributing replicas and to do that efficiently. We provide details of the testing on Intel Skylake/Nvidia V100 based hardware running in parallel more than two million replicas of the Ising model sample. The results are quite optimistic because the…
Multi-Mode Quantum Annealing for Generative Representation Learning with Boltzmann Priors
Department of Statistics and Data Science, Yonsei University, Seoul 03722, Republic of Korea（统计与数据科学系，延世大学，首尔03722，大韩民国）；Department of Applied Statistics, Yonsei University, Seoul 03722, Republic of Korea（应用统计学系，延世大学，首尔03722，大韩民国）；Department of Quantum Information, Yonsei University, Seoul 03722, Republic of Korea（量子信息系，延世大学，首尔03722，大韩民国）
Fundamental operating regimes, hyper-parameter fine-tuning and glassiness: towards an interpretable replica-theory for trained restricted Boltzmann machines
Sapienza University of Roma（罗马第一大学）；Radboud University（拉德堡德大学）；Saddle Point Science Europe（鞍点科学欧洲研究院）
Frustration-enhanced quantum annealing correction models with additional inter-replica interactions
Graduate School of Science and Technology, Keio University（庆应义塾大学理工学部）；Department of Applied Physics and Physico-Informatics, Keio University（庆应义塾大学应用物理与物理信息学系）；Keio University Sustainable Quantum Artificial Intelligence Center (KSQAIC), Keio University（庆应义塾大学可持续量子人工智能中心）；Human Biology-Microbiome-Quantum Research Center (WPI-Bio2Q), Keio University（庆应义塾大学人类生物学-微生物组-量子研究中心）
Quantum annealing aims to provide a faster method for finding the minima of complicated functions, compared to classical computing, so there is an increasing interest in the relaxation dynamics of quantum spin systems. Moreover, it is known that problems in quantum annealing caused by first order phase transitions can be reduced via appropriate temporal adjustment of control parameters, aimed at steering the system away from local minima. To do this optimally, it would be helpful to predict the evolution of the system at the level of macroscopic observables. Solving the dynamics of a quantum e…
We study the expectation value of the logarithm of the partition function of large binary-to-binary lattice-gas Restricted Boltzmann Machines (RBMs) within a replica-symmetric ansatz, averaging over the disorder represented by the parameters of the RBM Hamiltonian. Averaging over the Hamiltonian parameters is done with a diagonal covariance matrix. Due to the diagonal form of the parameter covariance matrix not being preserved under the isomorphism between the Ising and lattice-gas forms of the RBM, we find differences in the behaviour of the quenched log partition function of the lattice-gas …
The population annealing algorithm introduced by Hukushima and Iba is described. Population annealing combines simulated annealing and Boltzmann weighted differential reproduction within a population of replicas to sample equilibrium states. Population annealing gives direct access to the free energy. It is shown that unbiased measurements of observables can be obtained by weighted averages over many runs with weight factors related to the free energy estimate from the run. Population annealing is well suited to parallelization and may be a useful alternative to parallel tempering for systems …
In this paper we consider the use of certain classical analogues to quantum tunneling behavior to improve the performance of simulated annealing on a discrete spin system of the general Ising form. Specifically, we consider the use of multiple simultaneous spin flips at each annealing step as an analogue to quantum spin coherence as well as modifications of the Boltzmann acceptance probability to mimic quantum tunneling. We find that the use of multiple spin flips can indeed be advantageous under certain annealing schedules, but only for long anneal times.
In this paper, we study the high temperature or low connectivity phase of the Viana-Bray model. This is a diluted version of the well known Sherrington-Kirkpatrick mean field spin glass. In the whole replica symmetric region, we obtain a complete control of the system, proving annealing for the infinite volume free energy, and a central limit theorem for the suitably rescaled fluctuations of the multi-overlaps. Moreover, we show that free energy fluctuations, on the scale 1/N, converge in the infinite volume limit to a non-Gaussian random variable, whose variance diverges at the boundary of th…
Revealing hidden features in unlabeled data is called unsupervised feature learning, which plays an important role in pretraining a deep neural network. Here we provide a statistical mechanics analysis of the unsupervised learning in a restricted Boltzmann machine with binary synapses. A message passing equation to infer the hidden feature is derived, and furthermore, variants of this equation are analyzed. A statistical analysis by replica theory describes the thermodynamic properties of the model. Our analysis confirms an entropy crisis preceding the non-convergence of the message passing eq…
Thermal boundary conditions has played an increasingly important role in revealing the nature of short-range spin glasses and is likely to be relevant also for other disordered systems. Diffusion method initializing each replica with a random boundary condition at the infinite temperature using population annealing has been used in recent large-scale simulations. However, the efficiency of this method can be greatly suppressed because of temperature chaos. For example, most samples have some boundary conditions that are completely eliminated from the population in the process of annealing at l…
We first review, following our earlier studies, the critical behavior of the quantum Sherrington-Kirkpatrick (SK) model at finite as well as at zero temperatures. Through the analysis of the Binder cumulant we determined the entire phase diagram of the model and from the scaling analysis of the numerical data we obtained the correlation length exponent. For both the critical Binder cumulant and the correlation length exponent, we observed a crossover from classical- to quantum-fluctuation-dominated values at a finite temperature. We studied the behavior of the order parameter distribution of t…
Sampling from a Boltzmann distribution is NP-hard and so requires heuristic approaches. Quantum annealing is one promising candidate. The failure of annealing dynamics to equilibrate on practical time scales is a well understood limitation, but does not always prevent a heuristically useful distribution from being generated. In this paper we evaluate several methods for determining a useful operational temperature range for annealers. We show that, even where distributions deviate from the Boltzmann distribution due to ergodicity breaking, these estimates can be useful. We introduce the concep…
Sampling complex potential energies is one of the most pressing challenges of contemporary computational science. Inspired by recent efforts that use quantum effects and discretized Feynman's path integrals to overcome large barriers we propose a replica exchange method. In each replica two copies of the same system with halved potential strengths interact via inelastic springs. The strength of the spring is varied in the different replicas so as to bridge the gap between the infinitely strong spring, that corresponds to the Boltzmann replica and the less tight ones. We enhance the spring leng…
Structural Comparison of Error Mitigation Methods for Ising Machines: Penalty-Spin Model versus Stacked Model
Graduate School of Science and Technology, Keio University（庆应义塾大学理工学研究科）；Department of Applied Physics and Physico-Informatics, Keio University（庆应义塾大学应用物理及物理信息学系）；Keio University Sustainable Quantum Artificial Intelligence Center (KSQAIC), Keio University（庆应义塾大学可持续量子人工智能中心）；Human Biology-Microbiome-Quantum Research Center (WPI-Bio2Q), Keio University（庆应义塾大学生物微生物组量子研究中心）；Green Computing System Research Organization, Waseda University（早稻田大学绿色计算系统研究组织）
Boltzmann-Gibbs measures generated by logarithmically correlated random potentials are multifractal. We investigate the abrupt change ("pre-freezing") of multifractality exponents extracted from the averaged moments of the measure - the so-called inverse participation ratios. The pre-freezing can be identified with termination of the disorder-averaged multifractality spectrum. Naive replica limit employed to study a one-dimensional variant of the model is shown to break down at the pre-freezing point. Further insights are possible when employing zero-dimensional and infinite-dimensional versio…
We analyze models of spin glasses on the two-dimensional square lattice by exploiting symmetry arguments. The replicated partition functions of the Ising and related spin glasses are shown to have many remarkable symmetry properties as functions of the edge Boltzmann factors. It is shown that the applications of homogeneous and Hadamard inverses to the edge Boltzmann matrix indicate reduced complexities when the elements of the matrix satisfy certain conditions, suggesting that the system has special simplicities under such conditions. Using these duality and symmetry arguments we present a co…
We investigate a kinetic Ising model with several single-spin flip dynamics (including Metropolis and heat-bath) on quenched and annealed random regular graphs. As expected, on the quenched structures all proposed algorithms reproduce the same results since the conditions for the detailed balance and the Boltzmann distribution in an equilibrium are satisfied. However, on the annealed graphs situation is far less clear -- the network annealing disturbs the equilibrium moving the system away from it. Consequently, distinct dynamics lead to different steady states. We show that some algorithms ar…
The Boltzmann-Gibbs probability distributions generated by logarithmically correlated random potentials provide a simple yet nontrivial example of disorder-induced multifractal measures. We introduce and discuss two analytically tractable models for such potentials. The first model uses multiplicative cascades and is equivalent to statistical mechanics of directed polymers on disordered trees studied long ago by B. Derrida and H. Spohn. Second model is the infinite-dimensional version of the problem in Euclidean space which can be solved by employing the replica trick. In particular, the latte…
Fast training and sampling of Restricted Boltzmann Machines
INRIA-Saclay（INRIA-萨克莱实验室）；LISN Paris-Saclay University（巴黎萨克雷大学LISN实验室）；Escuela Técnica Superior de Ingenieros Industriales Universidad Politécnica de Madrid（马德里理工大学工业工程师学院）；Departamento de Física Teórica Universidad Complutense de Madrid（马德里complutense大学理论物理系）；LCQB, Sorbonne Université, Laboratoire de Physique Théorique École Normale Supérieure, Paris（巴黎高等师范学院理论物理实验室，索邦大学LCQB）
