We study the free energy of a most used deep architecture for restricted Boltzmann machines, where the layers are disposed in series. Assuming independent Gaussian distributed random weights, we show that the error term in the so-called replica symmetric sum rule can be optimised as a saddle point. This leads us to conjecture that in the replica symmetric approximation the free energy is given by a min max formula, which parallels the one achieved for two-layer case.
Fundamental operating regimes, hyper-parameter fine-tuning and glassiness: towards an interpretable replica-theory for trained restricted Boltzmann machines
Sapienza University of Roma（罗马第一大学）；Radboud University（拉德堡德大学）；Saddle Point Science Europe（鞍点科学欧洲研究院）
We study the expectation value of the logarithm of the partition function of large binary-to-binary lattice-gas Restricted Boltzmann Machines (RBMs) within a replica-symmetric ansatz, averaging over the disorder represented by the parameters of the RBM Hamiltonian. Averaging over the Hamiltonian parameters is done with a diagonal covariance matrix. Due to the diagonal form of the parameter covariance matrix not being preserved under the isomorphism between the Ising and lattice-gas forms of the RBM, we find differences in the behaviour of the quenched log partition function of the lattice-gas …
We consider a multi-layer Sherrington-Kirkpatrick spin-glass as a model for deep restricted Boltzmann machines and we solve for its quenched free energy, in the thermodynamic limit and allowing for a first step of replica symmetry breaking. This result is accomplished rigorously exploiting interpolating techniques and recovering the expression already known for the replica-symmetry case. Further, we drop the restriction constraint by introducing intra-layer connections among spins and we show that the resulting system can be mapped into a modular Hopfield network, which is also addressed rigor…
In this paper we study the properties of the quenched pressure of a multi-layer spin-glass model (a deep Boltzmann Machine in artificial intelligence jargon) whose pairwise interactions are allowed between spins lying in adjacent layers and not inside the same layer nor among layers at distance larger than one. We prove a theorem that bounds the quenched pressure of such a K-layer machine in terms of K Sherrington-Kirkpatrick spin glasses and use it to investigate its annealed region. The replica-symmetric approximation of the quenched pressure is identified and its relation to the annealed on…
A class of deep Boltzmann machines is considered in the simplified framework of a quenched system with Gaussian noise and independent entries. The quenched pressure of a $K$-layers spin glass model is studied allowing interactions only among consecutive layers. A lower bound for the pressure is found in terms of a convex combination of $K$ Sherrington-Kirkpatrick models and used to study the annealed and replica symmetric regimes of the system. A map with a one dimensional monomer-dimer system is identified and used to rigorously control the annealed region at arbitrary depth $K$ with the meth…
Restricted Boltzmann machines (RBMs) constitute one of the main models for machine statistical inference and they are widely employed in Artificial Intelligence as powerful tools for (deep) learning. However, in contrast with countless remarkable practical successes, their mathematical formalization has been largely elusive: from a statistical-mechanics perspective these systems display the same (random) Gibbs measure of bi-partite spin-glasses, whose rigorous treatment is notoriously difficult. In this work, beyond providing a brief review on RBMs from both the learning and the retrieval pers…
Aim of this paper is to give an extensive treatment of bipartite mean field spin systems, ordered and disordered: at first, bipartite ferromagnets are investigated, achieving an explicit expression for the free energy trough a new minimax variational principle. Furthermore via the Hamilton-Jacobi technique the same free energy structure is obtained together with the existence of its thermodynamic limit and the minimax principle is connected to a standard max one. The same is investigated for bipartite spin-glasses: By the Borel-Cantelli lemma a control of the high temperature regime is obtaine…
We sketch a new framework for the analysis of disordered systems, in particular mean field spin glasses, which is variational in nature and within the formalism of classical thermodynamics. For concreteness, only the Sherrington-Kirkpatrick model is considered here. For this we show how the Parisi solution (replica symmetric, or when replica symmetry is broken) emerges, in large but finite volumes, from a high temperature expansion to second order of the Gibbs potential with respect to order parameters encoding the law of the effective fields. In contrast with classical systems where convexity…
The effective potential for the two-replica system of the random energy model is exactly derived. It is an analytic function of the magnetizations of two replicas, $φ^1$ and $φ^2$ in the high-temperature phase. In the low-temperature phase, where the replica symmetry breaking takes place, the effective potential becomes non-analytic when $φ^1=φ^2$. The non-analyticity is considered as a consequence of the condensation of the Boltzmann measure, which is a typical property of a glass phase.
Revealing hidden features in unlabeled data is called unsupervised feature learning, which plays an important role in pretraining a deep neural network. Here we provide a statistical mechanics analysis of the unsupervised learning in a restricted Boltzmann machine with binary synapses. A message passing equation to infer the hidden feature is derived, and furthermore, variants of this equation are analyzed. A statistical analysis by replica theory describes the thermodynamic properties of the model. Our analysis confirms an entropy crisis preceding the non-convergence of the message passing eq…
We study increasing subsequences (IS) for an ensemble of sequences given by permutation of numbers {1,2,...,n}. We consider a Boltzmann ensemble at temperature T. Thus each IS appears with the corresponding Boltzmann probability where the energy is the negative length -l of the IS. For T -> 0, only ground states, i.e. longest IS (LIS) contribute, also called Ulam's problem. We introduce an algorithm which allows us to directly sample IS in perfect equilibrium in polynomial time, for any given sequence and any temperature. Thus, we can study very large sizes. We obtain averages for the first an…
Yamagata University（山形大学）；Tokyo Institute of Technology（东京工业大学）
In this study, we consider an empirical Bayes method for Boltzmann machines and propose an algorithm for it. The empirical Bayes method allows estimation of the values of the hyperparameters of the Boltzmann machine by maximizing a specific likelihood function referred to as the empirical Bayes likelihood function in this study. However, the maximization is computationally hard because the empirical Bayes likelihood function involves intractable integrations of the partition function. The proposed algorithm avoids this computational problem by using the replica method and the Plefka expansion.…
Extracting automatically the complex set of features composing real high-dimensional data is crucial for achieving high performance in machine--learning tasks. Restricted Boltzmann Machines (RBM) are empirically known to be efficient for this purpose, and to be able to generate distributed and graded representations of the data. We characterize the structural conditions (sparsity of the weights, low effective temperature, nonlinearities in the activation functions of hidden units, and adaptation of fields maintaining the activity in the visible layer) allowing RBM to operate in such a composit…
Università del Salento（萨伦托大学）；Universität Zürich（苏黎世大学）；King College London（伦敦国王学院）；Scuola Normale Superiore（比萨高等师范学校）
We study Generalised Restricted Boltzmann Machines with generic priors for units and weights, interpolating between Boolean and Gaussian variables. We present a complete analysis of the replica symmetric phase diagram of these systems, which can be regarded as Generalised Hopfield models. We underline the role of the retrieval phase for both inference and learning processes and we show that retrieval is robust for a large class of weight and unit priors, beyond the standard Hopfield scenario. Furthermore we show how the paramagnetic phase boundary is directly related to the optimal size of the…
Deep learning methods relying on multi-layered networks have been actively studied in a wide range of fields in recent years, and deep Boltzmann machines(DBMs) is one of them. In this study, a model of DBMs with some properites of weight parameters obtained by learning is studied theoretically by a statistical-mechanical approach based on the replica method. The phases characterizing the role of DBMs as a generator and their phase diagram are derived, depending meaningfully on the numbers of layers and units in each layer. It is found, in particular, that the correlation between the weight par…
Boltzmann-Gibbs measures generated by logarithmically correlated random potentials are multifractal. We investigate the abrupt change ("pre-freezing") of multifractality exponents extracted from the averaged moments of the measure - the so-called inverse participation ratios. The pre-freezing can be identified with termination of the disorder-averaged multifractality spectrum. Naive replica limit employed to study a one-dimensional variant of the model is shown to break down at the pre-freezing point. Further insights are possible when employing zero-dimensional and infinite-dimensional versio…
Restricted Boltzmann Machines are key tools in Machine Learning and are described by the energy function of bipartite spin-glasses. From a statistical mechanical perspective, they share the same Gibbs measure of Hopfield networks for associative memory. In this equivalence, weights in the former play as patterns in the latter. As Boltzmann machines usually require real weights to be trained with gradient descent like methods, while Hopfield networks typically store binary patterns to be able to retrieve, the investigation of a mixed Hebbian network, equipped with both real (e.g., Gaussian) and…
In this work we consider the {\em analog bipartite spin-glass} (or {\em real-valued restricted Boltzmann machine} in a neural network jargon), whose variables (those quenched as well as those dynamical) share standard Gaussian distributions. First, via Guerra's interpolation technique, we express its quenched free energy in terms of the natural order parameters of the theory (namely the self- and two-replica overlaps), then, we re-obtain the same result by using the replica-trick: a mandatory tribute, given the special occasion. Next, we show that the quenched free energy of this model is the …
The Boltzmann-Gibbs probability distributions generated by logarithmically correlated random potentials provide a simple yet nontrivial example of disorder-induced multifractal measures. We introduce and discuss two analytically tractable models for such potentials. The first model uses multiplicative cascades and is equivalent to statistical mechanics of directed polymers on disordered trees studied long ago by B. Derrida and H. Spohn. Second model is the infinite-dimensional version of the problem in Euclidean space which can be solved by employing the replica trick. In particular, the latte…
We construct a N-dimensional Gaussian landscape with multiscale, translation invariant, logarithmic correlations and investigate the statistical mechanics of a single particle in this environment. In the limit of high dimension N>>1 the free energy of the system and overlap function are calculated exactly using the replica trick and Parisi's hierarchical ansatz. In the thermodynamic limit, we recover the most general version of the Derrida's Generalized Random Energy Model (GREM). The low-temperature behaviour depends essentially on the spectrum of length scales involved in the construction of…
The population annealing algorithm introduced by Hukushima and Iba is described. Population annealing combines simulated annealing and Boltzmann weighted differential reproduction within a population of replicas to sample equilibrium states. Population annealing gives direct access to the free energy. It is shown that unbiased measurements of observables can be obtained by weighted averages over many runs with weight factors related to the free energy estimate from the run. Population annealing is well suited to parallelization and may be a useful alternative to parallel tempering for systems …
Exploring the Energy Landscape of RBMs: Reciprocal Space Insights into Bosons, Hierarchical Learning and Symmetry Breaking
TRIUMF；Perimeter Institute for Theoretical Physics（佩里米特理论物理研究所）；University of Virginia（弗吉尼亚大学）；Computer Science and Biocomplexity Institute（计算机科学与生物复杂性研究所）；Department of Physics and Astronomy（物理与天文学系）；University of Waterloo（滑铁卢大学）
