Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Models
Ecole Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）；International Center for Theoretical Physics（国际理论物理中心）；Université Pierre et Marie Curie（皮埃尔与玛丽·居里大学）；École Normale Supérieure（巴黎高等师范学院）；PSL Université（巴黎文理研究大学）；Inria（法国国家信息与自动化研究所）；Institut de Physique Théorique（理论物理研究所）；CEA（法国原子能和替代能源委员会）；Université Paris-Saclay（巴黎-萨克雷大学）
Jean Barbier, Florent Krzakala, Nicolas Macris, Léo Miolane, Lenka Zdeborová
备注：
101 pages, 5 figures
发表信息：
Proceedings of the National Academy of Sciences 116. 12 (2019): 5451-5460
Generalized linear models (GLMs) arise in high-dimensional machine learning, statistics, communications and signal processing. In this paper we analyze GLMs when the data matrix is random, as relevant in problems such as compressed sensing, error-correcting codes or benchmark models in neural networks. We evaluate the mutual information (or "free entropy") from which we deduce the Bayes-optimal estimation and generalization errors. Our analysis applies to the high-dimensional limit where both the number of samples and the dimension are large and their ratio is fixed. Non-rigorous predictions for the optimal errors existed for special cases of GLMs, e.g. for the perceptron, in the field of statistical physics based on the so-called replica method. Our present paper rigorously establishes those decades old conjectures and brings forward their algorithmic interpretation in terms of performance of the generalized approximate message-passing algorithm. Furthermore, we tightly characterize, for many learning problems, regions of parameters for which this algorithm achieves the optimal performance, and locate the associated sharp phase transitions separating learnable and non-learnable regions. We believe that this random version of GLMs can serve as a challenging benchmark for multi-purpose algorithms. This paper is divided in two parts that can be read independently: The first part (main part) presents the model and main results, discusses some applications and sketches the main ideas of the proof. The second part (supplementary informations) is much more detailed and provides more examples as well as all the proofs.
Alfréd Rényi Institute of Mathematics（阿尔弗雷德·雷尼数学研究所）
Entropic regularization provides a generalization of the original optimal transport problem. It introduces a penalty term defined by the Kullback-Leibler divergence, making the problem more tractable via the celebrated Sinkhorn algorithm. Replacing the Kullback-Leibler divergence with a general $f$-divergence leads to a natural generalization. The case of divergences defined by superlinear functions was recently studied by Di Marino and Gerolin. Using convex analysis, we extend the theory developed so far to include all $f$-divergences defined by functions of Legendre type, and prove that unde…
University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；University of Chicago（芝加哥大学）
This paper describes a flexible framework for generalized low-rank tensor estimation problems that includes many important instances arising from applications in computational imaging, genomics, and network analysis. The proposed estimator consists of finding a low-rank tensor fit to the data under generalized parametric models. To overcome the difficulty of non-convexity in these problems, we introduce a unified approach of projected gradient descent that adapts to the underlying low-rank structure. Under mild conditions on the loss function, we establish both an upper bound on statistical er…
In high-dimensional statistical inference, sparsity regularizations have shown advantages in consistency and convergence rates for coefficient estimation. We consider a generalized version of Sparse-Group Lasso which captures both element-wise sparsity and group-wise sparsity simultaneously. We state one universal theorem which is proved to obtain results on consistency and convergence rates for different forms of double sparsity regularization. The universality of the results lies in an generalization of various convergence rates for single regularization cases such as LASSO and group LASSO a…
The generalized Poor-Verdu error lower bound for multihypothesis testing is revisited. Its asymptotic expression is established in closed-form as its tilting parameter grows to infinity. It is also shown that the asymptotic generalized bound achieves the error exponent (or reliability function) of the memoryless binary symmetric channel at zero coding rates.
We derive the maximum entropy of a flow (information utility) which conforms to traffic constraints imposed by a generalized token bucket regulator, by taking into account the covert information present in the randomness of packet lengths. Under equality constraints of aggregate tokens and aggregate bucket depth, a generalized token bucket regulator can achieve higher information utility than a standard token bucket regulator. The optimal generalized token bucket regulator has a near-uniform bucket depth sequence and a decreasing token increment sequence.
New upper and lower bounds for the error probability over an erasure channel are provided, making use of Wei's generalized weights, hierarchy and spectra. In many situations the upper and lower bounds coincide and this allows improvement of existing bounds. Results concerning MDS and AMDS codes are deduced from those bounds.
We find the exact typical error exponent of constant composition generalized random Gilbert-Varshamov (RGV) codes over DMCs channels with generalized likelihood decoding. We show that the typical error exponent of the RGV ensemble is equal to the expurgated error exponent, provided that the RGV codebook parameters are chosen appropriately. We also prove that the random coding exponent converges in probability to the typical error exponent, and the corresponding non-asymptotic concentration rates are derived. Our results show that the decay rate of the lower tail is exponential while that of th…
Except for a few simple digital modulation techniques, derivation of average bit error probability over fading channels is difficult and is an involved process. In this letter, curve fitting technique has been employed to express bit error probability over AWGN of any digital modulation scheme in terms of a simple Gaussian function. Using this Gaussian function, a generalized closed form expression for computing average probability of bit error over Rayleigh fading channels has been derived. Excellent agreement has been found between error probabilities computed with our method and the rigorou…
This paper introduces a novel class of code sets, called "symmetrical Z-complementary code sets (SZCCSs)" , whose aperiodic auto- and cross- correlation sums exhibit zero-correlation zones (ZCZs) at both the front-end and tail-end of the entire correlation window. Three constructions of (optimal) SZCCSs based on general Boolean functions are presented. As a second major contribution, we apply SZCCSs to design optimal training sequences for broadband generalized spatial modulation (GSM) systems over frequency-selective channels. Key words: Complementary code set, channel estimation, training se…
There are three classical divergence measures exist in the literature on information theory and statistics. These are namely, Jeffryes-Kullback-Leiber J-divergence. Sibson-Burbea-Rao Jensen-Shannon divegernce and Taneja Arithmetic-Geometric divergence. These three measures bear an interesting relationship among each other. The divergence measures like Hellinger discrimination, symmetric chi-square divergence, and triangular discrimination are also known in the literature. In this paper, we have considered generalized symmetric divergence measures having the measures given above as particular c…
Sum-rank metric codes have recently attracted the attention of many researchers, due to their relevance in several applications. Mathematically, the sum-rank metric is a natural generalization of both the Hamming metric and the rank metric. In this paper, we provide an Anticode Bound for the sum-rank metric, which extends the corresponding Hamming and rank-metric Anticode bounds. We classify then optimal anticodes, i.e., codes attaining the sum-rank metric Anticode Bound. We use these optimal anticodes to define generalized sum-rank weights and we study their main properties. In particular, we…
In this paper we extend the works \cite{gegeng2,XLZD} further in two directions and compute the weight distribution of these cyclic codes under more relaxed conditions. It is interesting to note that many cyclic codes in the family are optimal and have only a few non-zero weights. Besides using similar ideas from \cite{gegeng2,XLZD}, we carry out some subtle manipulation of certain exponential sums.
Recently efforts have been made to use generalized sinc functions to perfectly reconstruct various kinds of non-bandlimited signals. As a consequence, perfect reconstruction sampling formulas have been established using such generalized sinc functions. This article studies the error of the reconstructed non-bandlimited signal when an adaptive truncation scheme is employed. Further, when there are noises present in the samples, estimation on the expectation and variance of the error pertinent to the reconstructed signal is also given. Finally discussed are the reproducing properties and the Sob…
This paper presents a novel unified performance analysis of Space-Time Block Codes (STBCs) operating in the Multiple Input Multiple Output (MIMO) network and subjected to generalized shadowed fading and noise scenarios. Specifically, we derive novel, simple and accurate average bit error rates (ABER) expressions for coherent modulation schemes in the generalized η-μ and shadowed κ-μ fading channels. The noise in the network is assumed to be modeled by the additive white generalized Gaussian noise (AWGGN), which encompasses the Laplacian and the Gaussian noise environments as special cases. The…
In this paper, quantizer design for weak-signal detection under arbitrary binary channel in generalized Gaussian noise is studied. Since the performances of the generalized likelihood ratio test (GLRT) and Rao test are asymptotically characterized by the noncentral chi-squared probability density function (PDF), the threshold design problem can be formulated as a noncentrality parameter maximization problem. The theoretical property of the noncentrality parameter with respect to the threshold is investigated, and the optimal threshold is shown to be found in polynomial time with appropriate nu…
Codebooks with small inner-product correlation are applied in many practical applications including direct spread code division multiple access (CDMA) communications, space-time codes and compressed sensing. It is extremely difficult to construct codebooks achieving the Welch bound or the Levenshtein bound. Constructing nearly optimal codebooks such that the ratio of its maximum cross-correlation amplitude to the corresponding bound approaches 1 is also an interesting research topic. In this paper, we firstly study a family of interesting character sums called generalized Jacobi sums over fini…
This letter presents a comprehensive framework analyzing the asymptotic error performance of a multiple-input-multiple-output (MIMO) wireless system employing spatial modulation (SM) with maximum likelihood detection and perfect channel state information. Generic analytical expressions for the diversity and coding gains are deduced that reveal fundamental properties of MIMO SM systems. The presented analysis can be used to obtain closed-form upper bounds for the average bit error probability (ABEP) of MIMO SM systems under generalized fading which become asymptotically tight in the high signal…
Physical Unclonable Functions can be used for secure key generation in cryptographic applications. It is explained how methods from coding theory must be applied in order to ensure reliable key regeneration. Based on previous work, we show ways how to obtain better results with respect to error probability and codeword length. Also, an example based on Generalized Concatenated codes is given, which improves upon used coding schemes for PUFs.
This paper introduces a new and unified bit error rates performance analysis of space-time block codes (STBC) deployed in wireless systems with spatial diversity in generalized shadowed fading and noise scenarios. Specifically, we derive a simple and a very accurate approximate expressions for the average error rates of coherent modulation schemes in generalized $η-μ$ and $κ-μ$ shadowed fading channels with multiple input multiple output (MIMO) systems. The noise in the network is assumed to be modeled using the additive white generalized Gaussian noise (AWGGN), which encompasses the classical…
In this letter, we formulate a generalized decision fusion problem (GDFP) for sensing with centralized hard decision fusion. We show that various new and existing decision fusion rules are special cases of the proposed GDFP. We then relate our problem to the classical $0-1$ Knapsack problem (KP). Consequently, we apply dynamic programming to solve the exponentially complex GDFP in polynomial time. Numerical results are presented to verify the effectiveness of the proposed solution.
Communication reliability improving is one of most important research requirements in cognitive networks, as 5G communications technology rapidly develop nowadays. In this paper, we propose generalized optimal cloud-based region relays subsets paring model in underlay dual-hop cognitive networks, this unified model reveals three relays nodes characteristics of cloud-based cooperative networks in the nearby area- A subset, only needs receiving for first hop, B subset, just receive and forward, C subset, only forwarding. In addition, this generalized model can be converted into various classical…
Optimal locally repairable codes with information locality are considered. Optimal codes are constructed, whose length is also order-optimal with respect to a new bound on the code length derived in this paper. The length of the constructed codes is super-linear in the alphabet size, which improves upon the well known pyramid codes, whose length is only linear in the alphabet size. The recoverable erasure patterns are also analyzed for the new codes. Based on the recoverable erasure patterns, we construct generalized sector-disk (GSD) codes, which can recover from disk erasures mixed with sect…
Protograph low-density-parity-check (LDPC) are considered to design near-capacity low-rate codes over the binary erasure channel (BEC) and binary additive white Gaussian noise (BIAWGN) channel. For protographs with degree-one variable nodes and doubly-generalized LDPC (DGLDPC) codes, conditions are derived to ensure equality of bit-error threshold and block-error threshold. Using this condition, low-rate codes with block-error threshold close to capacity are designed and shown to have better error rate performance than other existing codes.
The small sample universal hypothesis testing problem is investigated in this paper, in which the number of samples $n$ is smaller than the number of possible outcomes $m$. The goal of this work is to find an appropriate criterion to analyze statistical tests in this setting. A suitable model for analysis is the high-dimensional model in which both $n$ and $m$ increase to infinity, and $n=o(m)$. A new performance criterion based on large deviations analysis is proposed and it generalizes the classical error exponent applicable for large sample problems (in which $m=O(n)$). This generalized err…
In estimating frequencies given that the signal waveforms are undersampled multiple times, Xia et. al. proposed to use a generalized version of Chinese remainder Theorem (CRT), where the moduli are $M_1, M_2, \cdots, M_k$ which are not necessarily pairwise coprime. If the errors of the corrupted remainders are within $τ=\sds \max_{1\le i\le k} \min_{\stackrel{1\le j\le k}{j\neq i}} \frac{\gcd(M_i,M_j)}4$, their schemes can be used to construct an approximation of the solution to the generalized CRT with an error smaller than $τ$. Accurately finding the quotients is a critical ingredient in the…
We study the performance of generalized polar (GP) codes when they are used for coding schemes involving erasure. GP codes are a family of codes which contains, among others, the standard polar codes of Arıkan and Reed-Muller codes. We derive a closed formula for the zero-undetected-error capacity $I_0^{GP}(W)$ of GP codes for a given binary memoryless symmetric (BMS) channel $W$ under the low complexity successive cancellation decoder with erasure. We show that for every $R<I_0^{GP}(W)$, there exists a generalized polar code of blocklength $N$ and of rate at least $R$ where the undetected-err…
We introduce a definition of perfect and quasi-perfect codes for symmetric channels parametrized by an auxiliary output distribution. This notion generalizes previous definitions of perfect and quasi-perfect codes and encompasses maximum distance separable codes. The error probability of these codes, whenever they exist, is shown to coincide with the estimate provided by the meta-converse lower bound. We illustrate how the proposed definition naturally extends to cover almost-lossless source-channel coding and lossy compression.
In this note, we present a new adaptive algorithm for generalized group testing, which is asymptotically optimal if $d=o(\log_2|E|)$, $E$ is a set of potentially contaminated sets, $d$ is a maximal size of elements of $E$. Also, we design a 3-stage algorithm, which is asymptotically optimal for $d=2$.
The goal of the classic football-pool problem is to determine how many lottery tickets are to be bought in order to guarantee at least $n-r$ correct guesses out of a sequence of $n$ games played. We study a generalized (second-order) version of this problem, in which any of these $n$ games consists of two sub-games. The second-order version of the football-pool problem is formulated using the notion of generalized-covering radius, recently proposed as a fundamental property of linear codes. We consider an extension of this property to general (not necessarily linear) codes, and provide an asym…
In a recent paper \cite{Zhang-Xiao}, Zhang and Xiao describe a technique on constructing almost optimal resilient functions on even number of variables. In this paper, we will present an extensive study of the constructions of almost optimal resilient functions by using the generalized Maiorana-McFarland (GMM) construction technique. It is shown that for any given $m$, it is possible to construct infinitely many $n$-variable ($n$ even), $m$-resilient Boolean functions with nonlinearity equal to $2^{n-1}-2^{n/2-1}-2^{k-1}$ where $k<n/2$. A generalized version of GMM construction is further desc…
