广义线性模型中高斯混合的普适性定律
Universality laws for Gaussian mixtures in generalized linear models
AI总结:
该研究针对广义线性模型的混合分布样本,刻画了经验风险最小化与吉布斯采样得到的估计量族的渐近联合统计量仅依赖类别条件特征分布均值与协方差的条件,证明了训练、泛化误差等的普适性,支撑高维统计相关研究并可应用于多个机器学习任务。
摘要:
设$(x_{i}, y_{i})_{i=1,\n}$为从一般混合分布$\sum_{c\in\mathcal{C}}ρ_{c}P_{c}^{x}$中抽取的独立样本,考虑广义线性模型的假设类$\hat{y} = F(Θ^{\top}x)$。本文研究了通过两种方式得到的广义线性估计量族$(Θ_{1}, \dots, Θ_{M})$的渐近联合统计量:(a)最小化经验风险$\hat{R}_{n}(Θ;X,y)$,或(b)从对应的吉布斯测度$\exp(-βn \hat{R}_{n}(Θ;X,y))$中采样。我们的主要贡献是刻画了在何种条件下,该估计量族的渐近联合统计量(在弱意义下)仅依赖于类别条件特征分布$P_{c}^{x}$的均值和协方差。特别地,这使得我们能够证明训练误差、泛化误差等多种关注量的普适性,为高维统计领域中在高斯混合假设下开展的一系列近期研究提供了理论支撑。最后,我们讨论了该结果在集成学习、不确定性估计等多个受关注的机器学习任务中的应用。
英文摘要:
Let $(x_{i}, y_{i})_{i=1,\dots,n}$ denote independent samples from a general mixture distribution $\sum_{c\in\mathcal{C}}ρ_{c}P_{c}^{x}$, and consider the hypothesis class of generalized linear models $\hat{y} = F(Θ^{\top}x)$. In this work, we investigate the asymptotic joint statistics of the family of generalized linear estimators $(Θ_{1}, \dots, Θ_{M})$ obtained either from (a) minimizing an empirical risk $\hat{R}_{n}(Θ;X,y)$ or (b) sampling from the associated Gibbs measure $\exp(-βn \hat{R}_{n}(Θ;X,y))$. Our main contribution is to characterize under which conditions the asymptotic joint statistics of this family depends (on a weak sense) only on the means and covariances of the class conditional features distribution $P_{c}^{x}$. In particular, this allow us to prove the universality of different quantities of interest, such as the training and generalization errors, redeeming a recent line of work in high-dimensional statistics working under the Gaussian mixture hypothesis. Finally, we discuss the applications of our results to different machine learning tasks of interest, such as ensembling and uncertainty
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。