arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1601.04650stat.MLcond-mat.dis-nncond-mat.stat-mechmath.STq-bio.QMstat.TH
收藏

高维推理的统计力学

Statistical Mechanics of High-Dimensional Inference

Madhu Advani, Surya Ganguli

摘要:

为了对现代大规模数据集进行建模,我们需要有效的算法来从$N$噪声测量中推断出一组$P$未知的模型参数。在有限信噪比、有限测量、先验信息和计算易处理性要求的情况下,参数推断准确性的基本限制是什么?我们如何将先验信息与测量相结合,以达到这些限制?经典统计学给出了这些问题的精辟回答,如测量密度$α= \frac{N}{P}\rightarrow \infty$。然而,这些经典的结果是不相关的现代高维推理问题,而不是发生在有限的$α$。我们制定和分析高维推理淬火无序的统计物理中的一个问题。我们的分析揭示了在高维推理的准确性的基本限制,并揭示了广泛珍视的推理算法,如极大似然(ML)和最大后验(MAP)推理不能达到这些限制。我们进一步找到最佳的,计算上易于处理的算法,可以实现这些限制。有趣的是,在高维中,这些优化算法在计算上比MAP和ML更简单,同时仍然优于它们。例如,这种最佳算法可以导致数据量减少多达20%,以实现相对于MAP的相同性能。此外,我们的分析揭示了最佳高维推理和低维标量贝叶斯推断之间的简单关系,对高维泛化和预测能力的本质的见解,压缩感知的信息理论限制,二次推理中的相变,以及凸优化理论和随机矩阵理论中的中心数学对象的连接。

英文摘要:

To model modern large-scale datasets, we need efficient algorithms to infer a set of $P$ unknown model parameters from $N$ noisy measurements. What are fundamental limits on the accuracy of parameter inference, given finite signal-to-noise ratios, limited measurements, prior information, and computational tractability requirements? How can we combine prior information with measurements to achieve these limits? Classical statistics gives incisive answers to these questions as the measurement density $α= \frac{N}{P}\rightarrow \infty$. However, these classical results are not relevant to modern high-dimensional inference problems, which instead occur at finite $α$. We formulate and analyze high-dimensional inference as a problem in the statistical physics of quenched disorder. Our analysis uncovers fundamental limits on the accuracy of inference in high dimensions, and reveals that widely cherished inference algorithms like maximum likelihood (ML) and maximum-a posteriori (MAP) inference cannot achieve these limits. We further find optimal, computationally tractable algorithms that can achieve these limits. Intriguingly, in high dimensions, these optimal algorithms become computationally simpler than MAP and ML, while still outperforming them. For example, such optimal algorithms can lead to as much as a 20% reduction in the amount of data to achieve the same performance relative to MAP. Moreover, our analysis reveals simple relations between optimal high dimensional inference and low dimensional scalar Bayesian inference, insights into the nature of generalization and predictive power in high dimensions, information theoretic limits on compressed sensing, phase transitions in quadratic inference, and connections to central mathematical objects in convex optimization theory and random matrix theory.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑