Learning machines which have hierarchical structures or hidden variables are singular statistical models because they are nonidentifiable and their Fisher information matrices are singular. In singular statistical models, neither the Bayes a posteriori distribution converges to the normal distribution nor the maximum likelihood estimator satisfies asymptotic normality. This is the main reason why it has been difficult to predict their generalization performances from trained states. In this paper, we study four errors, (1) Bayes generalization error, (2) Bayes training error, (3) Gibbs generalization error, and (4) Gibbs training error, and prove that there are mathematical relations among these errors. The formulas proved in this paper are equations of states in statistical estimation because they hold for any true distribution, any parametric model, and any a priori distribution. Also we show that Bayes and Gibbs generalization errors are estimated by Bayes and Gibbs training errors, and propose widely applicable information criteria which can be applied to both regular and singular statistical models.
A likelihood approach to nonparametric estimation of a singular distribution using deep generative models
Pohang University of Science and Technology（浦项科技大学）；Sungshin Women’s University（诚信女子大学）；Seoul National University（首尔国立大学）；University of Notre Dame（圣母大学）
Estimation of minimum miscibility pressure (MMP) in impure/pure N2 based enhanced oil recovery process: A comparative study of statistical and machine learning algorithms
University of Shanghai for Science and Technology（上海理工大学）；University of Alberta（阿尔伯塔大学）
Unified estimation framework for unnormalized models with statistical efficiency
Harvard University（哈佛大学）；Tokyo Institute of Technology（东京工业大学）；RIKEN AIP（理化学研究所人工智能研究中心）；Future University Hakodate（函馆未来大学）；The University of Tokyo（东京大学）；RIKEN Center for Brain Science（理化学研究所脑科学中心）
Projection-Free Algorithms in Statistical Estimation
H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology（佐治亚理工学院工业与系统工程学院）；Department of Electrical Engineering, Technion（以色列理工学院电气工程系）
Communication Lower Bounds for Statistical Estimation Problems via a Distributed Data Processing Inequality
Princeton University（普林斯顿大学）；Toyota Technological Institute at Chicago（芝加哥丰田技术研究院）；IBM Research Almaden（IBM阿尔马登研究中心）
我们研究了高维分布式统计估计问题的统计误差和通信成本之间的权衡。在分布式稀疏高斯均值估计问题中，每个$m$机器接收来自具有未知均值$θ$的$d$维高斯分布的$n$数据点，该均值被承诺为$k$ -稀疏的。机器通过消息传递进行通信，目的是估计平均值$θ$。我们提供了一个紧密的（对数因子）之间的权衡估计误差和机器之间通信的位数。这直接导致了分布式\textit{sparse linear regression}问题的下限：为了实现统计极大极小误差，总通信至少是$Ω(\min\{n,d\}m)$，其中$n$是每台机器接收的观测数，$d$是环境维度。通过允许多轮迭代通信模型，这些较低的结果改进了[Sha14，SD'14]。我们也给出了第一个最佳的同时协议在稠密的情况下，均值估计。作为我们的主要技术，我们证明了一个\textit{distributed data processing inequality}，作为一般的数据处理不等式的推广，这可能是独立的兴趣和有用的其他问题。
Statistical Mean Estimation with Coded Relayed Observations
Department of Computer Science, School of Computing, National University of Singapore（计算机科学系，计算学院，新加坡国立大学）；Department of Applied Mathematics and Statistics（应用数学与统计系）；Department of Computer Science, Department of Mathematics, and Institute of Data Science, National university of Singapore（计算机科学系，数学系，数据科学研究所，新加坡国立大学）
