本文专注于均值方差估计网络（MVE network）（Nix and Weigend, 1994）的最优实现。这种网络通常作为回归环境中不确定性估计方法的构建模块，例如Concrete dropout（Gal et al., 2017）和Deep Ensembles（Lakshminarayanan et al., 2017）。具体而言，MVE network假设数据由具有均值函数和方差函数的正态分布生成。MVE network输出均值和方差估计，并通过最小化负对数似然来优化网络参数。在我们的论文中，我们提出了两个重要见解。首先，近期工作中报告的收敛困难可以通过遵循原作者简单但经常被忽视的建议来相对容易地避免，即应使用预热期。在此期间，仅以固定方差优化均值。我们通过实验证明了这一步骤的有效性，强调它应成为标准做法。作为旁注，我们研究了在预热之后，是固定均值优化方差更有利，还是同时优化两者更有利。在这里，我们没有观察到显著差异。其次，我们引入了MVE network的一项新颖改进：对均值和方差估计分别进行正则化。我们在玩具示例和多个基准UCI回归数据集上证明，遵循原始建议和新颖的独立正则化可以带来显著改进。
英文摘要：
This paper focusses on the optimal implementation of a Mean Variance Estimation network (MVE network) (Nix and Weigend, 1994). This type of network is often used as a building block for uncertainty estimation methods in a regression setting, for instance Concrete dropout (Gal et al., 2017) and Deep Ensembles (Lakshminarayanan et al., 2017). Specifically, an MVE network assumes that the data is produced from a normal distribution with a mean function and variance function. The MVE network outputs a mean and variance estimate and optimizes the network parameters by minimizing the negative loglikelihood. In our paper, we present two significant insights. Firstly, the convergence difficulties reported in recent work can be relatively easily prevented by following the simple yet often overlooked recommendation from the original authors that a warm-up period should be used. During this period, only the mean is optimized with a fixed variance. We demonstrate the effectiveness of this step through experimentation, highlighting that it should be standard practice. As a sidenote, we examine whether, after the warm-up, it is beneficial to fix the mean while optimizing the variance or to optimize both simultaneously. Here, we do not observe a substantial difference. Secondly, we introduce a novel improvement of the MVE network: separate regularization of the mean and the variance estimate. We demonstrate, both on toy examples and on a number of benchmark UCI regression data sets, that following the original recommendations and the novel separate regularization can lead to significant improvements.
Confidence Interval Construction and Conditional Variance Estimation with Dense ReLU Networks
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；University of California, Santa Cruz（加利福尼亚大学圣克鲁兹分校）；Hong Kong University of Science and Technology（香港科技大学）；Washington University in St. Louis（圣路易斯华盛顿大学）
Gradient Estimation for Binary Latent Variables via Gradient Variance Clipping
Columbia University（哥伦比亚大学）；Memorial Sloan Kettering Cancer Center（纪念斯隆-凯特琳癌症中心）；Howard Hughes Medical Institute（霍华德·休斯医学研究所）；University of Florida（佛罗里达大学）
Variance reduced Shapley value estimation for trustworthy data valuation
Virginia Tech（弗吉尼亚理工大学）；China Aerospace Academy of Systems Science and Engineering（中国航天系统科学与工程研究院）；Southern University of Science and Technology（南方科技大学）
Cooperative Variance Estimation and Bayesian Neural Networks for Disentangling Aleatoric and Epistemic Uncertainties
Faculty of Mechanical Engineering, Delft University of Technology, Mekelweg 2, Delft, 2628 CD, The Netherlands（代尔夫特理工大学机械工程学院）；School of Engineering, Brown University, 184 Hope St., Providence, RI 02912, USA（布朗大学工程学院）
Conditional Mean and Variance Estimation via \textit{k}-NN Algorithm with Automated Variance Selection
Harvard T.H. Chan School of Public Health（哈佛大学T.H. Chan公共卫生学院）；University of Santiago de Compostela（圣西蒙大学）；University of California, Los Angeles（加州大学洛杉矶分校）
Multivariate Density Estimation via Variance-Reduced Sketching
Committee on Computational and Applied Mathematics, University of Chicago（芝加哥大学计算与应用数学委员会）；Department of Statistics, University of Chicago（芝加哥大学统计学系）；Department of Applied and Computational Mathematics and Statistics, University of Notre Dame（南达科他大学应用与计算数学与统计学系）
