Choosing appropriate hyperparameters plays a crucial role in the success of neural networks as hyper-parameters directly control the behavior and performance of the training algorithms. To obtain efficient tuning, Bayesian optimization methods based on Gaussian process (GP) models are widely used. Despite numerous applications of Bayesian optimization in deep learning, the existing methodologies are developed based on a convenient but restrictive assumption that the tuning parameters are independent of each other. However, tuning parameters with conditional dependence are common in practice. In this paper, we focus on two types of them: branching and nested parameters. Nested parameters refer to those tuning parameters that exist only within a particular setting of another tuning parameter, and a parameter within which other parameters are nested is called a branching parameter. To capture the conditional dependence between branching and nested parameters, a unified Bayesian optimization framework is proposed. The sufficient conditions are rigorously derived to guarantee the validity of the kernel function, and the asymptotic convergence of the proposed optimization framework is proven under the continuum-armed-bandit setting. Based on the new GP model, which accounts for the dependent structure among input variables through a new kernel function, higher prediction accuracy and better optimization efficiency are observed in a series of synthetic simulations and real data applications of neural networks. Sensitivity analysis is also performed to provide insights into how changes in hyperparameter values affect prediction accuracy.
Nested Slice Sampling: Vectorized Nested Sampling for GPU-Accelerated Inference
Kavli Institute for Cosmology Cambridge（剑桥卡维利宇宙研究所）；Institute of Astronomy, University of Cambridge（剑桥大学天文研究所）；Cavendish Laboratory and Department of Engineering, University of Cambridge（剑桥大学卡文迪许实验室和工程系）
Bilevel optimization for learning hyperparameters: Application to solving PDEs and inverse problems with Gaussian processes
Department of Mathematics, Cornell University（数学系，康奈尔大学）；Department of Computing and Mathematical Sciences, California Institute of Technology（计算与数学科学系，加州理工学院）
Nested cross-validation when selecting classifiers is overzealous for most practical applications
University of Campinas（坎皮纳斯大学）；University of East Anglia（东英吉利大学）
当选择要应用于特定问题的分类算法时，必须同时选择该数据集的最佳算法\emph{and}和所选模型的最佳超参数集。通常的方法是应用嵌套交叉验证过程;在内部交叉验证中执行超参数选择，而外部交叉验证计算算法\emph{with cross-validation based hyperparameter tuning}的预期准确度的无偏估计。另一种方法，我们称之为“平面交叉验证”，使用一个单一的交叉验证步骤来选择最佳超参数值，并提供算法的预期准确度的估计，尽管有偏差，但仍然可以用于选择最佳学习算法。我们使用12种不同的算法对115个真实生活中的二进制数据集进行了测试，并得出结论，使用计算成本较低的平面交叉验证程序通常会导致选择一种算法，该算法对于所有实际目的而言，与通过嵌套交叉验证选择的算法具有相似的质量，前提是学习算法具有相对较少的超参数进行优化。
